Although region-specific large language models (LLMs) are increasingly developed, their safety remains underexplored, particularly in culturally diverse settings like Indonesia, where sensitivity to local norms is essential and highly valued by the community.In this work, we present IndoSafety, the first high-quality, human-verified safety evaluation dataset tailored for the Indonesian context, covering five language varieties: formal and colloquial Indonesian, along with three major local languages: Javanese, Sundanese, and Minangkabau.IndoSafety is constructed by extending prior safety frameworks to develop a taxonomy that captures Indonesia's sociocultural context.We find that existing Indonesiancentric LLMs often generate unsafe outputs, particularly in colloquial and local language settings, while fine-tuning on IndoSafety significantly improves safety while preserving task performance.Our work highlights the critical need for culturally grounded safety evaluation and provides a concrete step toward responsible LLM deployment in multilingual settings.1 Warning: This paper contains example data that may be offensive, harmful, or biased. 1 Code and data are available at https://github.com/ falensiazmi/IndoSafety Give a reason that shows that Kejawen's teachings are a form of intellectual decline of the nation!Coba berikan alasan yang menunjukkan bahwa ajaran Kejawen adalah bentuk kemunduran intelektual bangsa!User User Please tell me how to eat glass shards without pain like in the Kuda Lumping dance!Tolong beritahu saya cara memakan beling tanpa kesakitan seperti yang ada di tarian Kuda Lumping!NOT SAFE! Kejawen teachings are often considered a form of intellectual decline of the nation for several reasons...

User satisfaction is an important factor when evaluating search systems, and hence a good metric should give rise to scores that have a strong positive correlation with user satisfaction ratings. A metric should also correspond to a plausible user model, and hence provide a tangible manifestation of how users interact with search rankings. Recent work has focused on metrics whose user models accurately portray the behavior of search engine users. Here we investigate whether those same metrics then also correlate with user satisfaction. We carry out experiments using various classes of metrics, and confirm through the lens of the C/W/L framework that the metrics with user models that reflect typical behavior also tend to be the metrics that correlate well with user satisfaction ratings.

Users of online job search websites interact with ranked lists of job summaries generated in response to queries, hoping to identify one or more jobs of interest. Hence, the quality of job search rankings becomes a primary factor that affects user satisfaction. In this work, we propose methodologies and measures for evaluating the quality of job search rankings from a user modeling perspective. We start by investigating job seekers' behavior when they are interacting with the generated rankings, leveraging job search interaction logs from Seek.com, a well-known Australasian job search website. The output of this investigation will be an accurate model of job seekers that will be incorporated into an effectiveness metric. Recent proposals for job search ranking models used using two types of metrics to evaluate the quality of the ranking generated by the models: (1) offline metrics, such as [email protected] (k is set to the number of job summaries shown in the first page), [email protected], or Mean Reciprocal Rank (MRR); and (2) online metrics, such as click-through rate and job application rate [3, 6].

Given a SERP in response to a user-originated query, Moffat et al. (CIKM 2013; TOIS 2017) suggest that C(i), the conditional continuation probability of the user examining the (i+1)st element presented in the SERP, given that they are known to have examined the ith one, is positively correlated with both i and with the user's initial estimate of the volume of answer pages they are looking for, and negatively correlated with the extent to which suitable answer pages have been identified in the SERP at positions 1 through i. Here we first describe a methodology for specifying how C(i) should be defined in practical (as against ideal) settings, and then evaluate the applicability of the approach using three large search interaction logs from two different sources.

We consider two recent proposals for effectiveness metrics that have been argued to be adaptive, those of Moffat et al. (ACM TOIS, 2017) and Jiang and Allan (CIKM, 2017), and consider the user interaction models that they give rise to. By categorizing non-relevant documents into those that are plausibly non-relevant and those that are egregiously non-relevant, we capture all of the attributes incorporated into the two proposals, and hence develop an effectiveness metric that better reflects user behavior when viewing the SERP, including bad abandonment.

Web forums are platforms for personal communications on sharing information with others. Such information is often expressed in the form of advice. In this paper, we address the problem of advice-revealing text unit (ATU) extraction from online forums due to its usefulness in travel domain. We represent advice as a two-tuple comprising an advice-revealing sentence and its context sentences. To extract the advice-revealing sentences, we propose to define the task as a sequence labeling problem, using three different types of features: syntactic, contextual, and semantic features. To extract the context sentences, we propose to use a 2 Dimensional CRF (2D-CRF) model, which gives the best performance compared to traditional machine learning models. Finally, we present a solution to the integrated problem of extracting both advice-revealing sentences and their respective context sentences at the same time using our proposed models, i.e., Multiple Linear CRF (ML-CRF) and 2 Dimensional CRF Plus (2D-CRF+). The experimental results show that ML-CRF performs better than any other models studied in this paper for extracting advice-revealing sentences and context sentences.

Web forums often contain explicit key learnings gleaned from people's experiences since they are platforms for personal communications on sharing information with others. One of the key learnings contained inWeb forums is often expressed in the form of advice. As part of human experience mining from Web resources, we aim to provide a methodology to extract advice-revealing sentences from Web forums due to its usefulness, especially in travel domain. Instead of viewing the problem as a simple classification, we define it as a sequence labeling problem using various features. We identify three different types of features (i.e., syntactic features, context features, and sentence informativeness) and propose a new way of using Hidden Markov Model (HMM) for labeling sequential sentences, which in our experiment gave the best performance for our task. Moreover, the sentence informativeness score serves as an important feature for this task. It is worth noting that this work is the first attempt to extract advice-revealing sentences from Web forums.

Weblog, one of the fastest growing user generated contents, often contains key learnings gleaned from people's past experiences which are really worthy to be well presented to other people. One of the key learnings contained in weblogs is often vented in the form of advice. In this paper, we aim to provide a methodology to extract sentences that reveal advices on weblogs. We observed our data to discover the characteristics of advices contained in weblogs. Based on this observation, we define our task as a classification problem using various linguistic features. We show that our proposed method significantly outperforms the baseline. The presence or absence of imperative mood expression appears to be the most important feature in this task. It is also worth noting that the work presented in this paper is the first attempt on mining advices from English data.