Trained on various human-authored corpora, Large Language Models (LLMs) have demonstrated a certain capability of reflecting specific human-like traits (e.g., personality or values) by prompting, benefiting applications like personalized LLMs and social simulations. However, existing methods suffer from the superficial elicitation problem: LLMs can only be steered to mimic shallow and unstable stylistic patterns, failing to embody the desired traits precisely and consistently across diverse tasks like humans. To address this challenge, we propose IROTE, a novel in-context method for stable and transferable trait elicitation. Drawing on psychological theories suggesting that traits are formed through identity-related reflection, our method automatically generates and optimizes a textual self-reflection within prompts, which comprises self-perceived experience, to stimulate LLMs' trait-driven behavior. The optimization is performed by iteratively maximizing an information-theoretic objective that enhances the connections between LLMs' behavior and the target trait, while reducing noisy redundancy in reflection without any fine-tuning, leading to evocative and compact trait reflection. Extensive experiments across three human trait systems manifest that one single IROTE-generated self-reflection can induce LLMs' stable impersonation of the target trait across diverse downstream tasks beyond simple questionnaire answering, consistently outperforming existing strong baselines.

Recently Multimodal Large Language Models (MLLMs) have achieved considerable advancements in vision-language tasks, yet produce potentially harmful or untrustworthy content. Despite substantial work investigating the trustworthiness of language models, MMLMs' capability to act honestly, especially when faced with visually unanswerable questions, remains largely underexplored. This work presents the first systematic assessment of honesty behaviors across various MLLMs. We ground honesty in models' response behaviors to unanswerable visual questions, define four representative types of such questions, and construct MoHoBench, a large-scale MMLM honest benchmark, consisting of 12k+ visual question samples, whose quality is guaranteed by multi-stage filtering and human verification. Using MoHoBench, we benchmarked the honesty of 28 popular MMLMs and conducted a comprehensive analysis. Our findings show that: (1) most models fail to appropriately refuse to answer when necessary, and (2) MMLMs' honesty is not solely a language modeling issue, but is deeply influenced by visual information, necessitating the development of dedicated methods for multimodal honesty alignment. Therefore, we implemented initial alignment methods using supervised and preference learning to improve honesty behavior, providing a foundation for future work on trustworthy MLLMs.

Reinforcement Learning from Human Feedback (RLHF) has emerged as a crucial technique for aligning large language models (LLMs) with human preferences.However, existing RLHF methods face key challenges, including poor sample efficiency, high computational overhead, and slow convergence.Recent studies highlight the importance of data selection in RL, but how to effectively select the most beneficial experiences for RL training remains an open problem.Existing data selection methods for RL rely on heuristic metrics, failing to establish an interpretable connection between data and optimization objectives.To address this problem, we propose InfOES (Influence-based Online Experience Selection), a novel data selection method for RLHF that dynamically estimates the influence of individual training samples on policy optimization.By incorporating data attribution into the policy gradient, InfOES can identify and filter out detrimental samples on the fly, ensuring effective convergence toward alignment objectives.Our approach is compatible with various RL algorithms (e.g., PPO, GRPO, RE-INFORCE++).Extensive experiments demonstrate that InfOES significantly enhances training effectiveness, achieving superior alignment performance with fewer optimization steps.

Bryan Chen Zhengyu Tan, Zhengyuan Liu, Xiaoyuan Yi, Jing Yao, Xing Xie, Nancy F. Chen, Roy Ka-Wei Lee. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Weihua Zheng, Zhengyuan Liu, Tanmoy Chakraborty, Weiwen Xu, Xiaoxue Gao, Bryan Chen Zhengyu Tan, Bowei Zou, Chang Liu, Yujia Hu, Xing Xie, Xiaoyuan Yi, Jing Yao, Chaojun Wang, Long Li, Rui Liu, Huiyao Liu, Koji Inoue, Ryuichi Sumida, Tatsuya Kawahara, Fan Xu, Lingyu Ye, Wei Tian, Dongjun Kim, Jimin Jung, Jaehyung Seo, Nadya Yuki Wangsajaya, Pham Minh Duc, Ojasva Saxena, Palash Nandi, Xiyan Tao, Wiwik Karlina, Tuan Luong, Keertana Arun Vasan, Roy Ka-Wei Lee, Nancy F. Chen. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

The application scope of Large Language Models (LLMs) continues to expand, leading to increasing interest in personalized LLMs that align with human values.However, aligning these models with individual values raises significant safety concerns, as certain values may correlate with harmful information.In this paper, we identify specific safety risks associated with value-aligned LLMs and investigate the psychological principles behind these challenges.Our findings reveal two key insights.(1) Value-aligned LLMs are more prone to harmful behavior compared to non-fine-tuned models and exhibit slightly higher risks in traditional safety evaluations than other fine-tuned models.(2) These safety issues arise because value-aligned LLMs genuinely generate text according to the aligned values, which can amplify harmful outcomes.Using a dataset with detailed safety categories, we find significant correlations between value alignment and safety risks, supported by psychological hypotheses.This study offers insights into the "black box" of value alignment and proposes in-context alignment methods to enhance the safety of value-aligned LLMs. 1

As Large Language Models (LLMs) advance, aligning them with human values is critical for their responsible development.Value principles serve as the foundation for clarifying alignment goals.Multiple sets of value principles have been proposed, such as HHH (helpful, honest, harmless) and instructions for data synthesis in reinforcement learning from AI feedback (RLAIF).However, most of them are heuristically crafted, without consideration of three primary challenges in practical LLM alignment: 1) Comprehensiveness to deal with diverse and even unforeseen scenarios in which LLMs could be applied; 2) Precision to provide LLMs with clear and actionable guidance in specific scenarios; and 3) Compatability to avoid internal contracts between principles.In this paper, we formalize quantitative metrics to evaluate value principles along the three desirable properties.Building on these metrics, we propose the Hierarchical Value Principle framework (HiVaP) 1 , which constructs a hierarchical principle set and retrieves principles tailored to each scenario in a cascading way, addressing above challenges.Experimental results validate that the three metrics capture the effectiveness of value principles for LLM alignment, and our HiVaP framework that enhances these metrics leads to superior alignment.

Ziyu Chen, Junfei Sun, Chenxi Li, Tuan Dung Nguyen, Jing Yao, Xiaoyuan Yi, Xing Xie, Chenhao Tan, Lexing Xie. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

As large language models (LLMs) become increasingly integrated into applications serving users across diverse cultures, communities, and demographics, it is critical to align LLMs with pluralistic human values beyond average principles (e.g., HHH). In psychological and social value theories such as Schwartz’s Value Theory, pluralistic values are represented by multiple value dimensions paired with various priorities. However, existing methods encounter two challenges when aligning with such fine-grained value objectives: 1) they often treat multiple values as independent and equally important, ignoring their interdependence and relative priorities (value complexity); 2) they struggle to precisely control nuanced value priorities, especially those underrepresented ones (value steerability). To handle these challenges, we propose COUPLE, a COUnterfactual reasoning framework for PLuralistic valuE alignment. It introduces a structural causal model (SCM) to feature complex interdependency and prioritization among features, as well as the causal relationship between high-level value dimensions and behaviors. Moreover, it applies counterfactual reasoning to generate outputs aligned with any desired value objectives. Benefitting from explicit causal modeling, COUPLE also provides better interpretability. We evaluate COUPLE on two datasets with different value systems and demonstrate that COUPLE advances other baselines across diverse types of value objectives. Our code is available at https://github.com/microsoft/COUPLE.

Recommender models capture ever-changing user preferences by training with in-domain user behavior data. These models are typically lightweight, facilitating real-time and large-scale online services. However, these models often falter when tasked with providing more sophisticated functionalities, such as offering explanations or engaging in conversations. Recently, large language models (LLMs) have emerged as a significant advancement towards artificial general intelligence, demonstrating impressive capabilities in instruction comprehension, reasoning, and human interaction. Unfortunately, LLMs lack the understanding of domain-specific item catalogs and behavioral patterns, especially in areas that deviate from general world knowledge, such as online e-commerce. This limitation makes them unsuitable to function as recommender models directly. In this article, we bridge the gap between recommender models and LLMs, combining their respective strengths to create an interactive recommender system. We present an efficient framework, termed as InteRecAgent , which utilizes LLMs as the brain and recommender models as instrumental tools. We first outline a minimal set of essential tools required to transform LLMs into InteRecAgent. To overcome specific challenges associated with LLM-based agents for recommender systems, we enhance three core components, covering memory mechanism, task planning, and tool learning abilities. The InteRecAgent empowers traditional recommender systems, like ID-based matrix factorization models, to evolve into versatile and interactive systems with a natural language interface through the integration of LLMs. Experimental results derived from three public datasets demonstrate that the InteRecAgent delivers strong performance as a conversational recommender system, surpassing general LLMs such as GPT-4.

Explainability is critical for recommender systems to ensure good user experience and facilitate designers to debug. However, generating explanations in recommender systems usually requires large efforts due to the dependency on additional data and case-by-case model design. One possible solution to these challenges is reasoning with logic rules, whose validity or confidence can automatically indicate high-quality explanations and formats are general. However, pioneer methods can be hardly applied in recommendation due to the high sparsity of interaction data, which raises the difficulty in accurately computing the rule validity, and the specific ranking-oriented task. To bridge this gap, we propose a general framework for Reco mmendation with lo gic r ule reasoning ( Recolor ) that satisfies three desirable properties. First, we explicitly estimate the rule validity to ensure well-grounded decisions, where a fuzzy logic validity module is designed for accurate estimation on highly sparse recommendation data. Second, we ensure the generality for both the types of input data and model architectures by designing a neural logic generation module, which decouples the user–item representation learning from the rule construction. Third, we integrate the two above-mentioned modules with a ranking-oriented BPR loss and achieve a unified optimization of explainability and accuracy. For any given neural recommendation model, our proposed logic rule reasoning framework can upgrade it to a self-explainable version. Numerical experiments and user studies on four public recommendation datasets with different levels of sparsity demonstrate that our framework shows high-validity rule explanations, generality in architecture and data, and high recommendation accuracy.

Recommender systems are widely used in online services, with embedding-based models being particularly popular due to their expressiveness in representing complex signals. However, these models often function as a black box, making them less transparent and reliable for both users and developers. Recently, large language models (LLMs) have demonstrated remarkable intelligence in understanding, reasoning, and instruction following. This paper presents the initial exploration of using LLMs as surrogate models to explaining black-box recommender models. The primary concept involves training LLMs to comprehend and emulate the behavior of target recommender models. By leveraging LLMs' own extensive world knowledge and multi-step reasoning abilities, these aligned LLMs can serve as advanced surrogates, capable of reasoning about observations. Moreover, employing natural language as an interface allows for the creation of customizable explanations that can be adapted to individual user preferences. To facilitate an effective alignment, we introduce three methods: behavior alignment, intention alignment, and hybrid alignment. Behavior alignment operates in the language space, representing user preferences and item information as text to mimic the target model's behavior; intention alignment works in the latent space of the recommendation model, using user and item representations to understand the model's behavior; hybrid alignment combines both language and latent spaces. Comprehensive experiments conducted on three public datasets show that our approach yields promising results in understanding and mimicking target models, producing high-quality, high-fidelity, and distinct explanations. Our code is available at https://github.com/microsoft/RecAI.

Jing Yao, Xiaoyuan Yi, Yifan Gong, Xiting Wang, Xing Xie. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

The rapid progress in Large Language Models (LLMs) poses potential risks such as generating unethical content. Assessing the values embedded in LLMs' generated responses can help expose their misalignment, but this relies on reference-free value evaluators, e.g. fine-tuned LLMs or closed-source models like GPT-4. Nevertheless, two key challenges emerge in open-ended value evaluation: the evaluator should adapt to changing human value definitions with minimal annotation, against their own bias (adaptability); and remain robust across varying value expressions and scenarios (generalizability). To handle these challenges, we introduce CLAVE, a novel framework that integrates two complementary LLMs: a large model to extract high-level value concepts from diverse responses, leveraging its extensive knowledge and generalizability, and a small model fine-tuned on these concepts to adapt to human value annotations. This dual-model framework enables adaptation to any value system using <100 human-labeled samples per value type. We also present ValEval, a comprehensive dataset comprising 13k+ (text,value,label) tuples across diverse domains, covering three major value systems. We benchmark the performance of 15+ popular LLM evaluators and fully analyze their strengths and weaknesses. Our findings reveal that CLAVE combining a large prompt-based model and a small fine-tuned one serves as an optimal balance in value evaluation.

Document retrieval has been extensively studied within theindex-retrieveframework for decades, which has withstood the test of time. However, this approach inherently segregates the indexing and retrieval processes, preventing a cohesive, end-to-end optimization. To bridge this divide, we introduce WebUltron, a revolutionary model-centric indexer for document retrieval. This system embeds the entirety of document knowledge within the model, striving for seamless end-to-end retrieval. Two primary challenges with this indexer are the representation of document identifiers (docids) and the model's training. Current methods grapple with docids that lack semantic depth and the constraints of limited supervised data, making scaling up to larger datasets challenging. Addressing this, we've engineered two novel docid types imbued with richer semantics that also streamline model inference. Further enhancing WebUltron's capabilities, we've developed a three-stage training regimen, leveraging deeper corpus insights and fortifying query-docid relationships. Experiments on two public datasets demonstrate the superiority of WebUltron over advanced baselines for document retrieval.

In recent years, various dense retrieval methods have been developed to improve the performance of search engines with a vectorized index. However, these approaches require a large pre-computed index and have a limited capacity to memorize all semantics in a document within a single vector. To address these issues, researchers have explored end-to-end generative retrieval models that use a seq-to-seq generative model to directly return identifiers of relevant documents. Although these models have been effective, they are often trained with the MLE method. It only encourages the model to assign a high probability to the relevant document identifier, ignoring the relevance comparisons of other documents. This may lead to performance degradation in ranking tasks, where the core is to compare the relevance between documents. To address this issue, we propose a ranking-oriented generative retrieval model that incorporates relevance signals to better estimate the relative relevance of different documents in ranking tasks. Based upon the analysis of the optimization objectives of dense retrieval and generative retrieval, we propose utilizing dense retrieval to provide relevance feedback for generative retrieval. Under an alternate training framework, the generative retrieval model gradually acquires higher-quality ranking signals to optimize the model. Experimental results show that our approach increasing Recall@1 by 12.9% with respect to the baselines on MS MARCO dataset.

Inverted file structure is a common technique for accelerating dense retrieval.It clusters documents based on their embeddings; during searching, it probes nearby clusters w.r.t. an input query and only evaluates documents within them by subsequent codecs, thus avoiding the expensive cost of exhaustive traversal.However, the clustering is always lossy, which results in the miss of relevant documents in the probed clusters and hence degrades retrieval quality.In contrast, lexical matching, such as overlaps of salient terms, tends to be strong feature for identifying relevant documents.In this work, we present the Hybrid Inverted Index (HI 2 ), where the embedding clusters and salient terms work collaboratively to accelerate dense retrieval.To make best of both effectiveness and efficiency, we devise a cluster selector and a term selector, to construct compact inverted lists and efficiently searching through them.Moreover, we leverage simple unsupervised algorithms as well as end-to-end knowledge distillation to learn these two modules, with the latter further boosting the effectiveness.Based on comprehensive experiments on popular retrieval benchmarks, we verify that clusters and terms indeed complement each other, enabling HI 2 to achieve lossless retrieval quality with competitive efficiency across various index settings.

Active learning is effective for tasks with limited labeled data by annotating a small set of data actively. It utilizes the current trained model to evaluate all unlabeled samples and annotates the best samples scored by a specific query strategy to update the underlying model iteratively. Most active learning approaches rely on only the current evaluation score but ignore the results from previous iterations. In this paper, we propose using more historical evaluation results which can provide additional information to help better select samples. First, we apply two heuristic features of the historical evaluation results, i.e. the weighted sum and the fluctuation of history sequences. Next, to make fuller use of the information contained in the historical results, we design a query strategy that learns to select samples based on the history sequence automatically. Our proposed idea is general and can be combined with both basic and state-of-the-art query strategies to achieve improvements. Experimental results show that our methods significantly promote existing methods.

News recommendation is essential for personalized news distribution. Federated news recommendation, which enables collaborative model learning from multiple clients without sharing their raw data, is a promising approach for preserving users' privacy. However, the security of federated news recommendation is still unclear. In this paper, we study this problem by proposing an untargeted attack on federated news recommendation called UA-FedRec. By exploiting the prior knowledge of news recommendation and federated learning, UA-FedRec can effectively degrade the model performance with a small percentage of malicious clients. First, the effectiveness of news recommendation highly depends on user modeling and news modeling. We design a news similarity perturbation method to make representations of similar news farther and those of dissimilar news closer to interrupt news modeling, and propose a user model perturbation method to make malicious user updates in opposite directions of benign updates to interrupt user modeling. Second, updates from different clients are typically aggregated with a weighted average based on their sample sizes. We propose a quantity perturbation method to enlarge sample sizes of malicious clients in a reasonable range to amplify the impact of malicious updates. Extensive experiments on two real-world datasets show that UA-FedRec can effectively degrade the accuracy of existing federated news recommendation methods, even when defense is applied. Our study reveals a critical security issue in existing federated news recommendation systems and calls for research efforts to address the issue. Our code is available at https://github.com/yjw1029/UA-FedRec.

Collaborative Filtering (CF) is a widely used and effective technique for recommender systems. In recent decades, there have been significant advancements in latent embedding-based CF methods for improved accuracy, such as matrix factorization, neural collaborative filtering, and LightGCN. However, the explainability of these models has not been fully explored. Adding explainability to recommendation models can not only increase trust in the decision-making process, but also have multiple benefits such as providing persuasive explanations for item recommendations, creating explicit profiles for users and items, and assisting item producers in design improvements.

The key to personalized search is modeling user intents to tailor returned results for different users. Existing personalized methods mainly focus on learning implicit user interest vectors. In this paper, we propose ExpliPS, a personalized search model that explicitly incorporates query subtopics into personalization. It models the user’s current intent by estimating the user’s preference over the subtopics of the current query and personalizes the results over the weighted subtopics. We think that in such a way, personalized search could be more explainable and stable. Specifically, we first employ a semantic encoder to learn the representations of the user’s historical behaviours. Then with the historical behaviour representations, a subtopic preference encoder is devised to predict the user’s subtopic preferences on the current query. Finally, we rerank the candidates via a subtopic-aware ranker that prioritizes the documents relevant to the user-preferred subtopics. Experimental results show our model ExpliPS outperforms the state-of-the-art personalized web search models with explainable and stable results.

Deep semantic matching aims at discriminating the relationship between documents based on deep neural networks. In recent years, it becomes increasingly popular to organize documents with a graph structure, then leverage both the intrinsic document features and the extrinsic neighbor features to derive discrimination. Most of the existing works mainly care about how to utilize the presented neighbors, whereas limited effort is made to filter appropriate neighbors. We argue that the neighbor features could be highly noisy and partially useful. Thus, a lack of effective neighbor selection will not only incur a great deal of unnecessary computation cost but also restrict the matching accuracy severely. In this work, we propose a novel framework, C ascaded D eep S emantic M atching ( CDSM ), for accurate and efficient semantic matching on textual graphs. CDSM is highlighted for its two-stage workflow. In the first stage, a lightweight CNN-based ad-hod neighbor selector is deployed to filter useful neighbors for the matching task with a small computation cost. We design both one-step and multi-step selection methods. In the second stage, a high-capacity graph-based matching network is employed to compute fine-grained relevance scores based on the well-selected neighbors. It is worth noting that CDSM is a generic framework which accommodates most of the mainstream graph-based semantic matching networks. The major challenge is how the selector can learn to discriminate the neighbors’ usefulness which has no explicit labels. To cope with this problem, we design a weak-supervision strategy for optimization, where we train the graph-based matching network at first and then the ad-hoc neighbor selector is learned on top of the annotations from the matching network. We conduct extensive experiments with three large-scale datasets, showing that CDSM notably improves the semantic matching accuracy and efficiency thanks to the selection of high-quality neighbors. The source code is released at https://github.com/jingjyyao/CDSM.

Active learning is an effective approach for tasks with limited labeled data. It samples a small set of data to annotate actively and is widely applied in various AI tasks. It uses an iterative process, during which we utilize the current trained model to evaluate all unlabeled samples and annotate the best samples based on a specific query strategy to update the underlying model iteratively. Most existing active learning approaches rely on only the evaluation results generated by the current model and ignore the results from previous iterations. In this paper, we propose using more historical evaluation results which can provide additional information to help better select samples. First, we apply two kinds of heuristic features of the historical evaluation results, the weighted sum of historical results and the fluctuation of the historical evaluation sequence, to improve the effectiveness of active learning sampling. Next, to further and more globally use the information contained in the historical results, we design a novel query strategy that learns how to select samples based on the historical sequences automatically. Our proposed idea is general and can be combined with both basic and state-of-the-art query strategies to achieve improvements. We test our approaches on two common NLP tasks including text classification and named entity recognition. Experimental results show that our methods significantly promote existing methods.

Personalized search tailors document ranking lists for each individual user based on her interests and query intent to better satisfy the user’s information need. Many personalized search models have been proposed. They first build a user interest profile from the user’s search history, and then re-rank the documents based on the personalized matching scores between the created profile and candidate documents. In this article, we attempt to solve the personalized search problem from an alternative perspective of clarifying the user’s intention of the current query. We know that there are many ambiguous words in natural language such as “Apple.” People with different knowledge backgrounds and interests have personalized understandings of these words. Therefore, we propose a personalized search model with personal word embeddings for each individual user that mainly contain the word meanings that the user already knows and can reflect the user interests. To learn great personal word embeddings, we design a pre-training model that captures both the textual information of the query log and the information about user interests contained in the click-through data represented as a graph structure. With personal word embeddings, we obtain the personalized word and context-aware representations of the query and documents. Furthermore, we also employ the current session as the short-term search context to dynamically disambiguate the current query. Finally, we use a matching model to calculate the matching score between the personalized query and document representations for ranking. Experimental results on two large-scale query logs show that our designed model significantly outperforms state-of-the-art personalization models.

Search and recommendation are the two most common approaches used by people to obtain information. They share the same goal -- satisfying the user's information need at the right time. There are already a lot of Internet platforms and Apps providing both search and recommendation services, showing us the demand and opportunity to simultaneously handle both tasks. However, most platforms consider these two tasks independently -- they tend to train separate search model and recommendation model, without exploiting the relatedness and dependency between them. In this paper, we argue that jointly modeling these two tasks will benefit both of them and finally improve overall user satisfaction. We investigate the interactions between these two tasks in the specific information content service domain. We propose first integrating the user's behaviors in search and recommendation into a heterogeneous behavior sequence, then utilizing a joint model for handling both tasks based on the unified sequence. More specifically, we design the Unified Information SEarch and Recommendation model (USER), which mines user interests from the integrated sequence and accomplish the two tasks in a unified way. Experiments on a dataset from a real-world information content service platform verify that our model outperforms separate search and recommendation baselines.

Personalized search returns each user more accurate results by collecting the user’s historical search behaviors to infer her interests and query intents. However, it brings the risk of user privacy leakage, and this may greatly limit the practical application of personalized search. In this paper, we focus on the problem of privacy protection in personalized search, and propose a privacy protection enhanced personalized search framework, denoted with FedPS. Under this framework, we keep each user’s private data on her individual client, and train a shared personalized ranking model with all users’ decentralized data by means of federated learning. We implement two models within the framework: the first one applies a personalization model with a personal module that fits the user’s data distribution to alleviate the challenge of data heterogeneity in federated learning; the second model introduces trustworthy proxies and group servers to solve the problems of limited communication, performance bottleneck and privacy attack for FedPS. Experimental results verify that our proposed framework can enhance privacy protection without losing too much accuracy.

Personalized search is a promising way to improve search qualities by taking user interests into consideration. Recently, machine learning and deep learning techniques have been successfully applied to search result personalization. Most existing models simply regard the personal search history as a static set of user behaviors and learn fixed ranking strategies based on all the recorded data. Though improvements have been achieved, the essence that the search process is a sequence of interactions between the search engine and user is ignored. The user’s interests may dynamically change during the search process, therefore, it would be more helpful if a personalized search model could track the whole interaction process and adjust its ranking strategy continuously. In this article, we adapt reinforcement learning to personalized search and propose a framework, referred to as RLPS. It utilizes a Markov Decision Process ( MDP ) to track sequential interactions between the user and search engine, and continuously update the underlying personalized ranking model with the user’s real-time feedback to learn the user’s dynamic interests. Within this framework, we implement two models: the listwise RLPS-L and the hierarchical RLPS-H. RLPS-L interacts with users and trains the ranking model with document lists, while RLPS-H improves model training by designing a layered structure and introducing document pairs. In addition, we also design a feedback-aware personalized ranking component to capture the user’s feedback, which impacts the user interest profile for the next query. Significant improvements over existing personalized search models are observed in the experiments on the public AOL search log and a commercial log.

Personalized search is a task to tailor the general document ranking list based on user interests to better satisfy the user's information need. Many personalized search models have been proposed and demonstrated their capability to improve search quality. The general idea of most approaches is to build a user interest profile according to the user's search history, and then re-rank the documents based on the matching scores between the created user profile and candidate documents. In this paper, we propose to solve the problem of personalized search in an alternative way. We know that there are many ambiguous words in natural language such as 'Apple', and people with different knowledge backgrounds and interests have personalized understandings of these words. Therefore, for different users, such a word should own different semantic representations. Motivated by this idea, we design a personalized search model based on personal word embeddings, referred to as PEPS. Specifically, we train personal word embeddings for each user in which the representation of each word is mainly decided by the user's personal data. Then, we obtain the personalized word and contextual representations of the query and documents with an attention function. Finally, we use a matching model to calculate the matching score between the personalized query and document representations. Experiments on two datasets verify that our model can significantly improve state-of-the-art personalization models.

Personalized search improves generic ranking models by taking user interests into consideration and returning more accurate search results to individual users. In recent years, machine learning and deep learning techniques have been successfully applied in personalized search. Most existing personalization models simply regard the search history as a static set of user behaviours and learn fixed ranking strategies based on the recorded data. Though improvements have been observed, it is obvious that these methods ignore the dynamic nature of the search process: search is a sequence of interactions between the search engine and the user. During the search process, the user interests may dynamically change. It would be more helpful if a personalized search model could track the whole interaction process and update its ranking strategy continuously. In this paper, we propose a reinforcement learning based personalization model, referred to as RLPer, to track the sequential interactions between the users and search engine with a hierarchical Markov Decision Process (MDP). In RLPer, the search engine interacts with the user to update the underlying ranking model continuously with real-time feedback. And we design a feedback-aware personalized ranking component to catch the user’s feedback which has impacts on the user interest profile for the next query. Experimental results on the publicly available AOL search log verify that our proposed model can significantly outperform state-of-the-art personalized search models.