IPQA: A Benchmark for Core Intent Identification in Personalized Question Answering
Abstract
Intent identification serves as the foundation for generating appropriate responses in personalized question answering (PQA). However, existing benchmarks evaluate only response quality or retrieval performance without directly measuring intent identification capabilities. This gap is critical because without understanding which intents users prioritize, systems cannot generate responses satisfying individual information needs. To address this, we introduce the concept of core intents: intents users prioritize when selecting answers to satisfy their information needs. To evaluate these core intents, we propose IPQA, a benchmark for core Intent identification in Personalized Question Answering. Since users do not explicitly state their prioritized intents, we derive core intents from observable behavior patterns in answer selection, grounded in satisficing theory where users choose answers meeting their acceptance thresholds. We construct a dataset with various domains through systematic filtering, LLM-based annotation, and rigorous quality control combining automated verification with human validation. Experimental evaluations across state-of-the-art language models reveal that current systems struggle with core intent identification in personalized contexts. Models fail to identify core intents from user histories, with performance degrading as question complexity increases.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Idan Szpektor | 57 | 72 On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method (2023) Conversational Search and Retrieval 71 Will My Question Be Answered? Predicting "Question Answerability" in Community Question-Answering Sites (2013) Retrieval-Augmented and Open-Domain QA 71 Learning from the past: answering new questions with past answers (2012) Retrieval-Augmented and Open-Domain QA |
|
| 2 | Jinheon Baek | 45 | 72 Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement (2023) Conversational Search and Retrieval 70 Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks (2023) Retrieval-Augmented and Open-Domain QA 58 Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion (2024) Conversational Search and Retrieval |
Area chair
| Area chair | Match | |
|---|---|---|
| Pengyue Jia | 46 |
Area-chair candidates (6)
| Pengyue Jia assigned | 46 | 10 papers | |
| Jun Xu 0001 | 45 | 2 papers | |
| Xiaoxue Zang | 44 | 4 papers | |
| Kam-Fai Wong | 42 | 6 papers | |
| Xiaoxi Li 0005 | 42 | 15 papers | |
| Zhijing Wu 0001 | 41 | 9 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Idan Szpektor assigned | 57 | 3 | 72 On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method (2023) Conversational Search and Retrieval 71 Will My Question Be Answered? Predicting "Question Answerability" in Community Question-Answering Sites (2013) Retrieval-Augmented and Open-Domain QA |
|
| 2 | Jing Liu 0022 | 50 | 3 | 71 A Regularized Competition Model for Question Difficulty Estimation in Community Question Answering Services (2014) Retrieval-Augmented and Open-Domain QA 67 Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation (2025) Knowledge Conflict and Faithfulness |
|
| 3 | Farzad Shami | 49 | 3 | 71 From Single to Multi-agent Reasoning: Advancing GeneGPT for Genomics QA (2026) Retrieval-Augmented and Open-Domain QA 37 Benchmarking Large Language Models for Knowledge Graph Validation (2026) KG-Augmented LLM Reasoning |
|
| 4 | Yang Zhang 0072 | 48 | 3 | 53 Don't Start Over: A Cost-Effective Framework for Migrating Personalized Prompts Between LLMs (2026) Efficiency and Deployment 51 Preliminary Study on Incremental Learning for Large Language Model-based Recommender Systems (2024) Efficiency and Deployment |
|
| 5 | Chen Zhao 0013 | 47 | 3 | 71 A dataset and baselines for sequential open-domain question answering (2018) Retrieval-Augmented and Open-Domain QA 59 Distantly-Supervised Dense Retrieval Enables Open-Domain Question Answering without Evidence Annotation (2021) Dense Bi-Encoder Training and Representation Learning |
|
| 6 | Adam Jatowt | 47 | 3 | 69 ArchivalQA: A Large-scale Benchmark Dataset for Open-Domain Question Answering over Historical News Collections (2022) Multi-hop and Complex Reasoning QA 46 Inferential Question Answering (2026) Multi-hop and Complex Reasoning QA |
|
| 7 | Bo Chen 0023 | 46 | 3 | 52 How Can Recommender Systems Benefit from Large Language Models: A Survey (2025) Efficiency and Deployment 52 Large language models make sample-efficient recommender systems (2025) Efficiency and Deployment |
|
| 9 | Yongqi Li 0001 | 45 | 3 | 54 EARN: Efficient Inference Acceleration for LLM-based Generative Recommendation by Register Tokens (2025) Generative Retrieval and Ranking 54 Efficient Inference for Large Language Model-based Generative Recommendation (2025) Generative Retrieval and Ranking |
|
| 11 | Mariano Barone | 45 | 3 | 67 A Multilingual Multimodal Medical Examination Dataset for Visual Question Answering in Healthcare (2025) Retrieval-Augmented and Open-Domain QA 66 SOLVE-Med: Specialized Orchestration for Leading Vertical Experts Across Medical Specialties (2025) Retrieval-Augmented and Open-Domain QA |
|
| 12 | Wenlin Zhang 0001 | 45 | 3 | 57 Personalized Deep Research: A User-Centric Framework, Dataset, and Hybrid Evaluation for Knowledge Discovery (2026) Deep Research and Report Generation 55 Bridging Personalization and AI: From RAG to Agent (2026) Personalized and Conversational RAG |
|
| 13 | Jinheon Baek assigned | 45 | 3 | 72 Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement (2023) Conversational Search and Retrieval 70 Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks (2023) Retrieval-Augmented and Open-Domain QA |
|
| 14 | Qinggang Zhang | 45 | 3 | 50 Self-Monitoring Large Language Models for Click-Through Rate Prediction (2026) Efficiency and Deployment 49 LLM Collaborative Filtering: User-Item Graph as New Language (2026) Efficiency and Deployment |
|
| 15 | Falk Scholer | 45 | 3 | 72 Document Summarization for Answering Non-Factoid Queries (2018) Retrieval-Augmented and Open-Domain QA 72 Ranking Documents by Answer-Passage Quality (2018) Retrieval-Augmented and Open-Domain QA |
|
| 16 | Junjie Zhang 0009 | 44 | 3 | 51 Tapping the Potential of Large Language Models as Recommender Systems: A Comprehensive Framework and Empirical Analysis (2025) Efficiency and Deployment 51 Large Language Models are Zero-Shot Rankers for Recommender Systems (2024) Efficiency and Deployment |
|
| 17 | Kaixin Ma | 44 | 3 | 71 Chain-of-Skills: A Configurable Model for Open-Domain Question Answering (2023) Retrieval-Augmented and Open-Domain QA 70 Open Domain Question Answering with A Unified Knowledge Interface (2022) Multimodal and Graph RAG |
|
| 18 | Jialong Wu 0007 | 44 | 3 | 53 PROPER: A Progressive Learning Framework for Personalized Large Language Models with Group-Level Adaptation (2025) Efficiency and Deployment 43 AdaRewriter: Unleashing the Power of Prompting-based Conversational Query Reformulation via Test-Time Adaptation (2025) Query Reformulation and Expansion |
|
| 19 | Jiapu Wang | 44 | 3 | 64 Large-Small Model Synergy with Multimodal Fine-Grained Heuristics for Knowledge-Based Visual Question Answering (2025) Retrieval-Augmented and Open-Domain QA 58 Decomposing and Revising What Language Models Generate (2025) Multi-Hop and Iterative Retrieval Reasoning |
|
| 20 | Liwei Wang 0009 | 44 | 3 | 71 MVP-Tuning: Multi-View Knowledge Retrieval with Prompt Tuning for Commonsense Reasoning (2023) Retrieval-Augmented and Open-Domain QA 46 Making Long-Context Language Models Better Multi-Hop Reasoners (2024) Multi-hop and Complex Reasoning QA |
|
| 22 | Wenyu Huang | 44 | 3 | 56 Prompting large language models with knowledge graphs for question answering involving long-tail facts (2025) KG-Augmented LLM Reasoning 46 Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025) Multi-hop and Complex Reasoning QA |
|
| 23 | Yujia Zhou 0002 | 43 | 3 | 68 RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation (2025) Evidence Selection and Context Refinement 46 Enhancing Generative Retrieval with Reinforcement Learning from Relevance Feedback (2023) Recommendation with Search and Ranking |
|
| 24 | Weiwei Sun 0001 | 43 | 3 | 45 Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023) Reasoning-Intensive Retrieval and Ranking 44 TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy (2025) Reasoning-Intensive Retrieval and Ranking |
|
| 25 | Huifeng Guo | 43 | 3 | 55 A Survey of Personalization: From RAG to Agent (2026) Agentic and Deep Search RAG 55 Personalize Before Retrieve: LLM-based Personalized Query Expansion for User-Centric Retrieval (2026) Personalized and Conversational RAG |
|
| 26 | Han Li 0005 | 43 | 3 | 56 Retrieval Augmented Generation with Collaborative Filtering for Personalized Text Generation (2025) Personalized and Conversational RAG 51 LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application (2025) Efficiency and Deployment |
|
| 27 | Jizhi Zhang | 42 | 3 | 51 Navigating Large Language Models for Recommendation: From Architecture to Learning Paradigms and Deployment (2025) Efficiency and Deployment 51 Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation (2023) Preference Alignment and Optimization |
|
| 28 | Gustavo Penha | 42 | 3 | 49 Evaluating Podcast Recommendations with Profile-Aware LLM-as-a-Judge (2025) Recommendation Systems and Evaluation 44 Curriculum Learning Strategies for IR (2020) Ranking Distillation and Adaptation |
|
| 30 | Yingyi Zhang 0001 | 42 | 3 | 57 Personalized Deep Research: A User-Centric Framework, Dataset, and Hybrid Evaluation for Knowledge Discovery (2026) Deep Research and Report Generation 55 Bridging Personalization and AI: From RAG to Agent (2026) Personalized and Conversational RAG |
|
| 32 | Shu Zhou 0002 | 42 | 3 | 72 LOSDF: A logical optimization and semantic decoupling framework for question answering in multi-party conversations (2025) Conversational Search and Retrieval 60 Enhancing Extractive Question Answering in Multiparty Dialogues with Logical Inference Memory Network (2025) Conversational Search and Retrieval |
|
| 33 | Yuchen Li 0006 | 42 | 3 | 45 S2phere: Semi-Supervised Pre-training for Web Search over Heterogeneous Learning to Rank Data (2023) Web-Scale Learning to Rank 43 ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (2026) LLM-Based Reranking |
|
| 34 | Xinyu Ma 0001 | 41 | 3 | 46 Knowledge Graph Retrieval-Augmented Generation for LLM-based Recommendation (2025) LLM-Derived Knowledge Infusion 45 Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023) Reasoning-Intensive Retrieval and Ranking |
|
| 35 | Hao Liu 0078 | 41 | 3 | 51 WeaveRec: An LLM-Based Cross-Domain Sequential Recommendation Framework with Model Merging (2026) Efficiency and Deployment 38 Mitigating Fine-tuning Bias: A Parameter-Efficient Debiasing Framework for Large Language Models (2026) Bias Mitigation Methods |