Following the Eye-Tracking Evidence: Established Web-Search Assumptions Fail in Carousel Interfaces
Abstract
Carousel interfaces have been the de-facto standard for streaming media services for over a decade. Yet, there has been very little research into user behavior with such interfaces, which thus remains poorly understood. Due to this lack of empirical research, previous work has assumed that behaviors established in single-list web-search interfaces, such as the F-pattern and the examination hypothesis, also apply to carousel interfaces, for instance when designing click models or evaluation metrics. We analyze a recently-released interaction and examination dataset resulting from an eye-tracking study performed on carousel interfaces to verify whether these assumptions actually hold. We find that (i)~the F-pattern holds only for vertical examination and not for horizontal swiping; additionally, we discover that, when conditioned on a click, user examination follows an L-pattern unique to carousel interfaces; (ii)~click-through-rates conditioned on examination indicate that the well-known examination hypothesis does not hold in carousel interfaces; and (iii)~contrary to the assumptions of previous work, users generally ignore carousel headings and focus directly on the content items. Our findings show that many user behavior assumptions, especially concerning examination patterns, do not transfer from web search interfaces to carousel recommendation settings. Our work shows that the field lacks a reliable foundation on which to build models of user behavior with these interfaces. Consequently, a re-evaluation of existing metrics and click models for carousel interfaces may be warranted.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Jiqun Liu | 81 | 100 The Paradox of Personalization: Does Task Prediction Require Individualized Models? (2018) User Behavior and Search Interaction 100 A Reference-Dependent Model for Web Search Evaluation: Understanding and Measuring the Experience of Boundedly Rational Users (2023) User Behavior and Search Interaction 100 Constructing and meta-evaluating state-aware evaluation metrics for interactive search systems (2023) User Behavior and Search Interaction |
|
| 2 | Fan Zhang 0053 | 59 | 100 Evaluating Web Search with a Bejeweled Player Model (2017) User Behavior and Search Interaction 100 Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics (2020) User Behavior and Search Interaction 100 Constructing Better Evaluation Metrics by Incorporating the Anchoring Effect into the User Model (2022) User Behavior and Search Interaction |
Area chair
| Area chair | Match | |
|---|---|---|
| Zhijing Wu 0001 | 34 |
Area-chair candidates (7)
| Ziyi Ye COI: co-author | 45 | 14 papers | |
| Zhijing Wu 0001 assigned | 34 | 9 papers | |
| Fernando Diaz 0001 COI: co-author | 27 | 12 papers | |
| Xiaoxi Li 0005 | 25 | 15 papers | |
| Jun Xu 0001 COI: co-author | 25 | 2 papers | |
| Chenyan Xiong | 24 | 15 papers | |
| Xiaoxue Zang | 23 | 4 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Jiqun Liu assigned | 81 | 3 | 100 The Paradox of Personalization: Does Task Prediction Require Individualized Models? (2018) User Behavior and Search Interaction 100 A Reference-Dependent Model for Web Search Evaluation: Understanding and Measuring the Experience of Boundedly Rational Users (2023) User Behavior and Search Interaction |
|
| 2 | Sara Allawati COI: co-author | 72 | 3 | 100 From Lab to Reality: An Eye-tracking Study of How Users are Influenced to Search in the Era of GenAI (2026) User Behavior and Search Interaction 100 What do Users Really Ask Large Language Models? An Initial Log Analysis of Google Bard Interactions in the Wild (2024) User Behavior and Search Interaction |
|
| 3 | Vuong Thanh Tung | 72 | 3 | 100 Associating cognitive abilities with naturalistic search behavior (2025) User Behavior and Search Interaction 100 Understanding user behavior in naturalistic information search tasks (2019) User Behavior and Search Interaction |
|
| 4 | Fan Zhang 0053 assigned | 59 | 3 | 100 Evaluating Web Search with a Bejeweled Player Model (2017) User Behavior and Search Interaction 100 Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics (2020) User Behavior and Search Interaction |
|
| 5 | Jaime Arguello | 47 | 3 | 100 Goal-setting in support of learning during search: An exploration of learning outcomes and searcher perceptions (2023) User Behavior and Search Interaction 100 Learning assessments in search-as-learning: A survey of prior work and opportunities for future research (2022) User Behavior and Search Interaction |
|
| 7 | Romain Deffayet COI: co-author | 43 | 3 | 100 Evaluating the Robustness of Click Models to Policy Distributional Shift (2023) User Behavior and Search Interaction 100 An Offline Metric for the Debiasedness of Click Models (2023) User Behavior and Search Interaction |
|
| 8 | Mark D. Smucker | 39 | 3 | 68 Time-based calibration of effectiveness measures (2012) User Behavior and Search Interaction 68 IR Evaluation: Modeling User Behavior for Measuring Effectiveness (2015) User Behavior and Search Interaction |
|
| 9 | Falk Scholer | 39 | 3 | 64 Can Users Predict Relative Query Effectiveness? (2022) User Behavior and Search Interaction 64 Classifying Term Variants in Query Formulation (2025) User Behavior and Search Interaction |
|
| 10 | Saber Zerhoudi | 37 | 3 | 72 From SERPs to Agents: A Platform for Comparative Studies of Information Interaction (2026) User Behavior and Search Interaction 72 Beyond the Click: A Framework for Inferring Cognitive Traces in Search (2026) User Behavior and Search Interaction |
|
| 11 | Xuanhui Wang COI: co-author | 36 | 3 | 100 Cross-Positional Attention for Debiasing Clicks (2021) User Behavior and Search Interaction 62 Revisiting Two-tower Models for Unbiased Learning to Rank (2022) User Behavior and Search Interaction |
|
| 12 | Paul Thomas 0001 | 36 | 3 | 100 Third Workshop on Building towards Information Interaction and Retrieval Resources Re-use (BIIRRR 2022) (2022) User Behavior and Search Interaction 68 User Models, Metrics and Measures of Search: A Tutorial on the C/W/L Evaluation Framework (2021) User Behavior and Search Interaction |
|
| 14 | Leif Azzopardi COI: co-author | 33 | 3 | 68 User Models, Metrics and Measures of Search: A Tutorial on the C/W/L Evaluation Framework (2021) User Behavior and Search Interaction 64 Query quality: user ratings and system predictions (2010) User Behavior and Search Interaction |
|
| 15 | Tetsuya Sakai | 30 | 3 | 36 Give me just one highly relevant document: P-measure (2006) Evaluation Metrics and User Models 36 Summaries, ranked retrieval and sessions: a unified framework for information access evaluation (2013) Evaluation Metrics and User Models |
|
| 16 | Shuaiqiang Wang COI: co-author | 29 | 3 | 61 RankExpert: A Mixture of Textual-and-Behavioral Experts for Multi-Objective Learning-to-Rank in Web Search (2025) User Behavior and Search Interaction 61 M2oERank: Multi-Objective Mixture-of-Experts Enhanced Ranking for Satisfaction-Oriented Web Search (2025) User Behavior and Search Interaction |
|
| 17 | Yujia Zhou 0002 | 29 | 3 | 62 Intent-Oriented Dynamic Interest Modeling for Personalized Web Search (2024) User Behavior and Search Interaction 62 Incorporating Explicit Subtopics in Personalized Search (2023) User Behavior and Search Interaction |
|
| 18 | Zhen Qin 0001 COI: co-author | 29 | 3 | 62 Cross-Positional Attention for Debiasing Clicks (2021) User Behavior and Search Interaction 62 Revisiting Two-tower Models for Unbiased Learning to Rank (2022) User Behavior and Search Interaction |
|
| 20 | Idan Szpektor | 27 | 3 | 100 Predicting web searcher satisfaction with existing community-based answers (2011) User Behavior and Search Interaction 33 Identifying Web Queries with Question Intent (2016) Personalized and Context-Aware Retrieval |
|
| 21 | Yuta Saito COI: co-author | 26 | 3 | 31 Long-term Off-Policy Evaluation and Learning (2024) Long-Term Engagement and Retention Optimization 31 Counterfactual Evaluation and Learning for Interactive Systems: Foundations, Implementations, and Recent Advances (2022) Long-Term Engagement and Retention Optimization |
|
| 22 | Gustavo Penha | 26 | 3 | 62 As It Was: Aligning LLM Search Evaluation with Historical User Preferences (2026) User Behavior and Search Interaction 29 Contextualizing Spotify's Audiobook List Recommendations with Descriptive Shelves (2025) Recommendation Systems and Evaluation |
|
| 23 | Bo Peng 0009 | 26 | 3 | 65 SWaT: Statistical Modeling of Video Watch Time through User Behavior Analysis (2025) User Behavior and Search Interaction 23 eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data (2024) Efficiency and Deployment |
|
| 24 | Xiao Zhang 0034 | 26 | 3 | 31 Counterfactual Reward Modification for Streaming Recommendation with Delayed Feedback (2021) Long-Term Engagement and Retention Optimization 31 HyperBandit: Contextual Bandit with Hypernewtork for Time-Varying User Preferences in Streaming Recommendation (2023) Long-Term Engagement and Retention Optimization |
|
| 25 | Negar Arabzadeh COI: co-author | 25 | 3 | 32 Query Performance Prediction: From Ad-hoc to Conversational Search (2023) Query Performance Prediction 28 Estimating Query Performance Through Rich Contextualized Query Representations (2024) Query Performance Prediction |
|
| 26 | Yifan Wang 0017 | 25 | 3 | 72 User Perception of Recommendation Explanation: Are Your Explanations What Users Need? (2023) User Behavior and Search Interaction 24 Unbiased Delayed Feedback Label Correction for Conversion Rate Prediction (2023) Advertising and Auction Mechanisms |
|
| 27 | Yong Li 0004 | 25 | 3 | 100 Towards an Inclusive Mobile Web: A Dataset and Framework for Focusability in UI Accessibility (2025) User Behavior and Search Interaction 26 ProBench: Benchmarking GUI Agents with Accurate Process Information (2026) Reasoning Evaluation and Benchmarks |
|
| 30 | Yang Song 0008 | 25 | 3 | 64 Query-Less: Predicting Task Repetition for NextGen Proactive Search and Recommendation Engines (2016) User Behavior and Search Interaction 30 Making Transformer Decoders Better Differentiable Indexers (2025) Generative Retrieval |
|
| 31 | Jialong Wu 0007 | 25 | 3 | 29 OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking (2025) Deep Research and Report Generation 28 Nested Browser-Use Learning for Agentic Information Seeking (2026) Deep Research and Report Generation |
|
| 32 | Tingyu Song | 24 | 3 | 61 IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval (2025) User Behavior and Search Interaction 20 Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems (2026) Reasoning-Intensive Retrieval and Ranking |
|
| 33 | Zechun Niu | 24 | 3 | 26 Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study (2025) Unbiased Learning to Rank 24 Addressing Personalized Bias for Unbiased Learning to Rank (2025) Unbiased Learning to Rank |
|
| 34 | Sunhao Dai | 24 | 3 | 72 NExT-Search: Rebuilding User Feedback Ecosystem for Generative AI Search (2025) User Behavior and Search Interaction 31 AdaO2B: Adaptive Online to Batch Conversion for Out-of-Distribution Generalization (2025) Long-Term Engagement and Retention Optimization |
|
| 35 | Theresia Veronika Rampisela | 24 | 3 | 24 Relevance-aware Individual Item Fairness Measures for Recommender Systems: Limitations and Usage Guidelines (2026) Fairness Metrics and Evaluation 24 Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study (2025) Fairness Metrics and Evaluation |