How Far Are We from Automatically Identifying Violations of the Data Minimization Principle in Privacy Policies?
sigir26-048 · Full research paper · Zhaozheng Zhou, Yanru He, Yunchuan Guo, Haoyang Yu, Liang Fang, Fenghua Li
Fairness Metrics and Evaluation · 100%
Abstract
Data protection laws and regulations require service providers to disclose data practices in privacy policies, specifying what personal information is processed and for what purposes. For compliance, these data practices must adhere to the data minimization principle, limiting the processing of personal information to what is directly relevant and necessary for the service purposes. However, data minimization is context-dependent, making violations difficult to define and quantify in privacy policies. Meanwhile, privacy policies are semantically complex and unstructured, hindering accurate extraction of fine-grained data practices and large-scale automated evaluation. To address these issues, we propose DataMini, a human--LLM collaborative evaluation framework for identifying violations of the data minimization principle in privacy policies. First, DataMini categorizes data minimization violations into two dimensions: inherent violations and contextual violations, establishing fine-grained evaluation criteria. Second, we construct a compliance baseline by mining high-frequency patterns from large-scale privacy policies and integrating expert knowledge to derive compliance mappings for human--LLM collaborative evaluation. Finally, the compliance baseline can automatically verify data practices that satisfy the data minimization principle, enabling the framework to focus exclusively on identifying suspected violations to improve efficiency and accuracy. Extensive evaluations demonstrate that DataMini exhibits superior data practice extraction accuracy of 83.46% and achieves an F1-score of 0.8180 for identifying data minimization violations in privacy policies, reducing manual evaluation effort by approximately 80%.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Theresia Veronika Rampisela | 100 | 100 Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study (2025) Fairness Metrics and Evaluation 100 Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance (2024) Fairness Metrics and Evaluation 100 Relevance-aware Individual Item Fairness Measures for Recommender Systems: Limitations and Usage Guidelines (2026) Fairness Metrics and Evaluation |
|
| 2 | Sruthi Gorantla | 63 | 100 Optimizing Learning-to-Rank Models for Ex-Post Fair Relevance (2024) Fairness Metrics and Evaluation 100 Sampling Individually-Fair Rankings that are Always Group Fair (2023) Fairness Metrics and Evaluation 100 Pairwise Sample Complexity for Fair Active Ranking with Cascaded Norm Objectives (2025) Fairness Metrics and Evaluation |
Area chair
| Area chair | Match | |
|---|---|---|
| Xin Xin 0003 | 33 |
Area-chair candidates (1)
| Xin Xin 0003 assigned | 33 | 10 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Theresia Veronika Rampisela assigned | 100 | 3 | 100 Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study (2025) Fairness Metrics and Evaluation 100 Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance (2024) Fairness Metrics and Evaluation |
|
| 2 | Sruthi Gorantla assigned | 63 | 3 | 100 Optimizing Learning-to-Rank Models for Ex-Post Fair Relevance (2024) Fairness Metrics and Evaluation 100 Sampling Individually-Fair Rankings that are Always Group Fair (2023) Fairness Metrics and Evaluation |
|
| 3 | Zechun Niu | 47 | 3 | 56 Addressing Personalized Bias for Unbiased Learning to Rank (2025) Unbiased Learning to Rank 43 Distributionally Robust Optimization for Unbiased Learning to Rank (2025) Unbiased Learning to Rank |
|
| 4 | Yimeng Bai | 43 | 3 | 62 Recommendation Unlearning via Influence Function (2025) Efficient Training and Data Optimization 42 Leveraging Watch-time Feedback for Short-Video Recommendations: A Causal Labeling Framework (2023) Recommendation Systems and Evaluation |
|
| 5 | Yifan Wang 0017 | 40 | 3 | 68 A Survey on the Fairness of Recommender Systems (2023) Popularity Bias and Long-Tail Debiasing 41 Popularity-Aware Alignment and Contrast for Mitigating Popularity Bias (2024) Popularity Bias and Debiasing in Recommendation |
|
| 6 | Haritz Puerto | 39 | 3 | 100 Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers (2025) Fairness Metrics and Evaluation 28 C-SEO Bench: Does Conversational SEO Work? (2025) Reasoning Evaluation and Benchmarks |
|
| 7 | Leif Azzopardi | 39 | 3 | 70 Efficiently Estimating Retrievability Bias (2014) Evaluation Metrics and User Models 70 Measuring Bias in a Ranked List Using Term-Based Representations (2024) Evaluation Metrics and User Models |
|
| 8 | Jizhi Zhang | 39 | 3 | 57 Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation (2023) Preference Alignment and Optimization 43 Item-side Fairness of Large Language Model-based Recommendation System (2024) Preference Alignment and Optimization |
|
| 9 | Chongming Gao | 38 | 3 | 43 SPRec: Self-Play to Debias LLM-based Recommendation (2025) Preference Alignment and Optimization 43 Process-Supervised LLM Recommenders via Flow-guided Tuning (2025) Preference Alignment and Optimization |
|
| 10 | Xialong Liu | 38 | 3 | 100 Fairness-Aware Design for Contextual Experiments: Guaranteeing Reliability and Equity in Heterogeneous Subgroups (2026) Fairness Metrics and Evaluation 41 CROLoss: Towards a Customizable Loss for Retrieval Models in Recommender Systems (2022) Recommendation with Search and Ranking |
|
| 11 | Andrew Parry | 37 | 3 | 67 Large Language Model Relevance Assessors Agree With One Another More Than With Human Assessors (2025) LLM-Based Relevance Assessment 41 [inline-graphic not available: see fulltext] SuiteEval: Simplifying Retrieval Benchmarks (2026) Retrieval Toolkits and Reproducibility |
|
| 12 | Yang Zhang 0072 | 37 | 3 | 71 Exact and Efficient Unlearning for Large Language Model-Based Recommendation (2025) Efficiency and Deployment 42 Preliminary Study on Incremental Learning for Large Language Model-based Recommender Systems (2024) Efficiency and Deployment |
|
| 13 | Zhengyi Yang 0007 | 37 | 3 | 43 Item-side Fairness of Large Language Model-based Recommendation System (2024) Preference Alignment and Optimization 41 Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (2025) Preference Alignment and Optimization |
|
| 14 | Min Hou 0004 | 37 | 3 | 100 Fair Personalized Learner Modeling Without Sensitive Attributes (2025) Fairness Metrics and Evaluation 46 InvDiff: Invariant Guidance for Bias Mitigation in Diffusion Models (2025) Popularity Bias and Debiasing in Recommendation |
|
| 15 | Sunhao Dai | 37 | 3 | 45 Unifying Bias and Unfairness in Information Retrieval: New Challenges in the LLM Era (2025) Bias Mitigation Methods 45 Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era (2024) Bias Mitigation Methods |
|
| 16 | Seongmin Park 0002 | 35 | 3 | 42 Dual Unbiased Recommender Learning for Implicit Feedback (2021) Popularity Bias and Debiasing in Recommendation 42 Bilateral Self-unbiased Learning from Biased Implicit Feedback (2022) Popularity Bias and Debiasing in Recommendation |
|
| 17 | Paul Thomas 0001 | 35 | 3 | 67 On the Use of LLMs for Relevance Labelling (2026) LLM-Based Relevance Assessment 40 cwl_eval: An Evaluation Tool for Information Retrieval (2019) Retrieval Toolkits and Reproducibility |
|
| 18 | Yu-An Liu 0028 | 35 | 3 | 40 Robust Information Retrieval (2025) Robustness and Adversarial IR 40 Robust Information Retrieval (2024) Robustness and Adversarial IR |
|
| 19 | Hao Liu 0078 | 35 | 3 | 45 Mitigating Fine-tuning Bias: A Parameter-Efficient Debiasing Framework for Large Language Models (2026) Bias Mitigation Methods 34 Mitigating Distribution Shifts in Sequential Recommendation: An Invariance Perspective (2025) Contrastive and Self-Supervised Sequential Recommendation |
|
| 20 | Julian J. McAuley | 34 | 3 | 63 The 1st International Workshop on Risks, Opportunities, and Evaluation of Generative Models in Recommendation (ROEGEN) (2024) Generative Models for Recommendation and Content › Others 63 Toward Holistic Evaluation of Recommender Systems Powered by Generative Models (2025) Generative Models for Recommendation and Content › Others |
|
| 21 | Tetsuya Sakai | 34 | 3 | 40 Diversification as Risk Minimization (2026) Evaluation Metrics and User Models 40 Summaries, ranked retrieval and sessions: a unified framework for information access evaluation (2013) Evaluation Metrics and User Models |
|
| 22 | Hyunsik Yoo | 34 | 3 | 68 Ensuring User-side Fairness in Dynamic Recommender Systems (2024) Popularity Bias and Long-Tail Debiasing 42 Continual Recommender Systems (2025) Recommendation Systems and Evaluation |
|
| 23 | Rolf Jagerman | 34 | 3 | 42 Bootstrapping Recommendations at Chrome Web Store (2021) Recommendation Systems and Evaluation 40 Safe Exploration for Optimizing Contextual Bandits (2020) Unbiased Learning to Rank |
|
| 24 | Bo Chen 0023 | 34 | 3 | 71 Towards efficient and effective unlearning of large language models for recommendation (2025) Efficiency and Deployment 42 A Comprehensive Survey on Automated Machine Learning for Recommendations (2024) Recommendation Systems and Evaluation |
|
| 25 | Keqin Bao | 34 | 3 | 68 Fair Recommendation with Biased-Limited Sensitive Attribute (2025) Popularity Bias and Long-Tail Debiasing 57 Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation (2023) Preference Alignment and Optimization |
|
| 26 | Zhen Qin 0001 | 34 | 3 | 42 Bootstrapping Recommendations at Chrome Web Store (2021) Recommendation Systems and Evaluation 42 Improving Recommendation Quality in Google Drive (2020) Recommendation Systems and Evaluation |
|
| 27 | Yuchen Li 0006 | 34 | 3 | 39 RankExpert: A Mixture of Textual-and-Behavioral Experts for Multi-Objective Learning-to-Rank in Web Search (2025) Unbiased Learning to Rank 38 FULTR: A Large-Scale Fusion Learning to Rank Dataset and Its Application for Satisfaction-Oriented Ranking (2025) Web-Scale Learning to Rank |
|
| 28 | Jin Chen 0008 | 34 | 3 | 42 RecStudio: Towards a Highly-Modularized Recommender System (2023) Recommendation Systems and Evaluation 40 Understanding the Effect of Loss Functions on the Generalization of Recommendations (2025) Recommendation Systems and Evaluation |
|
| 29 | Xiao Wang 0043 | 34 | 3 | 41 Retrieve, Rerank, Answer, Experiment: Hands-On IR Research with PyTerrier (2026) Retrieval Toolkits and Reproducibility 33 Effective Contrastive Weighting for Dense Query Expansion (2023) Query Reformulation and Expansion |
|
| 30 | Xuanhui Wang | 34 | 3 | 39 TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank (2019) Retrieval Toolkits and Reproducibility 39 Rax: Composable Learning-to-Rank Using JAX (2022) Retrieval Toolkits and Reproducibility |