Demo Data from SIGIR 2026 / reviewers in the wild / expert
Houmaan Chamani
dblp:317/3369
· DBLP profile ↗
1ranked-venue papers
0as first author
1since 2021 · last 2025
0009-0009-9725-4309ORCID · reported
Domains — the database's venue-derived domains; a paper can count in several
Databases, data management, data science and information retrieval · 1 · 1 since 2021
Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.
| Databases, data mining, and information retrieval
1 paper |
Information retrieval · 100% |
Topics — the 3 heaviest of 3, each with the papers that count most for it
| Topic | Weight | Papers | Last | Evidence papers |
|---|---|---|---|---|
Information retrieval
evaluation |
0.9 | 1 | 2025 | Extending MovieLens-32M to Provide New Evaluation Objectives · SIGIR 2025 |
Information retrieval › evaluation › test collection
test collection construction |
0.9 | 1 | 2025 | Extending MovieLens-32M to Provide New Evaluation Objectives · SIGIR 2025 |
Information retrieval › evaluation
offline evaluation |
0.3 | 1 | 2025 | Extending MovieLens-32M to Provide New Evaluation Objectives · SIGIR 2025 |
Methods — techniques the papers use, named apart from their topics
user assessment · 0.9pooling · 0.9
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2025 | Extending MovieLens-32M to Provide New Evaluation ObjectivesabstractOffline evaluation of recommender systems has traditionally treated the problem as a machine learning problem. In the classic case of recommending movies, where the user has provided explicit ratings of which movies they like and don't like, each user's ratings are split into test and train sets, and the evaluation task becomes to predict the held out test data using the training data. This machine learning style of evaluation makes the objective to recommend the movies that a user has watched and rated highly, which is not the same task as helping the user find movies that they would enjoy if they watched them. This mismatch in objective between evaluation and task is a compromise to avoid the cost of asking a user to evaluate recommendations by watching each movie. As a resource available for download, we offer an extension to the MovieLens-32M dataset that provides for new evaluation objectives. Our primary objective is to predict the movies that a user would be interested in watching, i.e. predict their watchlist. To construct this extension, we recruited MovieLens users, collected their profiles, made recommendations with a diverse set of algorithms, pooled the recommendations, and had the users assess the pools. This paper demonstrates the feasibility of using pooling to construct a test collection for recommender systems. Notably, we found that the traditional machine learning style of evaluation ranks the Popular algorithm, which recommends movies based on total number of ratings in the system, in the middle of the twenty-two recommendation runs we used to build the pools. In contrast, when we rank the runs by users' interest in watching movies, we find that recommending popular movies as a recommendation algorithm becomes one of the worst performing runs. It appears that by asking users to assess their personal recommendations, we can alleviate the issue of popularity bias in the evaluation of top-n recommendation. Mark D. Smucker, Houmaan Chamani |
SIGIR | 2 |