Correct, comparable, and reproducible model evaluation is essential for progress in machine learning.Over twenty years, thousands of language and vision models have been evaluated with a popular metric called ROUGE.Does this widespread benchmark metric meet these three evaluation criteria?This systematic review of over two thousand publications using ROUGE finds: (A) Critical evaluation decisions and parameters are routinely omitted, making most reported scores irreproducible.(B) Differences in evaluation protocol are common, affect scores, and impact the comparability of results reported in many papers.(C) Thousands of papers use nonstandard evaluation packages with software defects that produce provably incorrect scores.Estimating the overall impact of these findings is difficult: because software citations are rare, it is nearly impossible to distinguish between correct ROUGE scores and incorrect "rogue scores." 1

Rich engagement data can shed light on how people interact with online content and how such interactions may be determined by the content of the page. In this work, we investigate a specific type of interaction, backtracking, which refers to the action of scrolling back in a browser while reading an online news article. We leverage a dataset of close to 700K instances of more than 15K readers interacting with online news articles, in order to characterize and predict backtracking behavior. We first define different types of backtracking actions. We then show that “full” backtracks, where the readers eventually return to the spot at which they left the text, can be predicted by using features that were previously shown to relate to text readability. This finding highlights the relationship between backtracking and readability and suggests that backtracking could help assess readability of content at scale.

Max Grusky, Mor Naaman, Yoav Artzi. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

Website measures of engagement captured from millions of users, such as in-page scrolling and viewport position, can provide deeper understanding of attention than possible with simpler measures, such as dwell time. Using data from 1.2M news reading sessions, we examine and evaluate three increasingly sophisticated models of sub-document attention computed from viewport time, the time a page component is visible on the user display. Our modeling incorporates prior eye-tracking knowledge about onscreen reading, and we validate it by showing how, when used to estimate user reading rate, it aligns with known empirical measures. We then show how our models reveal an interaction between article topic and attention to page elements. Our approach supports refined large-scale measurement of user engagement at a level previously available only from lab-based eye-tracking studies.