This study explores users' learning gains and experiences from reading within three versions of the same text. The versions included 1) a traditional digital text, 2) a productive text that required participants to produce knowledge about what they read, but without any feedback on the knowledge produced, and 3) an interactive text that required participants to produce knowledge and provided AI feedback and interaction. Learning gains were assessed in a randomized control trial where crowd-sourced users were assigned to one of the three versions and were examined based on the quality of constructed responses and summaries provided as well as through differences from a pre-test and a post-test. User experiences were investigated using survey results. Results indicated that users were generally satisfied with interacting with all versions the text, except the summary portion of the interactive text. Conversely, results indicated that users of the interactive text consistently wrote better summaries than in the productive condition, and they revised summaries to a greater degree and to a greater success. Lastly, results showed that knowledge gains occurred in all reading conditions and that readers in the interactive condition who spent more time reading the text showed stronger test scores overall.

The recent explosion in popularity of large language models (LLMs) has inspired learning engineers to incorporate them into adaptive educational tools that automatically score summary writing. Understanding and evaluating LLMs is vital before deploying them in critical learning environments, yet their unprecedented size and expanding number of parameters inhibits transparency and impedes trust when they underperform. Through a collaborative user-centered design process with several learning engineers building and deploying summary scoring LLMs, we characterized fundamental design challenges and goals around interpreting their models, including aggregating large text inputs, tracking score provenance, and scaling LLM interpretability methods. To address their concerns, we developed iScore, an interactive visual analytics tool for learning engineers to upload, score, and compare multiple summaries simultaneously. Tightly integrated views allow users to iteratively revise the language in summaries, track changes in the resulting LLM scores, and visualize model weights at multiple levels of abstraction. To validate our approach, we deployed iScore with three learning engineers over the course of a month. We present a case study where interacting with iScore led a learning engineer to improve their LLM’s score accuracy by three percentage points. Finally, we conducted qualitative interviews with the learning engineers that revealed how iScore enabled them to understand, evaluate, and build trust in their LLMs during deployment.

Readability research and the entailing derivation of theoretically valid and high-performing readability assessment models are important components of reading education. There have been notable advances in the research of readability assessment following the advances of natural language processing techniques. Specifically, newer formulas make use of complex linguistic features that were previously unavailable, as well as contextually representative word-embeddings derived by training neural networks. However, there remain limitations for newer readability formulas, especially with respect to the employment of newer formulas by educational content creators. This paper provides an overview of readability research and introduces a new web app to help facilitate the application of state-of-the-art readability formulas for educational content creators and researchers through both an online interface and a separate API. An increase in accessibility and exposure for state-of-the-art readability formulas will assist in addressing many limitations of integrating readability research into educational technologies.