The Interactive Strategy Training for Active Reading and Thinking (iSTART) game-based intelligent tutoring system (ITS) was developed with a foundation of comprehension theory and principles of learning science to improve students’ comprehension of complex scientific texts. iSTART has been shown to improve reading comprehension for learners from middle school through adulthood, particularly lower knowledge readers, through strategy instruction and game-based practice. This paper describes iSTART, the theoretical foundations that have guided iSTART development, and evidence for the feasibility of game-based practice to improve learning outcomes. This paper also introduces a novel method of assessing students’ reading comprehension through game-based literacy assessments that have been incorporated in iSTART. The development of these stealth assessments was guided by recent work emphasizing the need for rapid, dynamic, and low stakes assessments that evaluate students’ reading skills in the context of brief, dynamic games. Stealth assessments can generate estimates of multiple aspects of students’ reading comprehension quickly and within a motivating environment. The work described in this paper is a promising method to assess students’ literacy in an unobtrusive and authentic way that may lead to improved learning outcomes for students.

Automated Writing Evaluation systems have been developed to help students improve their writing skills through the automated delivery of both summative and formative feedback. These systems have demonstrated strong potential in a variety of educational contexts; however, they remain limited in their personalization and scope. The purpose of the current study was to begin to address this gap by examining whether individual differences could be modeled in a source-based writing context. Undergraduate students (n=106) wrote essays in response to multiple sources and then completed an assessment of their vocabulary knowledge. Natural language processing tools were used to characterize the linguistic properties of the source-based essays at four levels: descriptive, lexical, syntax, and cohesion. Finally, machine learning models were used to predict students’ vocabulary scores from these linguistic features. The models accounted for approximately 29% of the variance in vocabulary scores, suggesting that the linguistic features of source-based essays are reflective of individual differences in vocabulary knowledge. Overall, this work suggests that automated text analyses can help to understand the role of individual differences in the writing process, which may ultimately help to improve personalization in computer-based learning environments.

This study examines the extent to which instructions to self-explain vs. other-explain a text lead readers to produce different forms of explanations. Natural language processing was used to examine the content and characteristics of the explanations produced as a function of instruction condition. Undergraduate students (n = 146) typed either self-explanations or other-explanations while reading a science text. The linguistic properties of these explanations were calculated using three automated text analysis tools. Machine learning classifiers in combination with the features were used to predict instruction condition (i.e., self- or other-explanation). The best machine learning model performed at rates above chance (kappa = .247; accuracy = 63%). Follow-up analyses indicated that students in the self-explanation condition generated explanations that were more cohesive and that contained words that were more related to social order (e.g., ethics). Overall, the results suggest that natural language processing techniques can be used to detect subtle differences in students' processing of complex texts.

The assessment of writing proficiency generally includes analyses of the specific linguistic and rhetorical features contained in the singular essays produced by students. However, researchers have recently proposed that an individual's ability to flexibly adapt the linguistic properties of their writing might more closely capture writing skill. However, the features of the task, learner, and educational context that influence this flexibility remain largely unknown. The current study extends this research by examining relations between linguistic flexibility, reading comprehension ability, and feedback in the context of an automated writing evaluation system. Students (n = 131) wrote and revised six essays in an automated writing evaluation system and were provided both summative and formative feedback on their writing. Additionally, half of the students had access to a spelling and grammar checker that provided lower-level feedback during the writing period. The results provide evidence for the fact that developing writers demonstrate linguistic flexibility across the essays that they produce. However, analyses also indicate that lower-level feedback (i.e., spelling and grammar feedback) have little to no impact on the properties of students' essays nor on their variability across prompts or drafts. Overall, the current study provides important insights into the role of flexibility in writing skill and develops a strong foundation on which to conduct future research and educational interventions.

Self-explanations are commonly used to assess on-line reading comprehension processes. However, traditional methods of analysis ignore important temporal variations in these explanations. This study investigated how dynamical systems theory could be used to reveal linguistic patterns that are predictive of self-explanation quality. High school students (n = 232) generated self-explanations while they read a science text. Recurrence Plots were generated to show qualitative differences in students' linguistic sequences that were later quantified by indices derived by Recurrence Quantification Analysis (RQA). To predict self-explanation quality, RQA indices, along with summative measures (i.e., number of words, mean word length, and type-token ration) and general reading ability, served as predictors in a series of regression models. Regression analyses indicated that recurrence in students' self-explanations significantly predicted human rated self-explanation quality, even after controlling for summative measures of self-explanations, individual differences, and the text that was read (R2 = 0.68). These results demonstrate the utility of RQA in exposing and quantifying temporal structure in student's self-explanations. Further, they imply that dynamical systems methodology can be used to uncover important processes that occur during comprehension.

This paper examines the effect of different linguistic features (as identified through Natural Language Processing tools) on affective measures of student engagement using a discovery with models approach. We build on previous literature, using automated detectors that identify when a middle-school student using an online mathematics tutor is experiencing boredom, confusion, frustration, or engaged concentration, to identify which problems are most engaging (or not) at scale. We then apply previously validated NLP tools to determine the degree to which engagement findings may be related to the linguistic properties of word problems, contributing to a growing literature on the effects of language on mathematics learning.

In this study, we investigated the degree to which the cognitive processes in which students engage during reading comprehension could be examined through dynamical analyses of their natural language responses to texts. High school students (n = 142) generated typed self-explanations while reading a science text. They then completed a comprehension test that measured their comprehension at both surface and deep levels. The recurrent patterns of the words in students' self-explanations were first visualized in recurrence plots. These visualizations allowed us to qualitatively analyze the different self-explanation processes of skilled and less skilled readers. These recurrence plots then allowed us to calculate recurrence indices, which represented the properties of these temporal word patterns. Results of correlation and regression analyses revealed that these recurrence indices were significantly related to the students' comprehension scores at both surface- and deep levels. Additionally, when combined with summative metrics of word use, these indices were able to account for 32% of the variance in students' overall text comprehension scores. Overall, our results suggest that recurrence quantification analysis can be utilized to guide both qualitative and quantitative assessments of students' comprehension.

There is untapped potential in achieving the full impact of learning analytics through the integration of tools into practical pedagogic contexts. To meet this potential, more work must be conducted to support educators in developing learning analytics literacy. The proposed workshop addresses this need by building capacity in the learning analytics community and developing an approach to resourcing for building 'writing analytics literacy'.

This paper presents a diachronic analysis centered on the exploration of differences between the writing styles of journalistic texts in Romanian language. This analysis is focused on the time evolution of this language across two adjacent regions, Bessarabia and Romania in two major periods that were marked by important historical differences. Our aim is to examine these language differences based on corpora of historical and contemporary texts. To this end, we employ the ReaderBench framework to calculate a number of textual complexity indices that can be reliably used to characterize writing style. These analyses are conducted on two independent corpora for each of the two language styles, covering the following time periods: 1941-1991, when Bessarabia was separated from Romania and became a state in the Soviet Union (and there were few connections and language influences with Romania), and after July 1991, when Bessarabia became an independent state, Republic of Moldavia (and many language interactions with Romania occurred). The results of our analyses highlight the lexical and cohesive textual complexity indices that best reflect the differences in writing style, ranging from sentence and paragraph structure to word entropy and cohesion, measured in terms of Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA).

Writing training systems have been developed to provide students with instruction and deliberate practice on their writing. Although generally successful in providing accurate scores, a common criticism of these systems is their lack of personalization and adaptive instruction. In particular, these systems tend to place the strongest emphasis on delivering accurate scores, and therefore, tend to overlook additional indices that may contribute to students' success, such as their affective states during writing practice. This study takes an initial step toward addressing this gap by building a predictive model of students' affect using information that can potentially be collected by computer systems. We used individual difference measures, text indices, and keystroke analyses to predict engagement and boredom in 132 writing sessions. The results suggest that these three categories of indices were successful in modeling students' affective states during writing. Taken together, indices related to students' academic abilities, text properties, and keystroke logs were able classify high and low engagement and boredom in writing sessions with accuracies between 76.5% and 77.3%. These results suggest that information readily available in writing training systems can inform affect detectors and ultimately improve student models within intelligent tutoring systems.

Writing Analytics focuses on the measurement and analysis of written texts for the purpose of understanding writing processes and products, in their educational contexts, and improving the teaching and learning of writing. This workshop adopts a critical, holistic perspective in which the definition of "the system" and "success" is not restricted to IR metrics such as precision and recall, but recognizes the many wider issues that aid or obstruct analytics adoption in educational settings, such as theoretical and pedagogical grounding, usability, user experience, stakeholder design engagement, practitioner development, organizational infrastructure, policy and ethics.

This study builds upon previous work aimed at developing a student model of reading comprehension ability within the intelligent tutoring system, iSTART. Currently, the system evaluates students' self-explanation performance using a local, sentence-level algorithm and does not adapt content based on reading ability. The current study leverages natural language processing tools to build models of students' comprehension ability from the linguistic properties of their self-explanations. Students (n = 126) interacted with iSTART across eight training sessions where they self-explained target sentences from complex science texts. Coh-Metrix was then used to calculate the linguistic properties of their aggregated self-explanations. The results of this study indicated that the linguistic indices were predictive of students' reading comprehension ability, over and above the current system algorithms. These results suggest that natural language processing techniques can inform stealth assessments and ultimately improve student models within intelligent tutoring systems.

This study investigates a new approach to automatically assessing essay quality that combines traditional approaches based on assessing textual features with new approaches that measure student attributes such as demographic information, standardized test scores, and survey results. The results demonstrate that combining both text features and student attributes leads to essay scoring models that are on par with state-of-the-art scoring models. Such findings expand our knowledge of textual and non-textual features that are predictive of writing success.

Writing researchers have suggested that students who are perceived as strong writers (i.e., those who generate texts that are rated as high quality) demonstrate flexibility in their writing style. While anecdotally this has been a commonly held belief among researchers, scientists, and educators, there is little empirical research to support this claim. This study investigates this hypothesis by examining how students vary in their use of linguistic features across 16 prompt-based essays. Forty-five high school students wrote 16 essays across 8 sessions within an Automated Writing Evaluation (AWE) system. Natural language processing (NLP) techniques and Entropy analyses were used to calculate how rigid or flexible students were in their use of narrative linguistic features over time and how this trait related to individual differences in literacy ability and essay quality. Additional analyses indicated that NLP and Entropy reliably detected narrative flexibility (or rigidity) after session 2 and was related to students' prior literacy skills. These exploratory methodologies are important for researchers and educators, as they indicate that writing flexibility is indeed a trait of strong writers and can be detected rather quickly using the combination of textual features and dynamic analyses.