Literacy and mathematics have been shown to be related to each other across languages, ages, and levels of proficiency (e.g., [5, 11, 6, 19, 29, 35, 39, 43, 46, 49, 53, 57]). More specifically, math instruction is further complicated and becomes more difficult when occurring in a non-native language of instruction (e.g., [2, 8, 16, 18, 20, 31, 41, 50]). In this paper, we perform a linear mixed-effects regression analysis on large-scale institutional student data to test the impact of a non-native, and in some cases - new, language of instruction on students' success as measured by course grades. Specifically, we compare the relationship between achievement in math and English classes for Chinese international students (who previously received math instruction in Chinese dialects), relative to Indian international students (who previously received math instruction in English), relative to a baseline of American students of varying ethnic backgrounds, who have previously received math instruction in English, and for many of whom it is a native language. Findings show that language barriers do not impede international students' math achievement. Future work should further characterize the factors that contribute to students' math achievement, overcoming any limitations that may be posed by language barriers.

Learning engineering applies data and learning science principles to better understand outcomes and support improvement research. One important approach is A/B testing-common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE), and the International Consortium for Innovation and Collaboration in Learning Engineering (IEEE ICICLE). Several systems supporting A/B testing in educational applications have arisen recently, including UpGrade, E-TRIALS, and Terracotta. A/B testing can help improve educational platforms, yet there are challenging issues unique to conducting such work in these contexts. In response, a number of digital learning platforms have opened their systems to learning-improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore how A/B testing is conducted in educational contexts, how digital learning platforms are accelerating education research, and how empirical approaches can be used to drive powerful gains in student learning. It will also discuss opportunities for funding to conduct platform-enabled learning engineering.

Learning engineering adds tools and processes to learning platforms to support improvement research. One kind of tool is A/B testing-common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE), and the International Consortium for Innovation and Collaboration in Learning Engineering (IEEE ICICLE). Recently, several A/B testing systems have arisen that focus on conducting research in educational environments, including UpGrade, Terracotta, and E-TRIALS. A/B testing can help improve educational platforms, yet there are challenging issues unique to conducting such work in these contexts. In response, a number of digital learning platforms have opened their systems to learning-improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore challenges of A/B testing in educational contexts, how learning platforms are accelerating education research, and how empirical approaches can be used to drive powerful gains in student learning. It will also discuss opportunities for funding to conduct platform-enabled learning research.

This study investigates the application of knowledge tracing to the domain of reading comprehension, a complex field characterized by rich contextual data and interrelated concepts. We propose adapting the Dynamic Key-Value Memory Networks (DKVMN) model to incorporate sentence embeddings to better capture the semantic richness of reading tasks, naming our new model Context-embedded DKVMN (CDKVMN). The study employs an extant dataset of 405 students that each completed the reading game "Map Conquest." This game was designed to evaluate students' mastery and use of key reading strategies, such as paraphrasing and bridging. Our findings indicate that CDKVMN outperforms Deep Knowledge Tracing and performs similarly or better than DKVMN in predicting students' performance. This research underscores the potential of advanced, context-sensitive knowledge tracing models to track students' mastery of reading strategies, which can be used to provide support and adapt learning activities to the user. Future work will focus on refining the contextual embeddings, expanding the dataset to a variety of reading games, and interpreting the detected latent concepts.

Reading comprehension is essential for students' ability to build knowledge. Students' comprehension abilities can be enhanced by providing students with deliberate practice and formative feedback on reading comprehension strategies. iSTART is an Intelligent Tutoring System (ITS) that is designed to provide instruction in reading strategies with minimal teacher supervision - affording the ability to teach reading strategies at scale. In the current study, undergraduate students received reading strategy instruction and opportunities for deliberate practice via the iSTART intelligent tutoring system or not (i.e., no-treatment control group). Participants' reading comprehension and psychology knowledge were assessed. The iSTART group demonstrated substantially greater scores than the control group on a post-training reading comprehension measure (Cohen's d > 1.0). The average psychology knowledge scores did not differ between iSTART (post-training) and control groups, but overall these scores were unexpectedly low. Within the iSTART group, there was no difference in reading comprehension and knowledge scores as a function of students' different behaviors in the system. Overall, the results indicate that iSTART is an effective tool to teach reading strategies at large scale. However, further work is required to test the extent to which iSTART supports knowledge building.

The Interactive Strategy Training for Active Reading and Thinking (iSTART) game-based intelligent tutoring system (ITS) was developed with a foundation of comprehension theory and principles of learning science to improve students’ comprehension of complex scientific texts. iSTART has been shown to improve reading comprehension for learners from middle school through adulthood, particularly lower knowledge readers, through strategy instruction and game-based practice. This paper describes iSTART, the theoretical foundations that have guided iSTART development, and evidence for the feasibility of game-based practice to improve learning outcomes. This paper also introduces a novel method of assessing students’ reading comprehension through game-based literacy assessments that have been incorporated in iSTART. The development of these stealth assessments was guided by recent work emphasizing the need for rapid, dynamic, and low stakes assessments that evaluate students’ reading skills in the context of brief, dynamic games. Stealth assessments can generate estimates of multiple aspects of students’ reading comprehension quickly and within a motivating environment. The work described in this paper is a promising method to assess students’ literacy in an unobtrusive and authentic way that may lead to improved learning outcomes for students.

This study presents the results of a computational discourse analysis of discussion threads within an online Math tutoring platform. This work is theoretically motivated by prior work that established the importance of linguistic and semantic features in the discourse in mathematics education. The end goal of this study is to understand the characteristics of language that is produced and used within a discussion board for math. The discussion board corpus comprises of posts from 4,720 students, teachers, and study experts who interacted within an online teaching and learning tutoring platform for math. Linguistic profiles of the discussion board discourse were estimated using Principal Component Analysis (PCA) based on Coh-Metrix linguistic features related to cohesion, language sophistication, and lexical characteristics. The PCA analysis yielded seven Math Discourse Linguistic Components, which collectively explained 49% of the variance in the dataset. Theoretical and conceptual validation of components revealed that the linguistic features align with the communication goal and the nature of mathematics. The linguistic profiles that characterized the discussion board discourse included referential cohesion, information density, instructional language, lexical variation, compare and contrast devices, explicit relations devices, and syntactic complexity. The dominance of cohesive cues within the linguistic profiles demonstrate the communication goals within the Math discourse such as elaboration, providing instruction, compare and contrast, establishing explicit relations, and presenting information. As such, these components characterize the Math Discussion Board discourse in terms of variations in cohesive and task-oriented cues within communication among students.

Learning engineering adds tools and processes to learning platforms to support improvement research. One kind of tool is A/B testing, which is common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE). A number of A/B testing systems focused on educational applications have arisen recently, including UpGrade and E-TRIALS. A/B testing can be part of the puzzle of how to improve educational platforms, and yet challenging issues in education go beyond the generic paradigm. For example, the importance of teachers and instructors to learning means that students are not only connecting with software as individuals, but also as part of a shared classroom experience. Further, learning in topics like mathematics can be highly dependent on prior learning, and thus A or B may not be better overall, but only in interaction with prior knowledge. In response, a set of learning platforms is opening their systems to improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore how A/B testing in educational contexts is different, how learning platforms are opening up new possibilities, and how these empirical approaches can be used to drive powerful gains in student learning. It will also discuss forthcoming opportunities for funding to conduct platform-enabled learning research.

Automated Writing Evaluation systems have been developed to help students improve their writing skills through the automated delivery of both summative and formative feedback. These systems have demonstrated strong potential in a variety of educational contexts; however, they remain limited in their personalization and scope. The purpose of the current study was to begin to address this gap by examining whether individual differences could be modeled in a source-based writing context. Undergraduate students (n=106) wrote essays in response to multiple sources and then completed an assessment of their vocabulary knowledge. Natural language processing tools were used to characterize the linguistic properties of the source-based essays at four levels: descriptive, lexical, syntax, and cohesion. Finally, machine learning models were used to predict students’ vocabulary scores from these linguistic features. The models accounted for approximately 29% of the variance in vocabulary scores, suggesting that the linguistic features of source-based essays are reflective of individual differences in vocabulary knowledge. Overall, this work suggests that automated text analyses can help to understand the role of individual differences in the writing process, which may ultimately help to improve personalization in computer-based learning environments.

The substantial expansion of secure messaging (SM) via the patient portal in the last decade suggests that it is becoming a standard of care, but few have examined SM use longitudinally. We examined SM patterns among a diverse cohort of patients with diabetes (N = 19 921) and the providers they exchanged messages with within a large, integrated health system over 10 years (2006-2015), linking patient demographics to SM use. We found a 10-fold increase in messaging volume. There were dramatic increases overall and for patient subgroups, with a majority of patients (including patients with lower income or with self-reported limited health literacy) messaging by 2015. Although more physicians than nurses and other providers messaged throughout the study, the distribution of health professions using SM changed over time. Given this rapid increase in SM, deeper understanding of optimizing the value of patient and provider engagement, while managing workflow and training challenges, is crucial.

Open-ended comprehension questions are a common type of assessment used to evaluate how well students understand one of multiple documents. Our aim is to use natural language processing (NLP) to infer the level and type of inferencing within readers' answers to comprehension questions using the linguistic and semantic features within their responses. Our taxonomy considers three types of responses to comprehension questions from students ( N=146) who read four documents: a) textbase responses (i.e., information required for the answer is present in a contiguous short sequence of text); b) single-document inference responses (i.e., requiring information from multiple text segments in a single document); and c) multi-document inference responses (i.e., information spanning multiple documents is required). The classification task was approached in two ways. First, we extracted features from students' answers to the comprehension questions using linguistic and semantic indices related to textual complexity and an extended Cohesion Network Analysis (CNA) graph to assess semantic links between the answers and the reference documents. Second, we compared different Recurrent Neural Networks (RNNs) architectures that rely on word embeddings to encode both answers and reference documents. Our best model based on RNNs predicts the answer type with an accuracy of 81%.

This study examines the extent to which instructions to self-explain vs. other-explain a text lead readers to produce different forms of explanations. Natural language processing was used to examine the content and characteristics of the explanations produced as a function of instruction condition. Undergraduate students (n = 146) typed either self-explanations or other-explanations while reading a science text. The linguistic properties of these explanations were calculated using three automated text analysis tools. Machine learning classifiers in combination with the features were used to predict instruction condition (i.e., self- or other-explanation). The best machine learning model performed at rates above chance (kappa = .247; accuracy = 63%). Follow-up analyses indicated that students in the self-explanation condition generated explanations that were more cohesive and that contained words that were more related to social order (e.g., ethics). Overall, the results suggest that natural language processing techniques can be used to detect subtle differences in students' processing of complex texts.

The assessment of writing proficiency generally includes analyses of the specific linguistic and rhetorical features contained in the singular essays produced by students. However, researchers have recently proposed that an individual's ability to flexibly adapt the linguistic properties of their writing might more closely capture writing skill. However, the features of the task, learner, and educational context that influence this flexibility remain largely unknown. The current study extends this research by examining relations between linguistic flexibility, reading comprehension ability, and feedback in the context of an automated writing evaluation system. Students (n = 131) wrote and revised six essays in an automated writing evaluation system and were provided both summative and formative feedback on their writing. Additionally, half of the students had access to a spelling and grammar checker that provided lower-level feedback during the writing period. The results provide evidence for the fact that developing writers demonstrate linguistic flexibility across the essays that they produce. However, analyses also indicate that lower-level feedback (i.e., spelling and grammar feedback) have little to no impact on the properties of students' essays nor on their variability across prompts or drafts. Overall, the current study provides important insights into the role of flexibility in writing skill and develops a strong foundation on which to conduct future research and educational interventions.

Self-explanations are commonly used to assess on-line reading comprehension processes. However, traditional methods of analysis ignore important temporal variations in these explanations. This study investigated how dynamical systems theory could be used to reveal linguistic patterns that are predictive of self-explanation quality. High school students (n = 232) generated self-explanations while they read a science text. Recurrence Plots were generated to show qualitative differences in students' linguistic sequences that were later quantified by indices derived by Recurrence Quantification Analysis (RQA). To predict self-explanation quality, RQA indices, along with summative measures (i.e., number of words, mean word length, and type-token ration) and general reading ability, served as predictors in a series of regression models. Regression analyses indicated that recurrence in students' self-explanations significantly predicted human rated self-explanation quality, even after controlling for summative measures of self-explanations, individual differences, and the text that was read (R2 = 0.68). These results demonstrate the utility of RQA in exposing and quantifying temporal structure in student's self-explanations. Further, they imply that dynamical systems methodology can be used to uncover important processes that occur during comprehension.

In this study, we investigated the degree to which the cognitive processes in which students engage during reading comprehension could be examined through dynamical analyses of their natural language responses to texts. High school students (n = 142) generated typed self-explanations while reading a science text. They then completed a comprehension test that measured their comprehension at both surface and deep levels. The recurrent patterns of the words in students' self-explanations were first visualized in recurrence plots. These visualizations allowed us to qualitatively analyze the different self-explanation processes of skilled and less skilled readers. These recurrence plots then allowed us to calculate recurrence indices, which represented the properties of these temporal word patterns. Results of correlation and regression analyses revealed that these recurrence indices were significantly related to the students' comprehension scores at both surface- and deep levels. Additionally, when combined with summative metrics of word use, these indices were able to account for 32% of the variance in students' overall text comprehension scores. Overall, our results suggest that recurrence quantification analysis can be utilized to guide both qualitative and quantitative assessments of students' comprehension.

A number of studies have demonstrated links between linguistic knowledge and performance in math. Studies examining these links in first language speakers of English have traditionally relied on correlational analyses between linguistic knowledge tests and standardized math tests. For second language (L2) speakers, the majority of studies have compared math performance between proficient and non-proficient speakers of English. In this study, we take a novel approach and examine the linguistic features of student language while they are engaged in collaborative problem solving within an on-line math tutoring system. We transcribe the students' speech and use natural language processing tools to extract linguistic information related to text cohesion, lexical sophistication, and sentiment. Our criterion variables are individuals' pretest and posttest math performance scores. In addition to examining relations between linguistic features of student language production and math scores, we also control for a number of non-linguistic factors including gender, age, grade, school, and content focus (procedural versus conceptual). Linear mixed effect modeling indicates that non-linguistic factors are not predictive of math scores. However, linguistic features related to cohesion affect and lexical proficiency explained approximately 30% of the variance (R2 = .303) in the math scores.

There is untapped potential in achieving the full impact of learning analytics through the integration of tools into practical pedagogic contexts. To meet this potential, more work must be conducted to support educators in developing learning analytics literacy. The proposed workshop addresses this need by building capacity in the learning analytics community and developing an approach to resourcing for building 'writing analytics literacy'.

Textual complexity is widely used to assess the difficulty of reading materials and writing quality in student essays. At a lexical level, word complexity can represent a building block for creating a comprehensive model of lexical networks that adequately estimates learners’ understanding. In order to best capture how lexical associations are created between related concepts, we propose automated indices of word complexity based on Age of Exposure (AoE). AOE indices computationally model the lexical learning process as a function of a learner's experience with language. This study describes a proof of concept based on the on a large-scale learning corpus (i.e., TASA). The results indicate that AoE indices yield strong associations with human ratings of age of acquisition, word frequency, entropy, and human lexical response latencies providing evidence of convergent validity.

Writing training systems have been developed to provide students with instruction and deliberate practice on their writing. Although generally successful in providing accurate scores, a common criticism of these systems is their lack of personalization and adaptive instruction. In particular, these systems tend to place the strongest emphasis on delivering accurate scores, and therefore, tend to overlook additional indices that may contribute to students' success, such as their affective states during writing practice. This study takes an initial step toward addressing this gap by building a predictive model of students' affect using information that can potentially be collected by computer systems. We used individual difference measures, text indices, and keystroke analyses to predict engagement and boredom in 132 writing sessions. The results suggest that these three categories of indices were successful in modeling students' affective states during writing. Taken together, indices related to students' academic abilities, text properties, and keystroke logs were able classify high and low engagement and boredom in writing sessions with accuracies between 76.5% and 77.3%. These results suggest that information readily available in writing training systems can inform affect detectors and ultimately improve student models within intelligent tutoring systems.

Completion rates for massive open online classes (MOOCs) are notoriously low. Identifying student patterns related to course completion may help to develop interventions that can improve retention and learning outcomes in MOOCs. Previous research predicting MOOC completion has focused on click-stream data, student demographics, and natural language processing (NLP) analyses. However, most of these analyses have not taken full advantage of the multiple types of data available. This study combines click-stream data and NLP approaches to examine if students' on-line activity and the language they produce in the online discussion forum is predictive of successful class completion. We study this analysis in the context of a subsample of 320 students who completed at least one graded assignment and produced at least 50 words in discussion forums, in a MOOC on educational data mining. The findings indicate that a mix of click-stream data and NLP indices can predict with substantial accuracy (78%) whether students complete the MOOC. This predictive power suggests that student interaction data and language data within a MOOC can help us both to understand student retention in MOOCs and to develop automated signals of student success.

Writing Analytics focuses on the measurement and analysis of written texts for the purpose of understanding writing processes and products, in their educational contexts, and improving the teaching and learning of writing. This workshop adopts a critical, holistic perspective in which the definition of "the system" and "success" is not restricted to IR metrics such as precision and recall, but recognizes the many wider issues that aid or obstruct analytics adoption in educational settings, such as theoretical and pedagogical grounding, usability, user experience, stakeholder design engagement, practitioner development, organizational infrastructure, policy and ethics.

ReaderBench is an automated software framework designed to support both students and tutors by making use of text mining techniques, advanced natural language processing, and social network analysis tools. ReaderBench is centered on comprehension prediction and assessment based on a cohesion-based representation of the discourse applied on different sources (e.g., textual materials, behavior tracks, metacognitive explanations, Computer Supported Collaborative Learning – CSCL – conversations). Therefore, ReaderBench can act as a Personal Learning Environment (PLE) which incorporates both individual and collaborative assessments. Besides the a priori evaluation of textual materials’ complexity presented to learners, our system supports the identification of reading strategies evident within the learners’ self-explanations or summaries. Moreover, ReaderBench integrates a dedicated cohesion-based module to assess participation and collaboration in CSCL conversations.

This study builds upon previous work aimed at developing a student model of reading comprehension ability within the intelligent tutoring system, iSTART. Currently, the system evaluates students' self-explanation performance using a local, sentence-level algorithm and does not adapt content based on reading ability. The current study leverages natural language processing tools to build models of students' comprehension ability from the linguistic properties of their self-explanations. Students (n = 126) interacted with iSTART across eight training sessions where they self-explained target sentences from complex science texts. Coh-Metrix was then used to calculate the linguistic properties of their aggregated self-explanations. The results of this study indicated that the linguistic indices were predictive of students' reading comprehension ability, over and above the current system algorithms. These results suggest that natural language processing techniques can inform stealth assessments and ultimately improve student models within intelligent tutoring systems.

This study investigates a new approach to automatically assessing essay quality that combines traditional approaches based on assessing textual features with new approaches that measure student attributes such as demographic information, standardized test scores, and survey results. The results demonstrate that combining both text features and student attributes leads to essay scoring models that are on par with state-of-the-art scoring models. Such findings expand our knowledge of textual and non-textual features that are predictive of writing success.

As Computer Supported Collaborative Learning (CSCL) becomes increasingly adopted as an alternative to classic educational scenarios, we face an increasing need for automatic tools designed to support tutors in the time consuming process of analyzing conversations and interactions among students. Therefore, building upon a cohesion-based model of the discourse, we have validated ReaderBench, a system capable of evaluating collaboration based on a social knowledge-building perspective. Through the inter-twining of different participants' points of view, collaboration emerges and this process is reflected in the identified cohesive links between different speakers. Overall, the current experiments indicate that textual cohesion successfully detects collaboration between participants as ideas are shared and exchanged within an ongoing conversation.

Writing researchers have suggested that students who are perceived as strong writers (i.e., those who generate texts that are rated as high quality) demonstrate flexibility in their writing style. While anecdotally this has been a commonly held belief among researchers, scientists, and educators, there is little empirical research to support this claim. This study investigates this hypothesis by examining how students vary in their use of linguistic features across 16 prompt-based essays. Forty-five high school students wrote 16 essays across 8 sessions within an Automated Writing Evaluation (AWE) system. Natural language processing (NLP) techniques and Entropy analyses were used to calculate how rigid or flexible students were in their use of narrative linguistic features over time and how this trait related to individual differences in literacy ability and essay quality. Additional analyses indicated that NLP and Entropy reliably detected narrative flexibility (or rigidity) after session 2 and was related to students' prior literacy skills. These exploratory methodologies are important for researchers and educators, as they indicate that writing flexibility is indeed a trait of strong writers and can be detected rather quickly using the combination of textual features and dynamic analyses.

We report preliminary data of an initial laboratory study examining the effectiveness of self-regulated learning (SRL) training versus no training on learners' ability to deploy SRL processes and learn about the circulatory system with MetaTutor. MetaTutor is an intelligent tutoring system (ITS) designed to train and foster learners' SRL processes while learning about several complex human body systems. We used a mixed methodology approach and include the results of a subset of the participants (N=30) whose product and process data we have analyzed. Overall, the results indicate that the SRL training group significantly outperformed the control group.

Researchers of educational technologies are often asked to do the impossible: make students learn and have them enjoy it. These two objectives, though not mutually exclusive, are frequently at odds with each other. Effective learning strategies require active knowledge use on the part of the student. Meanwhile, students typically seek to learn through the path of least effort. This can cause conflict during system interaction, and it is often the case that attitudes toward the learning environment suffer. The current study indicates that students' prior expectations of what technology can (or cannot) do may actually have a greater impact than their initial level of motivation, previous domain knowledge, and familiarity with technology, combined. Knowing these prior expectations may be a crucial step to help researchers perform the impossible.

Comprehension of science texts is challenging, particularly when the reader lacks the skills or knowledge necessary to fill in conceptual gaps in the text content. The iSTART system was developed to help readers learn and practice reading strategies to improve their ability to comprehend challenging text. This study describes a new iSTART module recently developed and tested, called Interactive Paraphrasing (IP), in which students are interactively and adaptively taught how to paraphrase sentences. We compared the effects of iSTART to iSTART with IP (IP-iSTART) with high school students on their strategy use and ability to comprehend text. IP-iSTART increased skilled readers' self-explanation quality, improved their ability to answer online comprehension questions, and increased their use of paraphrases after training. Less skilled readers benefited most in self-explanation quality from the original version of iSTART. Results are discussed in terms of tailoring reading strategy training to the needs of the reader.

We present in this paper an approach to assessing student paraphrases in the intelligent tutoring system iSTART. The approach is based on measuring the semantic similarity between a student paraphrase and a reference text, called the textbase. The semantic similarity is estimated using knowledge-based word relatedness measures. The relatedness measures rely on knowledge encoded in Word-Net, a lexical database of English. We also experiment with weighting words based on their importance. The word importance information was derived from an analysis of word distributions in 2,225,726 documents from Wikipedia. Performance is reported for 12 different models which resulted from combining 3 different relatedness measures, 2 word sense disambiguation methods, and 2 word-weighting schemes. Furthermore, comparisons are made to other approaches such as Latent Semantic Analysis and the Entailer.

The domain of artificial intelligence (AI) progresses with extraordinary vicissitude. Whereas prior authors have divided AI into the two categories of analysis and synthesis, Raine and op den Akker distinguish between four types of AI: that of appearance, function, simulation and interpretation. These subdomains of AI are differentiated by user goals, creator methodologies, and environmental constraints. In this paper, we focus on how analysis and synthesis could improve the subdomain of Functional-AI.

This paper evaluates four of the most commonly used, freely available, state-of-the-art parsers on a standard benchmark as well as with respect to a set of data relevant for measuring text cohesion, as one example of a learning technology application that requires fast and accurate syntactic parsing. We outline advantages and disadvantages of existing technologies and make recommendations. Our performance report uses traditional measures based on a gold standard as well as novel dimensions for parsing evaluation. To our knowledge, this is the first attempt to evaluate parsers across genres and grade levels for the implementation in learning technology using both gold standard and directed evaluation methods.