Teachers’ in-the-moment support is a limited resource in technology-supported classrooms, and teachers must decide whom to help and when during ongoing student work. However, less is known about how students’ prior help history (whether they were helped earlier) and their engagement states (e.g., idle, struggle) shape teachers’ decisions, and whether observed learning benefits associated with teacher help extend beyond the current class session. To address these questions, we first conducted interviews with nine K–12 mathematics teachers to identify candidate decision factors for teacher help. We then analyzed 1.4 million student–system interactions from 339 students across 14 classes in the MATHia intelligent tutoring system by linking teacher-logged help events with fine-grained engagement states. Mixed-effects models show that students who received help earlier were more likely to receive additional help later, even after accounting for current engagement state. Cross-lagged panel analyses further show that teacher help recurred across sessions, whereas idle behavior did not receive sustained attention over time. Finally, help coincided with immediate learning within sessions, but did not predict skill acquisition in later sessions, as estimated by additive factor modeling. These findings suggest that teacher help is “sticky” in that it recurs for previously supported students, while its measurable learning benefits in our data are largely session-bound. We discuss implications for designing real-time analytics that track attention coverage and highlight under-visited students to support a more equitable and effective allocation of teacher attention.

Measures of disengagement provide insights into unproductive use of learning opportunities. Although measures of active disengagement, such as gaming the system and mind-wandering, are well studied, loss of practice time due to outright task avoidance remains relatively understudied. The current study addresses this gap by extending existing within-task measures (idle time) with two new session-level measures (delayed start and early stop) to capture loss of practice time due to task avoidance. We characterize the combined lost time as coasted time and the associated behavior as coasting behavior. Using ASSISTments logs (N = 1,425), we find that students dedicate only 40% of available classwork time to math practice and coast through the remaining 60%. Of the coasted time, 36% resulted from delayed starts, 2% from mid-practice idling, and 62% from stopping early. Delayed start and early stop showed moderate temporal stability (G = 0.73 and 0.71, respectively), suggesting that coasting is a consistent behavioral pattern. Even after excluding early stops attributable to assignment completion (i.e., early stop = 0), coasted time remained substantial at 32%. While we observe significant differences in coasting by gender and IEP status, we do not observe them by other demographic factors or school locale. Critically, students who continued working beyond the first assignment completion (''extra effort'') performed significantly better on standardized tests. For research, coasting offers a new lens on opportunity loss by combining session-level disengagement with within-task disengagement. For practitioners, our results highlight the need for platform affordances that support sustained engagement and more productive use of available practice time.

Personalized problem selection enhances student practice in tutoring systems.Prior research has focused on transparent problem selection that supports learner control but rarely engages learners in selecting practice materials.We explored how different levels of control (i.e., full AI control, shared control, and full learner control), combined with showing learning analytics on skill mastery and visual what-if explanations, can support students in practice contexts requiring high degrees of self-regulation, such as homework.Semistructured interviews with six middle school students revealed three key insights: (1) participants highly valued learner control for an enhanced learning experience and better self-regulation, especially because most wanted to avoid losses in skill mastery;(2) only seeing their skill mastery estimates often made participants base problem selection on their weaknesses; and (3) what-if explanations stimulated participants to focus more on their strengths and improve skills until they were mastered.These findings show how explainable learning analytics could shape students' selection strategies when they have control over what to practice.They suggest promising avenues for helping students learn to regulate their effort, motivation, and goals during practice with tutoring systems.

Caregivers (i.e., parents and members of a child's caring community) are underappreciated stakeholders in learning analytics. Although caregiver involvement can enhance student academic outcomes, many obstacles hinder involvement, most notably knowledge gaps with respect to modern school curricula. An emerging topic of interest in learning analytics is hybrid tutoring, which includes instructional and motivational support. Caregivers assert similar roles in homework, yet it is unknown how learning analytics can support them. Our past work with caregivers suggested that conversational support is a promising method of providing caregivers with the guidance needed to effectively support student learning. We developed a system that provides instructional support to caregivers through conversational recommendations generated by a Large Language Model (LLM). Addressing known instructional limitations of LLMs, we use instructional intelligence from tutoring systems while conducting prompt engineering experiments with the open-source Llama 3 LLM. This LLM generated message recommendations for caregivers supporting their child's math practice via chat. Few-shot prompting and combining real-time problem-solving context from tutoring systems with examples of tutoring practices yielded desirable message recommendations. These recommendations were evaluated with ten middle school caregivers, who valued recommendations facilitating content-level support and student metacognition through self-explanation. We contribute insights into how tutoring systems can best be merged with LLMs to support hybrid tutoring settings through conversational assistance, facilitating effective caregiver involvement in tutoring systems.

Numerous studies demonstrate the importance of self-regulation during learning by problem-solving. Recent work in learning analytics has largely examined students’ use of SRL concerning overall learning gains. Limited research has related SRL to in-the-moment performance differences among learners. The present study investigates SRL behaviors in relationship to learners’ moment-by-moment performance while working with intelligent tutoring systems for stoichiometry chemistry. We demonstrate the feasibility of labeling SRL behaviors based on AI-generated think-aloud transcripts, identifying the presence or absence of four SRL categories (processing information, planning, enacting, and realizing errors) in each utterance. Using the SRL codes, we conducted regression analyses to examine how the use of SRL in terms of presence, frequency, cyclical characteristics, and recency relate to student performance on subsequent steps in multi-step problems. A model considering students’ SRL cycle characteristics outperformed a model only using in-the-moment SRL assessment. In line with theoretical predictions, students’ actions during earlier, process-heavy stages of SRL cycles exhibited lower moment-by-moment correctness during problem-solving than later SRL cycle stages. We discuss system re-design opportunities to add SRL support during stages of processing and paths forward for using machine learning to speed research depending on the assessment of SRL based on transcription of think-aloud data.

Learning analytics research increasingly studies classroom learning with AI-based systems through rich contextual data from outside these systems, especially student-teacher interactions. One key challenge in leveraging such data is generating meaningful insights into effective teacher practices. Quantitative ethnography bears the potential to close this gap by combining multimodal data streams into networks of co-occurring behavior that drive insight into favorable learning conditions. The present study uses transmodal ordered network analysis to understand effective teacher practices in relationship to traditional metrics of in-system learning in a mathematics classroom working with AI tutors. Incorporating teacher practices captured by position tracking and human observation codes into modeling significantly improved the inference of how efficiently students improved in the AI tutor beyond a model with tutor log data features only. Comparing teacher practices by student learning rates, we find that students with low learning rates exhibited more hint use after monitoring. However, after an extended visit, students with low learning rates showed learning behavior similar to their high learning rate peers, achieving repeated correct attempts in the tutor. Observation notes suggest conceptual and procedural support differences can help explain visit effectiveness. Taken together, offering early conceptual support to students with low learning rates could make classroom practice with AI tutors more effective. This study advances the scientific understanding of effective teacher practice in classrooms learning with AI tutors and methodologies to make such practices visible.

Artificial intelligence (AI) applications to support human tutoring have potential to significantly improve learning outcomes, but engagement issues persist, especially among students from low-income backgrounds. We introduce an AI-assisted tutoring model that combines human and AI tutoring and hypothesize this synergy will have positive impacts on learning processes. To investigate this hypothesis, we conduct a three-study quasi-experiment across three urban and low-income middle schools: 1) 125 students in a Pennsylvania school; 2) 385 students (50% Latinx) in a California school, and 3) 75 students (100% Black) in a Pennsylvania charter school, all implementing analogous tutoring models. We compare learning analytics of students engaged in human-AI tutoring compared to students using math software only. We find human-AI tutoring has positive effects, particularly in student’s proficiency and usage, with evidence suggesting lower achieving students may benefit more compared to higher achieving students. We illustrate the use of quasi-experimental methods adapted to the particulars of different schools and data-availability contexts so as to achieve the rapid data-driven iteration needed to guide an inspired creation into effective innovation. Future work focuses on improving the tutor dashboard and optimizing tutor-student ratios, while maintaining annual costs per student of approximately $700 annually.

Enabling students to dynamically transition between individual and collaborative learning activities has great potential to support better learning. We explore how technology can support teachers in orchestrating dynamic transitions during class. Working with five teachers and 199 students over 22 class sessions, we conducted classroom-based prototyping of a co-orchestration technology ecosystem that supports the dynamic pairing of students working with intelligent tutoring systems. Using mixed-methods data analysis, we study the resulting observed classroom dynamics, and how teachers and students perceived and experienced dynamic transitions as supported by our technology. We discover a potential tension between teachers’ and students’ preferred level of control: students prefer a degree of control over the dynamic transitions that teachers are hesitant to grant. Our study reveals design implications and challenges for future human-AI co-orchestration in classroom use, bringing us closer to realizing the vision of highly-personalized smart classrooms that address the unique needs of each student.

Abstract In numerous studies, intelligent tutoring systems (ITSs) have proven effective in helping students learn mathematics. Prior work posits that their effectiveness derives from efficiently providing eventually-correct practice opportunities. Yet, there is little empirical evidence on how learning processes with ITSs compare to other forms of instruction. The current study compares problem-solving with an ITS versus solving the same problems on paper. We analyze the learning process and pre-post gain data from N = 97 middle school students practicing linear graphs in three curricular units. We find that (i) working with the ITS, students had more than twice the number of eventually-correct practice opportunities than when working on paper and (ii) omission errors on paper were associated with lower learning gains. Yet, contrary to our hypothesis, tutor practice did not yield greater learning gains, with tutor and paper comparing differently across curricular units. These findings align with tutoring allowing students to grapple with challenging steps through tutor assistance but not with eventually-correct opportunities driving learning gains. Gaming-the-system, lack of transfer to an unfamiliar test format, potentially ineffective tutor design, and learning affordances of paper can help explain this gap. This study provides first-of-its-kind quantitative evidence that ITSs yield more learning opportunities than equivalent paper-and-pencil practice and reveals that the relation between opportunities and learning gains emerges only when the instruction is effective.

Gaming the system, a behavior in which learners exploit a system's properties to make progress while avoiding learning, has frequently been shown to be associated with lower learning. However, when we applied a previously validated gaming detector across conditions in experiments with an algebra tutor, the detected gaming was not associated with reduced learning, challenging its validity in our study context. Our exploratory data analysis suggested that varying contextual factors across and within conditions contributed to this lack of association. We present a new approach, latent variable-based gaming detection (LV-GD), that controls for contextual factors and more robustly estimates student-level latent gaming tendencies. In LV-GD, a student is estimated as having a high gaming tendency if the student is detected to game more than the expected level of the population given the context. LV-GD applies a statistical model on top of an existing action-level gaming detector developed based on a typical human labeling process, without additional labeling effort. Across three datasets, we find that LV-GD consistently outperformed the original detector in validity measured by association between gaming and learning as well as reliability. LV-GD also afforded high practical utility: it more accurately revealed intervention effects on gaming, revealed a correlation between gaming and perceived competence in math and helped understand productive detected gaming behaviors. Our approach is not only useful for others wanting a cost-effective way to adapt a gaming detector to their context but is also generally applicable in creating robust behavioral measures.

Analytics of student learning data are increasingly important for continuous redesign and improvement of tutoring systems and courses. There is still a lack of general guidance on converting analytics into better system design, and on combining multiple methods to maximally improve a tutor. We present a multi-method approach to data-driven redesign of tutoring systems and its empirical evaluation. Our approach systematically combines existing and new learning analytics and instructional design methods. In particular, our methods involve identifying difficult skills and creating focused tasks for learning these difficult skills effectively following content redesign strategies derived from analytics. In our past work, we applied this approach to redesigning an algebraic modeling unit and found initial evidence of its effectiveness. In the current work, we extended this approach and applied it to redesigning two other tutor units in addition to a second iteration of redesigning the previously redesigned unit. We conducted a one-month classroom experiment with 129 high school students. Compared to the original tutor, the redesigned tutor led to significantly higher learning outcomes, with time mainly allocated to focused tasks rather than original full tasks. Moreover, it reduced over- and under-practice, yielded a more effective practice experience, and selected skills progressing from easier to harder to a greater degree. Our work provides empirical evidence of the effectiveness and generality of a multi-method approach to data-driven instructional redesign.

Despite the potential of spatial displays for supporting teachers’ classroom orchestration through real-time classroom analytics, the process to design these displays is a challenging and under-explored topic in the learning analytics (LA) community. This paper proposes a mid-fidelity Virtual Prototyping method (VPM), which involves simulating a classroom environment and candidate designs in virtual space to address these challenges. VPM allows for rapid prototyping of spatial features, requires no specialized hardware, and enables teams to conduct remote evaluation sessions. We report observations and findings from an initial exploration with five potential users through a design process utilizing VPM to validate designs for an AR-based spatial display in the context of middle-school orchestration tools. We found that designs created using virtual prototyping sufficiently conveyed a sense of three-dimensionality to address subtle design issues like occlusion and depth perception. We discuss the opportunities and limitations of applying virtual prototyping, particularly its potential to allow for more robust co-design with stakeholders earlier in the design process.

AI-based educational technologies may be most welcome in classrooms when they align with teachers' goals, preferences, and instructional practices. Teachers, however, have scarce time to make such customizations themselves. How might the crowd be leveraged to help time-strapped teachers? Crowdsourcing pipelines have traditionally focused on content generation. It is an open question how a pipeline might be designed so the crowd can succeed in a revision/customization task. In this paper, we explore an initial version of a teacher-guided crowdsourcing pipeline designed to improve the adaptive math hints of an AI-based tutoring system so they fit teachers' preferences, while requiring minimal expert guidance. In two experiments involving 144 math teachers and 481 crowdworkers, we found that such an expert-guided revision pipeline could save experts' time and produce better crowd-revised hints (in terms of teacher satisfaction) than two comparison conditions. The revised hints however, did not improve on the existing hints in the AI tutor, which were carefully-written but still have room for improvement and customization. Further analysis revealed that the main challenge for crowdworkers may lie in understanding teachers' brief written comments and implementing them in the form of effective edits, without introducing new problems. We also found that teachers preferred their own revisions over other sources of hints, and exhibited varying preferences for hints. Overall, the results confirm that there is a clear need for customizing hints to individual teachers' preferences. They also highlight the need for more elaborate scaffolds so the crowd can have specific knowledge of the requirements that teachers have for hints. The study represents a first exploration in the literature of how to support crowds with minimal expert guidance in revising and customizing instructional materials.

This paper aims to link student facing Learning Analytics Dashboards (LADs) to the corpus of research on Open Learner Models (OLMs), as both have similar goals. We conducted a systematic review of literature on OLMs and compared the results with a previously conducted review of LADs for learners in terms of (i) data use and modelling, (ii) key publication venues, (iii) authors and articles, (iv) key themes, and (v) system evaluation. We highlight the similarities and differences between the research on LADs and OLMs. Our key contribution is a bridge between these two areas as a foundation for building upon the strengths of each. We report the following key results from the review: in reports of new OLMs, almost 60% are based on a single type of data; 33% use behavioral metrics; 39% support input from the user; 37% have complex models; and just 6% involve multiple applications. Key associated themes include intelligent tutoring systems, learning analytics, and self-regulated learning. Notably, compared with LADs, OLM research is more likely to be interactive (81% of papers compared with 31% for LADs), report evaluations (76% versus 59%), use assessment data (100% versus 37%), provide a comparison standard for students (52% versus 38%), but less likely to use behavioral metrics, or resource use data (33% against 75% for LADs). In OLM work, there was a heightened focus on learner control and access to their own data.

When used in classrooms, personalized learning software allows students to work at their own pace, while freeing up the teacher to spend more time working one-on-one with students. Yet such personalized classrooms also pose unique challenges for teachers, who are tasked with monitoring classes working on divergent activities, and prioritizing help-giving in the face of limited time. This paper reports on the co-design, implementation, and evaluation of a wearable classroom orchestration tool for K-12 teachers: mixed-reality smart glasses that augment teachers' realtime perceptions of their students' learning, metacognition, and behavior, while students work with personalized learning software. The main contributions are: (1) the first exploration of the use of smart glasses to support orchestration of personalized classrooms, yielding design findings that may inform future work on real-time orchestration tools; (2) Replay Enactments: a new prototyping method for real-time orchestration tools; and (3) an in-lab evaluation and classroom pilot using a prototype of teacher smart glasses (Lumilo), with early findings suggesting that Lumilo can direct teachers' time to students who may need it most.

Accurately modeling individual students' knowledge growth is important in many applications of learning analytics. A key step is to decompose the knowledge targeted in the instruction into detailed knowledge components (KCs). We search for an accurate KC model for basic equation solving skills, using data from an intelligent tutoring system (ITS), Lynnette. Key criteria are data fit and predictive accuracy based on a standard logistic model called the Additive Factors Model (AFM). We focus on three difficulty factors for equation solving: understanding of variables, the negative sign, and the complexity of the equation. Fine-grained KC models were found to have greater fit and predictive accuracy than an "ideal," more abstract model, indicating that there is substantial under-generalization in students' equation-solving skill related to all three difficulty factors. The work enhances scientific understanding of the challenges students face in learning equation solving. It illustrates how learning analytics could inform the improvement of technology-enhanced learning environments.

Instruction that adapts to individual learner characteristics is often more effective than instruction that treats all learners as the same. A practical approach to making MOOCs adapt to learners may be by integrating frameworks for intelligent tutoring systems (ITSs). Using the Learning Tools Interoperability standard (LTI), we integrated two intelligent tutoring frameworks (GIFT and CTAT) into edX. We describe our initial explorations of four adaptive instructional patterns in the PennX MOOC "Big Data and Education." The work illustrates one route to adaptivity at scale.

Intelligent tutoring systems (ITSs) are commonly designed to enhance student learning. However, they are not typically designed to meet the needs of teachers who use them in their classrooms. ITSs generate a wealth of analytics about student learning and behavior, opening a rich design space for real-time teacher support tools such as dashboards. Whereas real-time dashboards for teachers have become popular with many learning technologies, we are not aware of projects that have designed dashboards for ITSs based on a broad investigation of teachers' needs. We conducted design interviews with ten middle school math teachers to explore their needs for on-the-spot support during blended class sessions, as a first step in a user-centered design process of a real-time dashboard. Based on multi-methods analyses of this interview data, we identify several opportunities for ITSs to better support teachers' needs, noting that the analytics commonly generated by existing teacher support tools do not strongly align with the analytics teachers expect to be most useful. We highlight key tensions and tradeoffs in the design of such real-time supports for teachers, as revealed by "Speed Dating" possible futures with teachers. This paper has implications for our ongoing co-design of a real-time dashboard for ITSs, as well as broader implications for the design of ITSs that can effectively collaborate with teachers in classroom settings.

Classroom experiments that evaluate the effectiveness of educational technologies do not typically examine the effects of classroom contextual variables (e.g., out-of-software help-giving and external distractions). Yet these variables may influence students' instructional outcomes. In this paper, we introduce the Spatial Classroom Log Explorer (SPACLE): a prototype tool that facilitates the rapid discovery of relationships between within-software and out-of-software events. Unlike previous tools for retrospective analysis, SPACLE replays moment-by-moment analytics about student and teacher behaviors in their original spatial context. We present a data analysis workflow using SPACLE and demonstrate how this workflow can support causal discovery. We share the results of our initial replay analyses using SPACLE, which highlight the importance of considering spatial factors in the classroom when analyzing ITS log data. We also present the results of an investigation into the effects of student-teacher interactions on student learning in K-12 blended classrooms, using our workflow, which combines replay analysis with SPACLE and causal modeling. Our findings suggest that students' awareness of being monitored by their teachers may promote learning, and that "gaming the system" behaviors may extend outside of educational software use.

The gold standard for identifying more effective pedagogical approaches is to perform an experiment. Unfortunately, frequently a hypothesized alternate way of teaching does not yield an improved effect. Given the expense and logistics of each experiment, and the enormous space of potential ways to improve teaching, it would be highly preferable if it were possible to estimate in advance of running a study whether an alternative teaching strategy would improve learning. This is true even in learning at scale situations, since even if it is logistically easier to recruit a large number of subjects, it remains a high stakes environment because the experiment is impacting many real students. For certain classes of alternate teaching approaches, such as new ways to sequence existing material, it is possible to build student models that can be used as simulators to estimate the performance of learners under new proposed teaching methods. However, existing methods for doing so can overestimate the performance of new teaching methods. We instead propose the Robust Evaluation Matrix (REM) method which explicitly considers model mismatch between the student model used to derive the teaching strategy and that used as a simulator to evaluate the teaching strategy effectiveness. We then present two case studies from a fractions intelligent tutoring system and from a concept learning task from prior work that show how REM could be used both to detect when a new instructional policy may not be effective on actual students and to detect when it may be effective in improving student learning.

Educational games and intelligent tutoring systems (ITS) both support learning by doing, although often in different ways. The current classroom experiment compared a popular commercial game for equation solving, DragonBox and a research-based ITS, Lynnette with respect to desirable educational outcomes. The 190 participating 7th and 8th grade students were randomly assigned to work with either system for 5 class periods. We measured out-of-system transfer of learning with a paper and pencil pre- and post-test of students’ equation-solving skill. We measured enjoyment and accuracy of self-assessment with a questionnaire. The students who used DragonBox solved many more problems and enjoyed the experience more, but the students who used Lynnette performed significantly better on the post-test. Our analysis of the design features of both systems suggests possible explanations and spurs ideas for how the strengths of the two systems might be combined. The study shows that intuitions about what works, educationally, can be fallible. Therefore, there is no substitute for rigorous empirical evaluation of educational technologies.

Learning-by-doing in MOOCs may be enhanced by embedding intelligent tutoring systems (ITSs). ITSs support learning-by-doing by guiding learners through complex practice problems while adapting to differences among learners. We extended the Cognitive Tutor Authoring Tools (CTAT), a widely-used non-programmer tool kit for building intelligent tutors, so that CTAT-built tutors can be embedded in MOOCs and e-learning platforms. We demonstrated the technical feasibility of this integration by adding simple CTAT-built tutors to an edX MOOC, "Big Data in Education." To the best of our knowledge, this integration is the first occasion that material created through an open-access non-programmer authoring tool for full-fledged ITS has been integrated in a MOOC. The work offers examples of key steps that may be useful in other ITS-MOOC integration efforts, together with reflections on strengths, weaknesses, and future possibilities.

As educational games have become a larger field of study, there has been a growing need for analytic methods that can be used to assess game design and inform iteration. While much previous work has focused on the measurement of student engagement or learning at a gross level, we argue that new methods are necessary for measuring the alignment of a game to its target learning goals at an appropriate level of detail to inform design decisions. We present a novel technique that we have employed to examine alignment in an open-ended educational game. The approach is based on examining how the game reacts to representative student solutions that do and do not obey target principles. We demonstrate this method using real student data and discuss how redesign might be informed by these techniques.

The field of Educational Games has seen many calls for added rigor. One avenue for improving the rigor of the field is developing more generalizable methods for measuring student learning within games. Throughout the process of development, what is relevant to measure and assess may change as a game evolves into a finished product. The field needs an approach for game developers and researchers to be able to prototype and experiment with different measures that can stand up to rigorous scrutiny, as well as provide insight into possible new directions for development. We demonstrate a toolkit and analysis tools that capture and analyze students' performance within open educational games. The system records relevant events during play, which can be used for analysis of player learning by designers. The tools support replaying student sessions within the original game's environment, which allows researchers and developers to explore possible explanations for student behavior. Using this system, we were able to facilitate a number of analyses of student learning in an open educational game developed by a team of our collaborators as well as gain greater insight into student learning with the game and where to focus as we iterate.

Designing interactive learning environments (ILEs; e.g., intelligent tutoring systems, educational games, etc.) is a challenging interdisciplinary process that needs to satisfy multiple stakeholders. ILEs need to function in real educational settings (e.g., schools) in which a number of goals interact. Several instructional design methodologies exist to help developers address these goals. However, they often lead to conflicting recommendations. Due to the lack of an established methodology to resolve such conflicts, developers of ILEs have to rely on ad-hoc solutions. We present a principled methodology to resolve such conflicts. We build on a well-established design process for creating Cognitive Tutors, a highly effective type of ILE. We extend this process by integrating methods from multiple disciplines to resolve design conflicts. We illustrate our methodology's effectiveness by describing the iterative development of the Fractions Tutor, which has proven to be effective in classroom studies with 3,000 4th-6th graders.

Gaming the system, attempting to succeed in an interactive learning environment by exploiting properties of the system rather than by learning the material (for example, by systematically guessing or abusing hints), is prevalent across many types of educational software. Past research on why students choose to game has focused on student individual differences. Many student individual differences, including attitudes towards mathematics, have been shown to be associated with gaming, but generally with low correlation. In this paper, we investigate how individual differences between learning environments can increase or decrease the probability of gaming. We enumerate ways intelligent tutor lessons vary from each other, and use data mining to discover hypotheses about how differences in software design and content influence the choice to game the system. We discover a set of tutor features that explain 56% of the variance in gaming, over five times the degree of variance explained in any prior study of student individual differences and gaming. These results provide an important step towards developing prescriptions for designing intelligent tutor software that students game significantly less.

While many computer tutoring systems have long been delivered as desktop applications, these systems have only recently begun to appear on mobile devices. In this work we apply principles of mobile human computer interaction and mobile learning to the design and development of an intelligent tutoring system delivered on a personal digital assistant. We developed a proof-of-concept mobile tutor for business math and tested it in a course designed for first-year college students. Our experiences with the tutor suggest that a tutor based on principles of both mobile learning and mobile human computer interaction can provide a mobile tutoring system capable of providing support to students consistent with mobile device usage patterns.

Democracy requires students to choose policy positions based on evidence, yet confirmation bias prevents them from doing so. As a preliminary step in building a policy reasoning tutor, this study identifies where bias occurs during the search and analysis of evidence in a policy reasoning task. 60 university students played an on-line game in which they chose which of four policies would increase school performance. The between-subjects design compared a free search group who searched for evidence in google-like environment, to a sequential presentation group who read all available evidence, and manipulated whether the evidence confirmed or disconfirmed students' prior beliefs. The study measured the impact on students' evidence-based recommendations, their change in beliefs, and their recall of the evidence. Results showed that students did not cherry-pick evidence nor discount disconfirming evidence. However, students' extreme confidence in their initial beliefs usually prevented them from changing position, and they mistakenly recalled the evidence as confirming their beliefs. The results suggest that a policy tutor should focus on evidence synthesis and making recommendations based on explicit evidence.

Educational games may be particularly suited to teaching social learning skills with virtual humans. We investigate the importance of social goals and engaging social interactions in learning from such games. In one experiment, students played a cultural negotiation game with an explicit social goal or only negotiation task goals. While the group without the explicit social goal learned significantly more, students who reported having social goals in a manipulation check learned the most. In future work, we will develop an intervention built into a virtual learning environment to implicitly scaffold social goals.

Games motivate through challenge, which is often supported by giving proximal, task-related goals. In an educational game for social skills, however, these may not be the most appropriate goals. We ran an experiment in which 54 students played a game designed to teach cross-cultural negotiation skills, with either negotiation task goals or with an additional social goal. The group not given the social goal performed significantly better on most measures. However, students who reported having social goals, regardless of condition, learned significantly more than students who did not report social goals. This experiment provides evidence that while explicit goals may not be the right scaffold, social goals are important in learning cross-cultural negotiation from an educational game.

This paper describes a study in which student-created diagrams about arguments in an ill-defined domain were manually graded by two independent human graders. Findings include that the graders overall agreed with each other on their grades, but their agreement was lower than one would expect in well-defined domains, and higher for solutions of extreme quality.

Although a solid understanding of fractions is foundational in mathematics, the concept of fractions remains a challenging one. Previous research suggests that multiple graphical representations (MGRs) may promote learning of fractions. Specifically, we hypothesized that providing students with MGRs of fractions, in addition to the conventional symbolic notation, leads to better learning outcomes as compared to instruction incorporating only one graphical representation. We anticipated, however, that MGRs would make the students' task more challenging, since they must link the representations and distill from them a common concept or principle. Therefore, we hypothesized further that self-explanation prompts would help students benefit from working with MGRs. To investigate these hypotheses, we conducted a classroom study in which 112 6th-grade students used intelligent tutors for fraction conversion and fraction addition. The results of the study show that students learned more with MGRs of fractions than with a single representation, but only when prompted to self-explain how the graphics relate to the symbolic fraction representations.

Cognitive Tutors have been shown to lead to impressive improvement in student learning in a range of domains, including middle school mathematics. Most Cognitive Tutors focus on providing learning support for tasks involving problem solving. They typically emphasize facilitating the acquisition of procedural knowledge, on the assumption that the acquisition of conceptual knowledge will be supported outside the tutor through classroom activities or an accompanying textbook. Many researchers agree that expertise in a domain involves not just procedural knowledge, but also conceptual knowledge, as well as tight interconnections between the two. In order to provide two types of knowledge instruction within the same Intelligent Tutoring System, we are now building a novel type of Cognitive Tutor to bridge the existing gap between teaching conceptual and procedural instruction with the same educational software.

The development of graphical argument models is an active and growing area of research in Artificial Intelligence and Law. The aim is to develop models which may be readily used by legal professionals and novices to produce and parse arguments. If this goal is to be realized it is important to develop models that human reasoners can manipulate and assess consistently. We report on an ongoing study of graph agreement in the context of the LARGO system.

Diagrammatic models of argument are increasingly prominent in AI and Law. Unlike everyday language these models formalize many of the the components and relationships present in arguments and permit a more formal analysis of an arguments' structural weaknesses. Formalization, however, can raise problems of agreement. In order for argument diagramming to be widely accepted as a communications tool, individual authors and readers must be able to agree on the quality and meaning of a diagram as well as the role that key components play. This is especially problematic when arguers seek to map their diagrams to or from more conventional prose. In this paper we present results from a grader agreement study that we have conducted using LARGO diagrams. We then describe a detailed example of disagreement and highlight its implications for both our diagram model and modeling argument diagrams in general.

Previous research shows that video viewing (a frequent activity in language courses) is more effective when students receive guidance. We investigate how to support students in an on-line environment in acquiring cultural knowledge and intercultural competence by viewing clips from feature films from the target culture. To test the effectiveness of a set of attention-focusing techniques (pause-predict-ponder), some of which have been shown to be effective in other contexts, we created ICCAT, a simple tutor that enhances an existing classroom model for the development of intercultural competence. We ran a study in two French Online classrooms with 35 participants, comparing ICCAT versions with and without attention-focusing techniques. We found that the addition of the pause-predict-ponder seemed to guide students in acquiring cultural knowledge and significantly increased students' ability to reason from an intercultural perspective. We discuss possible implications for intelligent tutoring systems in such difficult and ill-defined domains.

This paper presents a process model of arguing with hypotheticals and uses it to explain examples of oral arguments before the U.S. Supreme Court that are like those employed in Socratic law teaching. The process model has been partially implemented in the LARGO (Legal ARgument Graph Observer) intelligent tutoring system. The program supports students in diagramming oral argument examples; its feedback on students' diagrammatic reconstructions of the examples enforces the expectations of the process model. The paper presents empirical evidence that features of the argument diagrams made with LARGO are correlated with independent measures of argumentation ability. The examples and empirical results support the model's explanatory and diagnostic utility.

This paper describes an intelligent tutoring system, LARGO, that helps students learn skills of legal reasoning with hypotheticals by analyzing oral arguments before the US Supreme Court. The skills involve proposing a rule-like test for deciding a case, posing hypotheticals to challenge the rule, and responding by analogizing or distinguishing the hypotheticals and/or modifying the proposed test. Students diagram arguments in a special-purpose graphical language and receive feedback in the form of reflection questions.

Intelligent Tutoring Systems (ITS) can both help improve student learning and serve as useful platforms for experiments in learning science [1,2]. But the difficulty of building or customizing ITSs has hindered their acceptance among educators and researchers [3]. The Cognitive Tutor Authoring Tools (CTAT) project aims to provide a suite of authoring tools that make tutor development more affordable by leveraging human-computer interaction and artificial intelligence techniques. Previous efforts on CTAT added the capability for nonprogrammers to create exampletracing tutors via a programming-by-demonstration technique that requires no coding [4]. While exampletracing tutors provide a student experience similar to that of the more general cognitive tutors, they also require that an author demonstrate and fully annotate each individual problem to be presented.

The transcripts of oral arguments before the US Supreme Court provide interesting opportunities from the viewpoint of legal education. As the pinnacle of legal argumentation, they illustrate, often in dramatic fashion, a sophisticated process of concept formation and testing driven by skillful posing of hypotheticals. Yet it is not easy to get beginning law students to understand the arguments and the underlying processes of hypothesis formation and testing. We introduce a novel project with the dual aims of developing an AI model of concept formation and testing as well as an intelligent tutoring system for beginning law students. We describe a planned experiment in which we will evaluate to what extent law students' study of the Supreme Court oral arguments can be improved by providing detailed and specific self-explanation prompts. It is hypothesized that detailed prompts to explain connections between tests, rationales, dimensions, and hypotheticals will help students to induce adequate mental models of concept formation processes.

Researchers in the field of AI and Law have developed a number of computational models of the arguments that skilled attorneys make based on past cases. However, these models have not accounted for the ways that attorneys use middle-level normative background knowledge (1) to organize multi-case arguments, (2) to reason about the significance of differences between cases, and (3) to assess the relevance of precedent cases to a given problem situation. We present a novel model, that accounts for these argumentation phenomena. An evaluation study showed that arguments about the significance of distinctions based on this model help predict the outcome of cases in the area of trade secrets law, confirming the quality of these arguments. The model forms the basis of an intelligent learning environment called CATO, which was designed to help beginning law students acquire basic argumentation skills. CATO uses the model for a number of purposes, including the dynamic generation of argumentation examples. In a second evaluation study, carried out in the context of an actual legal writing course, we compared instruction with CATO against the best traditional legal writing instruction. The results indicate that CATO's example-based instructional approach is effective in teaching basic argumentation skills. However, a more “integrated” approach appears to be needed if students are to achieve better transfer of these skills to more complex contexts. CATO's argumentation model and instructional environment are a contribution to the research fields of AI and Law, Case-Based Reasoning, and AI and Education.

CAT0 is an intelligent learning environment designed to help &@ting law students learn basic skills of making arguments with cases, through practice in theory-testing and argumentation tasks.CAT0 models ways in which experts compare and contrast cases, assess the significance of similarities and differences between cases in light of general domain knowledge, and use the same general knowledge to organize multi-case arguments by issues.CAT0 communicates its model to students by presenting dynamically-generated argumentation examples and reifying (i.e., making visible) argument structure.Also, the CAT0 Tools reduce some of the distracting complexity of the students' task.CAT0 employs a computational model of case-based argumentation that addresses eight basic argument moves and more elaborate multi-case arguments.The model includes a "Factor Hierarchy," which represents more nbstract.,but still domain-specific, legal knowledge about the meaning of the factors used to represent cases.CAT0 uses the Factor Hierarchy for a number of purposes, among them to organize multi-case arguments by issues and to make arguments about the significance of distinctions.To generate the latter, CAT0 strategically selects alternative interpretations of cases to elaborate a "deeper" (or more.abstract) contrast or parallel between cases.

We conducted an experiment to investigate whether a human tutor could employ the CATO model and instructional program to teach legal research and argumentation skills to beginning law stu&nts. The CATO model covem arguments comparing and contrasting casesin terms of factors, abstractionsof facts that tend to strengthen or weaken a party’s position on a legal claim. At the time of the experimen~ the CATO program comprised tools and resources that help apply the CATO model to specific problems, most importantly, a case database and tools for retrieving, displaying, and comparing casesin terms of factors. We compmed human-led itulruction with CATO against more traditional classroom instruction designed to teach the same skills, without the use of the CATO model or tools. The subjects were 17 fmtsemester students from the University of Pittsburgh Law School. We found that human-guided instruction with CATO was as good as classroom instruction, We also found that answers generated by the CATO program were scored higher than the students’ answers, suggestingthat the model can potentially be employed even more effectively to teach students. Examples drawn from protocols of CATO sessionsih.trate that students can use the CATO model to guide and facilitate the construction of arguments and often go beyond the model’s limitations, at least under the guidance of a human tutor.

To make legal arguments, one needs certain information about how to use cases effectively - dialectical information. In the broadest sense, dialectical information includes strategies for employing cases to justify legal conclusions (and responding to such justifications) and criteria for finding cases and deciding which cases to use. Making dialectical information explicit is important for teaching case-based argument. It is our experience that typically, law students do not have a very good set of dialectical strategies nor are they aware of the criteria. Even the most sophisticated legal information retrieval tools do not make such dialectical information explicit and assume that users have already learned it.

This paper describes a research project to devise and test an intelligent, case-baaed tutorial program for teaching law students to argue with cases.In order to present pedagogically interesting lessons and develop a Student Model, we have designed memory structures such as Argument Contexts and a hierarchy of Issues in Case-Baaed Legal Reaaoning.Using logical expressions in the knowledge representation language Loom, we also explicitly represent case-based argument concepts such as a case's being on point to a problem, more on point than another case, most on point of all the cases, a best case to cite, and a counterexample to another case.The program will be able to reason with the explicit concepts in selecting cases from a Case Library, assembling lessons and examples, analyzing student inputs, and in generating explanations and feedback.We hope to demonstrate empirically that, by providing law students a conceptual model of the criteria for selecting and describing precedents that would be useful in an argument, the tutorial program will help them to learn to select and apply cases more efficiently and to make more effective arguments.