The emerging field of multimodal learning analytics evaluates natural communication modalities (digital pen, speech, images) to identify domain expertise, learning, and learning-oriented precursors. Using the Math Data Corpus, this research investigated students' digital pen input as small groups collaborated on solving math problems. Compared with non-experts, findings indicated that domain experts have an opposite pattern of accelerating total written activity as problem difficulty increases, a lower written and spoken disfluency rate, and they express different content--including a higher ratio of nonlinguistic symbolic representations and structured diagrams to elemental marks. Implications are discussed for developing reliable multimodal learning analytics systems that incorporate digital pen input to automatically track the consolidation of domain expertise. This includes prediction based on a combination of activity patterns, fluency, and content analysis. New MMLA systems are expected to have special utility on cell phones, which already have multimodal interfaces and are the dominant educational platform worldwide.

One aim of multimodal learning analytics is to analyze rich natural communication modalities to identify domain expertise and learning rapidly and reliably. In this research, written and multimodal representations are analyzed from the Math Data Corpus, which involves multimodal data (digital pen, speech, images) on collaborating students as they solve math problems. Findings reveal that in 96-97% of cases the correctness of a group's solution was predictable in advance based on students' written work content. In addition, a linear regression revealed that 65% of the variance in individual students' domain expertise rankings could be accounted for based on their written work content. A multimodal content analysis based on both written and spoken input correctly predicted the dominant domain expert in a group 100% of the time, exceeding unimodal prediction rates. Further analysis revealed a reversal between experts and non-experts in the percentage of time that a match versus mismatch was present between their oral and written answer contributions, with non-experts demonstrating higher mismatches. Implications are discussed for developing reliable multimodal learning analytics systems that incorporate digital pen input to automatically track consolidation of domain expertise.

This paper provides documentation on dataset resources for establishing a new research area called multimodal learning analytics (MMLA). Research on this topic has the potential to transform the future of educational practice and technology, as well as computational techniques for advancing data analytics. The Math Data Corpus includes high-fidelity time-synchronized multimodal data recordings (speech, digital pen, images) on collaborating groups of students as they work together to solve mathematics problems that vary in difficulty level. The Math Data Corpus resources include initial coding of problem segmentation, problem-solving correctness, and representational content of students' writing. These resources are made available to participants in the data-driven grand challenge for the Second International Workshop on Multimodal Learning Analytics. The primary goal of this event is to analyze coherent signal, activity, and lexical patterns that can identify domain expertise and change in domain expertise early, reliably, and objectively, as well as learning-oriented precursors. An additional aim is to build an international research community in the emerging area of multimodal learning analytics by organizing a series of workshops that bring together multidisciplinary scientists to work on MMLA topics.

This article presents two studies investigating how computer interface affordances influence basic cognition, including ideational fluency, problem solving, and inferential reasoning. In one study comparing interfaces with different input capabilities, students expressed 56% more nonlinguistic representations (diagrams, symbols, numbers) when using pen interfaces. A linear regression confirmed that nonlinguistic communication directly mediated a substantial increase (38.5%) in students' ability to produce appropriate science ideas. In contrast, students expressed 41% more linguistic content when using a keyboard-based interface, which mediated a drop in science ideation. A follow-up study pursued the question of how interfaces that prime nonlinguistic communication so effectively facilitate cognition. This study examined the relation between students' expression of nonlinguistic representations and their inference accuracy when using analogous digital and non-digital pen tools. Perhaps surprisingly, the digital pen interface stimulated construction of more diagrams, more correct Venn diagrams, and more accurate domain inferences. Students' construction of multiple diagrams to represent a problem also directly suppressed overgeneralization errors, which were the most common inference failure. These research results reveal that computer interfaces have communications affordances which elicit communication patterns that can substantially stimulate or impede basic cognition. Implications are discussed for designing new digital tools for thinking, with an emphasis on nonlinguistic and especially spatial representations that are most poorly supported by current keyboard-based interfaces.