Actionability is a critical, but understudied, issue in learning analytics for driving impact on learning. This study investigated access and action-taking of 91 students in an online undergraduate statistics course who received analytics designed for actionability twice a week for five weeks in the semester. Findings showed high levels of access, but little direct action through the provided links. The major contribution of the study was the identification of unexpected indirect actions taken by students in response to the analytics which requires us to think (and look for evidence of impact) more broadly than has been done previously. The study also found that integrating analytics into existing learning tools and routines can increase access rates to the analytics, but may not guarantee meaningful engagement without better strategies to manage analytic timing. Together, this study advances an understanding of analytic actionability, calling for a broader examination of both direct and indirect actions within a larger learning ecosystem.

This study addresses the gap in knowledge about differences in how instructors use analytics to inform teaching by examining the ways that thirteen college instructors engaged with a set of university-provided analytics. Using multiple walk-through interviews with the instructors and qualitative inductive coding, two profiles of instructor analytics use were identified that were distinct from each other in terms of the goals of analytics use, how instructors made sense of and took actions upon the analytics, and the ways that ethical concerns were conceived. Specifically, one group of instructors used analytics to help students get aligned to and engaged in the course, whereas the other group used analytics to align the course to meet students’ needs. Instructors in both profiles saw ethical questions as central to their learning analytics use, with instructors in one profile focusing on transparency and the other on student privacy and agency. These findings suggest the need to view analytics use as an integrated component of instructor teaching practices and envision complementary sets of technical and pedagogical support that can best facilitate the distinct activities aligned with each profile.

Despite growing implementation of teacher-facing analytics in higher education, relatively little is known about the detailed processes through which instructors make sense of analytics in their teaching practices beyond their initial encounters with tools. This study unpacked the sensemaking process of thirteen instructors with analytic experience, using interviews that included walkthroughs of their analytics use. Qualitative inductive analysis was used to identify themes related to (1) the questions they asked of the analytics, (2) the techniques they used to interpret them, and (3) the challenges they encountered. Findings indicated that instructors went beyond a general curiosity to develop three types of questions of the analytics (goal-oriented, problem-oriented, and instruction modification questions). Instructors also used specific techniques to read and explain data by (a) developing expectations about the answers the analytics would provide, and (b) making comparisons to reveal student diversity, identify effects of instructional revision and diagnose issues. The study found instructors faced an initial learning curve when seeking and making use of relevant information, but also continued to revisit these challenges when they were not able to develop a routine of analytics use. These findings both contribute to a conceptual understanding of instructor analytic sensemaking and have practical implications for its systematic support.

Reflection assessment is a critical component of health professions education that can be used for personalized learning support. However, reflection assessment at scale remains a challenge due to the demanding nature of tasks and the common use of simplified criteria of quality. This study addressed this issue by developing a multi-dimensional automated assessment that uses linguistic models to classify reflections by overall quality (depth) and the presence of six constituent elements denoting quality (description, analysis, feeling, perspective, evaluation, and outcome). 1500 reflections from 369 dental students were manually coded to establish ground truth. Classifiers for each of the six elements were trained and tested based on linguistic features extracted using the LIWC tool applying both single-label and multi-label classification approaches. Classifiers for depth were built both directly from linguistic features and based on the presence of the six elements. Results showed that linguistic modeling can be used to reliably detect the presence of reflection elements and the level of depth. However, the depth classifier showed a heavy reliance on cognitive elements (description, analysis, and evaluation) rather than the others. These findings indicate the feasibility of implementing multidimensional automated assessment in health professions education and the need to reconsider how quality of reflection is conceptualized.