Generative artificial intelligence (GenAI) is increasingly embedded in higher education, yet little is known about how it shapes students’ cognitive engagement in authentic learning tasks. This study applies a learning analytics perspective to investigate the types and dynamics of cognition that emerge when students interact with GenAI during a question–answering task. We collected 294 GPT-4 assisted conversations comprising 618 student questions using a purpose-built system designed to support student GenAI questioning. Student–GenAI interactions were coded with Bloom’s revised taxonomy to capture levels of cognition, and then analyzed with two complementary learning analytics approaches: Epistemic Network Analysis (ENA), which identifies co-occurrence structures among cognitive categories, and Frequency-based Transition Network Analysis (FTNA), which models their sequential flow over time. The analyses revealed that prompts generated by the AI tool elicited higher cognitive processes, particularly Analyze with a spillover to Apply, whereas student-authored prompts clustered around Understand and Evaluate. FTNA showed that Remember commonly served as the entry point and Analyze as the main convergence node; although rare, Create occupied a central position when it appeared. Students often engaged in cyclical loops across Bloom’s categories rather than following a linear progression through the taxonomy. This work contributes to learning analytics by demonstrating how combining Bloom’s revised taxonomy with ENA and FTNA provides methodological innovations for capturing both the structural and temporal dynamics of cognition in GenAI-supported learning environments. Pedagogically, the findings highlight how learning analytics can inform the design of AI-generated and student-authored prompts to foster deeper cognitive engagement, thus offering actionable insights for empowering students and guiding educators in higher education contexts.

With generative artificial intelligence driving the growth of dialogic data in education, automated coding is a promising direction for learning analytics to improve efficiency. This surge highlights the need to understand the nuances of student-AI interactions, especially those rare yet crucial. However, automated coding may struggle to capture these rare codes due to imbalanced data, while human coding remains time-consuming and labour-intensive. The current study examined the potential of large language models (LLMs) to approximate or replace humans in deductive, theory-driven coding, while also exploring how human–AI collaboration might support such coding tasks at scale. We compared the coding performance of small transformer classifiers (e.g., BERT) and LLMs in two datasets, with particular attention to imbalanced head–tail distributions in dialogue codes. Our results showed that LLMs did not outperform BERT-based models and exhibited systematic errors and biases in deductive coding tasks. We designed and evaluated a human–AI collaborative workflow that improved coding efficiency while maintaining coding reliability. Our findings reveal both the limitations of LLMs – especially their difficulties with semantic similarity and theoretical interpretations – and the indispensable role of human judgment, while demonstrating the practical promise of human–AI collaborative workflows for coding.

Assessing handwritten mathematical solutions is essential for identifying students’ weaknesses and fostering personalized learning. However, scaling such assessment remains challenging for Learning Analytics, which has traditionally focused on digital or typed data. The current study investigated the potential of Large Language Models (LLMs) to automating the assessment of handwritten mathematical solutions and explore how they can be incorporated into large scale learning analytics pipelines. We curated 300 student solution images, annotated them using to a taxonomy of math error types, and compared open-source (Qwen2.5-7B and Gemma3 12B-IT) and closed-source (Gemini 2.0 Flash and GPT-4) LLMs. Two prompting strategies were tested: from adapted from related work and one tailed to the taxonomy of math error types using established prompt design principles. The results revealed that LLMs, particularly Gemini, achieved strong to moderate performance in diagnosing and classifying student errors, while exposing recurring model specific errors. These findings highlight both the promise and limitations of LLMs for integrating handwritten work in LA and recommend that learning analytics practitioners and researchers combine careful model selection, principled prompt design, and error-level analysis to develop AI-powered LA systems that are accurate, equitable, and pedagogically actionable.

Simulation-based learning is central to healthcare education, yet its effectiveness depends on high-quality debriefing. Traditional debriefs often overlook detailed team dialogue dynamics. Advances in large language models (LLMs) open new possibilities for learning analytics (LA) by automatically coding and visualising teamwork behaviours from dialogue data. This study investigates the effectiveness of different prompting strategies for LLM-based coding, comparing their performance and environmental impacts (CO2e) to identify approaches suitable for transfer into professional practice. Building on these results, we evaluate the generalisability of the optimised model from university student simulations to in-situ, hospital settings, and explore how healthcare professionals perceive the interpretability, usefulness, and trustworthiness of LLM-driven learning analytics in professional learning debriefs. Findings illustrate that responsible uses of AI can help extend LA beyond a controlled university environment into an authentic, in-hospital healthcare context, offering potentially scalable and sustainable support for reflective practice and professional development.

The learning analytics (LA) community has recently reached two important milestones: celebrating the 15th LAK conference and updating the 2011 definition of LA to reflect the 15 years of changes in the discipline. However, despite LA’s growth, little is known about how research topics, funding, and collaboration, as well as the relationships among them, have developed within the community over time. This study addressed this gap by analyzing all 936 full and short papers published at LAK over a 15-year period using unsupervised machine learning, natural language processing, and network analytics. The analysis revealed a stable core of prolific authors alongside high turnover of newcomers, systematic links between funding sources and research directions, and six enduring topical centers that remain globally shared but vary in prominence across countries. These six topical centers, which encompass LA research, are: self-regulated learning, dashboards and theory, social learning, automated feedback, multimodal analytics, and outcome prediction. Our findings highlight key challenges for the future: widening participation, reducing dependency on a narrow set of funders, and ensuring that emerging research trajectories remain responsive to educational practice and societal needs.

Assessment of medication history-taking has traditionally relied on human observation, limiting scalability and detailed performance data. While Generative AI (GenAI) platforms enable extensive data collection and learning analytics provide powerful methods for analyzing educational traces, these approaches remain largely underexplored in pharmacy clinical training. This study addresses this gap by applying learning analytics to understand how students develop clinical communication competencies with GenAI-powered virtual patients—a crucial endeavor given the diversity of student cohorts, varying language backgrounds, and the limited opportunities for individualized feedback in traditional training settings. We analyzed 323 students’ interaction logs across Australian and Malaysian institutions, comprising 50,871 coded utterances from 1,487 student-GenAI dialogues. Combining Epistemic Network Analysis to model inquiry co-occurrences with Sequential Pattern Mining to capture temporal sequences, we found that high performers demonstrated strategic deployment of information recognition behaviors. Specifically, high performers centered inquiry on recognizing clinically relevant information, integrating rapport-building and structural organization, while low performers remained in routine question-verification loops. Demographic factors including first-language background, prior pharmacy work experience, and institutional context, also shaped distinct inquiry patterns. These findings reveal inquiry patterns that may indicate clinical reasoning development in GenAI-assisted contexts, providing methodological insights for health professions education assessment and informing adaptive GenAI system design that supports diverse learning pathways.

The emergence of generative artificial intelligence (GenAI) has created unprecedented opportunities to provide individualized learning support in classrooms at scale. However, concerns have been raised that students may engage with these tools in ways that do not support learning, such as asking for direct answers. Moreover, student engagement with GenAI Tutors may vary across instructional contexts, potentially leading to unequal learning experiences. In this study, we utilized de-identified student interaction logs from a large-scale GenAI Tutor and the learning management system in which it is embedded. We systematically examined student engagement (N = 11,406) with the tool across 200 classes in ten post-secondary institutions through a two-stage pipeline: First, we aggregated student-GenAI conversations to sessions and identified four distinct engagement types. In particular, 10.4% of them were ''shallow engagement'' where copy-pasting behavior was prevalent. Then, at the student level, we showed that students transitioned across engagement types over time. However, students who exhibited shallow engagement were more likely to remain in this mode, whereas those who engaged deeply transitioned more flexibly across modes. Finally, at both the session and student levels, we show substantial heterogeneity in student engagement across institution selectivity and course disciplines. In particular, students from less selective institutions were more likely to exhibit shallow engagement. This study advances understanding of real-world GenAI Tutor use by identifying the prevalence, temporal dynamics, and institutional variation of student engagement. It also contributes a scalable two-stage analytic framework for analyzing GenAI interaction data in educational settings and beyond.

The increasing incidents of data breaches and personal data misuse highlight the urgent need for robust identity management systems. Self-Sovereign Identity (SSI) emerges as the future solution for digital identity management, underpinned by anonymous credentials (AC) and the distributed ledger technology (DLT) for security measures. However, current SSI models only achieve partial decentralization and none of them can fully meet the complex security requirements of real-world applications. In this paper, we address these limitations by constructing a Decentralized Anonymous Credential (DAC) scheme inspired by large universe attribute-based cryptographic primitives. Building on this foundation, we design a distributed identity management system (DisIMS), a comprehensive SSI system built on blockchain, achieving attribute flexibility, anonymity, unlinkability, revocability and selective disclosure. Compared to earlier blockchain-based identity management systems, our DisIMS allows users to selectively link previous transactions to generate verifiable eligibility proofs for the current transaction without leaking their real identities. We also implement DisIMS on both permissioned (Hyperledger Fabric v2.5) and permissionless (Ethereum Sepolia testnet) blockchains. Experimental results show that batch verification outperforms single verification by reducing execution times by approximately 76% to 81% on Hyperledger Fabric and 50% to 71% on Ethereum, based on 200 tests with 10 to 50 credentials containing 50 attributes each, which demonstrates DisIMS practicality for real-world batch verification scenarios.

The emerging concept of data storytelling (DS) suggests that enhancing visualisations with annotations and narratives can make complex data more insightful than conventional visualisations. Previous works found that DS-enhanced visualisations are more effective than conventional visualisations for simple tasks like identifying key data points or the main message. However, no previous work has explored the extent to which DS enhancements influence task completion across different levels of cognitive complexity. We address this gap by presenting the results of a study where 128 participants completed tasks based on four visualisations (two line charts and two choropleth maps, either with or without DS elements) spanning a range of complexity based on Bloom's taxonomy, which has been applied in data visualisation to categorise tasks hierarchically from lower to higher-order thinking. Results suggest that while DS-enhanced visualisations effectively support lower-order tasks (finding data points and understanding insights), they don't necessarily aid the correct completion of higher-order tasks (application, analysis, evaluation and creation). However, DS enhancements improve how efficiently participants complete complex tasks.

Continually updating model-based indexes in generative retrieval with new documents remains challenging, as full retraining is computationally expensive and impractical under resource constraints.We propose MixLoRA-DSI, a novel framework that combines an expandable mixture of Low-Rank Adaptation experts with a layer-wise out-of-distribution (OOD)driven expansion strategy.Instead of allocating new experts for each new corpus, our proposed expansion strategy enables sublinear parameter growth by selectively introducing new experts only when significant number of OOD documents are detected.Experiments on NQ320k and MS MARCO Passage demonstrate that MixLoRA-DSI outperforms full-model update baselines, with minimal parameter overhead and substantially lower training costs. 1

High staff turnover in higher education often burdens teachers with laborious handovers of teaching tasks every semester. To boost teachers' workflow efficiency, we present an innovative knowledge management tool that allows experienced teachers to seamlessly capture task steps (i.e., denoted as ShareFlow) that can be subsequently recommended to novices via proactive push, all happening during teachers' natural workflow to minimize disruptions. We conducted a controlled experiment with 30 participants and compared our tool against a state-of-the-art baseline knowledge management system powered by a large language model (Claude 3 Haiku). We found that our knowledge management tool reduced task completion time and improved task quality (with statistical significance). Feedback from the participants also indicated the high usability of our tool, suggesting its strong potential for practical adoption for improving teacher workflows.

Team teaching in higher education can be challenging, especially for educators managing large classes with limited pedagogical training and few opportunities to reflect on their practices. Emerging sensing technologies and analytics can capture and analyse patterns of collaboration, communication, and movement of team teaching. Yet, few studies have presented these data to educators for reflection. To address this gap, we examine the benefits, challenges, and concerns of presenting multimodal teaching data (positional, audio, and spatial pedagogy observations) to educators via the TeamTeachingViz dashboard. We evaluated TeamTeachingViz in an authentic classroom context where educators explored their own data and team teaching strategies. Multimodal data was collected from 36 in-the-wild classroom sessions involving 12 educators grouped in various combinations over 4 weeks, followed by semi-structured interviews to reflect on their practices. Findings suggest that educators improved their self-awareness by using data-driven insights to understand their movements and interactions, enabling continuous improvement in team teaching. However, they noted the need for additional data, such as student behaviours and speech content, to better contextualise these insights.

Temporal patterns play a significant role in understanding dynamic changes in Self-regulated Learning (SRL) engagement over time. Several previous studies have proposed approaches for automated detection of SRL strategies through analysis of temporal patterns. However, these approaches are mostly focused on the analysis of patterns in sequential ordering of SRL processes. This offers a useful yet limited temporal perspective to SRL. As noted in the literature, temporality of SRL has two dimensions - passage of time and ordering of events. To address this gap, this paper specifically proposes a time series approach that can automatically detect SRL strategies by accounting for both dimensions of temporality. Our approach also explores when specific processes occur and how learners engage metacognitively or cognitively with learning tasks. In particular, this study investigated SRL engagement as students composed essays using multiple sources within a 120-minute time frame. The results indicated that five distinct strategies with varying levels of engagement were detected. The correlation between these identified strategies and students' scores was not statistically significant; however, further exploration revealed that students who adopted a specific strategy could outperform other groups based on obtained scores. We also noticed additional factors that had a positive effect on learners' performance.

While the capacity to self-regulate has been found to be crucial for secondary school students, prior studies often rely on self-report surveys and think-aloud protocols that present notable limitations in capturing self-regulated learning (SRL) processes. This study advances the understanding of SRL in secondary education by using trace data to examine SRL processes during multi-source writing tasks, with higher education participants included for comparison. We collected fine-grained trace data from 66 secondary school students and 59 university students working on the same writing tasks within a shared SRL-oriented learning environment. The data were labelled using Bannert's validated SRL coding scheme to reflect specific SRL processes, and we examined the relationship between these processes, essay performance, and educational levels. Using epistemic network analysis (ENA) to model and visualise the interconnected SRL processes in Bannert's coding scheme, we found that: (a) secondary school students predominantly engaged in three SRL processes - Orientation, Re-reading, and Elaboration/Organisation; (b) high-performing secondary students engaged more in Re-reading, while low-performing students showed more Orientation process; and (c) higher education students exhibited more diverse SRL processes such as Monitoring and Evaluation than their secondary education counterparts, who heavily relied on following task instructions and rubrics to guide their writing. These findings highlight the necessity of designing scaffolding tools and developing teacher training programs to enhance awareness and development of SRL skills for secondary school learners.

Learning analytics dashboards (LADs) simplify complex learner data into accessible visualisations, providing actionable insights for educators and students. However, their educational effectiveness has not always matched the sophistication of the technology behind them. Explanatory and interactive LADs, enhanced by generative AI (GenAI) chatbots, hold promise by enabling dynamic, dialogue-based interactions with data visualisations and offering personalised feedback through text. Yet, the effectiveness of these tools may be limited by learners' varying levels of GenAI literacy, a factor that remains underexplored in current research. This study investigates the role of GenAI literacy in learner interactions with conventional (reactive) versus scaffolding (proactive) chatbot-assisted LADs. Through a comparative analysis of 81 participants, we examine how GenAI literacy is associated with learners' ability to interpret complex visualisations and their cognitive processes during interactions with chatbot-assisted LADs. Results show that while both chatbots significantly improved learner comprehension, those with higher GenAI literacy benefited the most, particularly with conventional chatbots, demonstrating diverse prompting strategies. Findings highlight the importance of considering learners' GenAI literacy when integrating GenAI chatbots in LADs and educational technologies. Incorporating scaffolding techniques within GenAI chatbots can be an effective strategy, offering a more guided experience that reduces reliance on learners' GenAI literacy.

In computer-based learning environments (CBLEs), adopting effective self-regulated learning (SRL) strategies requires sophisticated coordination of multiple SRL processes. While various studies have proposed adaptive SRL scaffolds (i.e. real-time advice on adopting effective SRL processes) and embedded them in CBLEs to facilitate learners' effective use of SRL strategies, two key research gaps remain. First, there is a lack of research on SRL scaffolds that are based on continuous assessment of both learners' SRL processes and learning conditions (e.g., awareness of learning resources) to provide adaptive support. Second, current analytics-based scaffolding mechanisms lack the scalability needed to effectively address multiple learning conditions. Integration of analytics of SRL with generative artificial intelligence (GenAI) can provide scalable scaffolding for real-time SRL processes and evolving conditions. Yet, empirical studies implementing and evaluating effects of this integration remain scarce. To address these limitations, we conducted a randomized control trial, assigning participants to three groups (control, process only, and process with condition groups) to investigate the effects of using GenAI to turn insights from real-time analytics about students' SRL processes and conditions into adaptive scaffolds. The results demonstrate that integrating real-time analytics with GenAI in adaptive SRL scaffolds - addressing both SRL processes and dynamic conditions - promotes more metacognitive learning patterns compared to the control and process-only groups. In addition, the learners showed varying levels of compliance with analytics-based GenAI scaffolds, and this was also reflected in how the learners coordinated their SRL processes, particularly in the performance phase of SRL. This study contributes to the literature by designing, implementing, and evaluating the impact of adaptive scaffolds on learners' SRL processes using real-time analytics with GenAI.

Assessing short answers in educational settings is challenging due to the need for scalability and accuracy, which led to the field of Automatic Short Answer Grading (ASAG). Traditional machine learning models, such as ensemble and embeddings, have been widely researched in ASAG, but they often suffer from generalizability issues. Recently, Large Language Models (LLMs) emerged as an alternative to optimize ASAG systems. However, previous research has failed to present a comprehensive analysis of LLMs' performance powered by prompt engineering strategies and compare its capabilities to traditional models. This study presents a comparative analysis between traditional machine learning models and GPT-4 in the context of ASAG. We investigated the effectiveness of different models and text representation techniques and explored prompt engineering strategies for LLMs. The results indicate that traditional machine learning models outperform LLMs. However, GPT-4 showed promising capabilities, especially when configured with optimized prompt components, such as few-shot examples and clear instructions. This study contributes to the literature by providing a detailed evaluation of LLM performance compared to traditional machine learning models in a multilingual ASAG context, offering insights for developing more efficient automatic grading systems.

Question-answering systems facilitate adaptive learning and respond to student queries, making education more responsive. Despite that, challenges such as natural language understanding and context management complicate their widespread adoption, where Large Language Models (LLMs) offer a promising solution. However, existing research is predominantly focused on English, proprietary models, and often limited to a single question type, subject, or skill, leaving a gap in understanding LLMs' performance in languages like Brazilian Portuguese and across questions of various characteristics. This study investigates how LLMs could be integrated in an educational question-answering system efficiently to answer different question types (multiple-choice, cloze, open-ended), subjects (mathematics and Portuguese language), and skills (summation/subtraction, multiplication, interpretation, and grammar), evaluating answers by GPT-4 - the main LLM at the time of writing - and Sabiá - the open-source Brazilian Portuguese LLM - based on grades assigned by two experienced teachers. Overall, both LLMs demonstrated strong overall performance, with mean scores close to 9.8 out of 10. However, specific challenges emerged, with distinct strengths and weaknesses observed for each model, such as GPT-4's error in a multiple-choice subtraction question and Sabiá's misinterpretation of a cloze question.

Multimodal Learning Analytics (MMLA) leverages advanced sensing technologies and artificial intelligence to capture complex learning processes, but integrating diverse data sources into cohesive insights remains challenging. This study introduces a novel methodology for integrating latent class analysis (LCA) within MMLA to map monomodal behavioural indicators into parsimonious multimodal ones. Using a high-fidelity healthcare simulation context, we collected positional, audio, and physiological data, deriving 17 monomodal indicators. LCA identified four distinct latent classes: Collaborative Communication, Embodied Collaboration, Distant Interaction, and Solitary Engagement, each capturing unique monomodal patterns. Epistemic network analysis compared these multimodal indicators with the original monomodal indicators and found that the multimodal approach was more parsimonious while offering higher explanatory power regarding students' task and collaboration performances. The findings highlight the potential of LCA in simplifying the analysis of complex multimodal data while capturing nuanced, cross-modality behaviours, offering actionable insights for educators and enhancing the design of collaborative learning interventions. This study proposes a pathway for advancing MMLA, making it more parsimonious and manageable, and aligning with the principles of learner-centred education.

Students are increasingly relying on Generative AI (GAI) to support their writing - a key pedagogical practice in education. In GAI-assisted writing, students can delegate core cognitive tasks (e.g., generating ideas and turning them into sentences) to GAI while still producing high-quality essays. This creates new challenges for teachers in assessing and supporting student learning, as they often lack insight into whether students are engaging in meaningful cognitive processes during writing or how much of the essay's quality can be attributed to those processes. This study aimed to help teachers better assess and support student learning in GAI-assisted writing by examining how different writing behaviors, especially those indicative of meaningful learning versus those that are not, impact essay quality. Using a dataset of 1,445 GAI-assisted writing sessions, we applied the cutting-edge method, X-Learner, to quantify the causal impact of three GAI-assisted writing behavioral patterns (i.e., seeking suggestions but not accepting them, seeking suggestions and accepting them as they are, and seeking suggestions and accepting them with modification) on four measures of essay quality (i.e., lexical sophistication, syntactic complexity, text cohesion, and linguistic bias). Our analysis showed that writers who frequently modified GAI-generated text - suggesting active engagement in higher-order cognitive processes - consistently improved the quality of their essays in terms of lexical sophistication, syntactic complexity, and text cohesion. In contrast, those who often accepted GAI-generated text without changes, primarily engaging in lower-order processes, saw a decrease in essay quality. Additionally, while human writers tend to introduce linguistic bias when writing independently, incorporating GAI-generated text - even without modification - can help mitigate this bias.

Productively engaging in SRL is challenging for learners since it involves coordinating multiple motivational, affective, cognitive, and metacognitive processes. Researchers have investigated methods to adaptively scaffold learners' productive engagement using SRL processes automatically captured by SRL detectors. However, most previous studies relied solely on the frequency of SRL processes to drive adaptive scaffolds (e.g., feedback, hints), possibly missing the sequential characteristics inherent to self-regulation, a crucial dimension of productive SRL. To address this gap, this study analysed the impact of sequential transitions between multiple SRL processes on learners' performance on a reading-writing task with a hypermedia environment called Flora. A sample of 66 secondary-school learners completed the task and trace data were collected. Grounded in the COPES model of SRL, a rule-based SRL detector was employed to capture SRL processes from collected trace data. We employed a method combining logistic regression with ordered network analysis (ONA) to analyse the transitions between the detected SRL processes. This exploratory study revealed several influential transitions to learners' performance in different temporal learning blocks of self-regulation. The implications suggest the potential of using COPES SRL process transitions to drive adaptive scaffolds to facilitate engagement in productive SRL, benefiting performance outcomes in hypermedia environments.

Knowledge management involves finding, expanding, and using knowledge in an organisation to achieve goals. Its role is crucial in higher education to improve problem-solving, research, and teaching by acquiring, sharing, and applying knowledge. Higher education institutions can promote knowledge management through Communities of Practice, but doing so remains challenging due to cultural, organisational, and technological reasons. We present findings of the first step of co-design workshops with authentic higher education teaching teams that sought to understand (a) their practices as a community and any motivators and impediments to their community development; (b) how they perceived the tools they use for knowledge management; and (c) the kinds of tools they believed could help them better conduct knowledge management and develop as Communities of Practice. Our findings suggested four essential design requirements and informed our development of a new tool to support the knowledge management needs of higher education teaching teams.

Automatic Essay Scoring (AES) is a well-established educational pursuit that employs machine learning to evaluate student-authored essays. While much effort has been made in this area, current research primarily focuses on either (i) boosting the predictive accuracy of an AES model for a specific prompt (i.e., developing prompt-specific models), which often heavily relies on the use of the labeled data from the same target prompt; or (ii) assessing the applicability of AES models developed on non-target prompts to the intended target prompt (i.e., developing the AES models in a cross-prompt setting). Given the inherent bias in machine learning and its potential impact on marginalized groups, it is imperative to investigate whether such bias exists in current AES methods and, if identified, how it intervenes with an AES model's accuracy and generalizability. Thus, our study aimed to uncover the intricate relationship between an AES model's accuracy, fairness, and generalizability, contributing practical insights for developing effective AES models in real-world education. To this end, we meticulously selected nine prominent AES methods and evaluated their performance using seven distinct metrics on an open-sourced dataset, which contains over 25,000 essays and various demographic information about students such as gender, English language learner status, and economic status. Through extensive evaluations, we demonstrated that: (1) prompt-specific models tend to outperform their cross-prompt counterparts in terms of predictive accuracy; (2) prompt-specific models frequently exhibit a greater bias towards students of different economic statuses compared to cross-prompt models; (3) in the pursuit of generalizability, traditional machine learning models (e.g., SVM) coupled with carefully engineered features hold greater potential for achieving both high accuracy and fairness than complex neural network models.

The recent large language models (LLMs), e.g., ChatGPT, have been able to generate human-like and fluent responses when provided with specific instructions. While admitting the convenience brought by technological advancement, educators also have concerns that students might leverage LLMs to complete their writing assignments and pass them off as their original work. Although many AI content detection studies have been conducted as a result of such concerns, most of these prior studies modeled AI content detection as a classification problem, assuming that a text is either entirely human-written or entirely AI-generated. In this study, we investigated AI content detection in a rarely explored yet realistic setting where the text to be detected is collaboratively written by human and generative LLMs (termed as hybrid text for simplicity). We first formalized the detection task as identifying the transition points between human-written content and AI-generated content from a given hybrid text (boundary detection). We constructed a hybrid essay dataset by partially and randomly removing sentences from the original student-written essays and then instructing ChatGPT to fill in for the incomplete essays. Then we proposed a two-step detection approach where we (1) separated AI-generated content from human-written content during the encoder training process; and (2) calculated the distances between every two adjacent prototypes (a prototype is the mean of a set of consecutive sentences from the hybrid text in the embedding space) and assumed that the boundaries exist between the two adjacent prototypes that have the furthest distance from each other. Through extensive experiments, we observed the following main findings: (1) the proposed approach consistently outperformed the baseline methods across different experiment settings; (2) the encoder training process (i.e., step 1 of the above two-step approach) can significantly boost the performance of the proposed approach; (3) when detecting boundaries for single-boundary hybrid essays, the proposed approach could be enhanced by adopting a relatively large prototype size (i.e., the number of sentences needed to calculate a prototype), leading to a 22% improvement (against the best baseline method) in the In-Domain evaluation and an 18% improvement in the Out-of-Domain evaluation.

Data storytelling (DS) is rapidly gaining attention as an approach that integrates data, visuals, and narratives to create data stories that can help a particular audience to comprehend the key messages underscored by the data with enhanced efficiency and effectiveness. It is been posited that DS can be especially advantageous for audiences with limited visualisation literacy, by presenting the data clearly and concisely. However, empirical studies confirming whether data stories indeed provide these benefits over conventional data visualisations are scarce. To bridge this gap, we conducted a study with 103 participants to determine whether DS indeed improve both efficiency and effectiveness in tasks related to information retrieval and insights comprehension. Our findings suggest that data stories do improve the efficiency of comprehension tasks, as well as the effectiveness of comprehension tasks that involve a single insight, compared with conventional visualisations. Interestingly, these benefits were not associated with participants’ visualisation literacy.

Learning analytics (LA) holds a promise to transform education by utilizing data for evidence-based decision-making. Yet, its application in early childhood education (ECE) remains relatively under-explored. ECE plays a crucial role in fostering fundamental numeracy and literacy skills. While standardized tests was intended to be used to monitor student progress, they have been increasingly assumed summative and high-stake due to the substantial impact. The pressures in succeeding in such standardized tests have been well-documented to negatively affect both students and teachers. Attempting to ease such stress and better support students and teachers, the current study delved into the LA potential for predicting standardized test performance using formative assessments. Beyond predictive accuracy, the study addressed ethical considerations related to fairness to uncover potential risks associated with LA adoption. Our findings revealed a promising opportunity to empower teachers and schools with more time and room to help students better prepared based on predictions obtained earlier before standardized tests. Notably, bias can be significantly observed in predictions for students with disabilities even they have same actual competence compared to students without disabilities. In addition, we noticed that inclusion of demographic attribute had no significant impact on the predictive accuracy, and not necessarily exacerbate the overall predictive bias, but may significantly affect the predictions received by certain demographic subgroups (e.g., students with different types of disability).

There is a growing interest in creating Learning Analytics (LA) systems that incorporate student perspectives. Yet, many LA systems still lean towards a technology-centric approach, potentially overlooking human values and the necessity of human oversight in automation. Although some recent LA studies have adopted a human-centred design stance, there is still limited research on establishing safe, reliable, and trustworthy systems during the early stages of LA design. Drawing from a newly proposed framework for human-centred artificial intelligence, we introduce SLADE, a method for ideating and identifying features of human-centred LA systems that balance human control and computer automation. We illustrate SLADE’s application in designing LA systems to support collaborative learning in healthcare. Twenty-one third-year students participated in design sessions through SLADE’s four steps: i) identifying challenges and corresponding LA systems; ii) prioritising these LA systems; iii) ideating human control and automation features; and iv) refining features emphasising safety, reliability, and trustworthiness. Our results demonstrate SLADE’s potential to assist researchers and designers in: 1) aligning authentic student challenges with LA systems through both divergent ideation and convergent prioritisation; 2) understanding students’ perspectives on personal agency and delegation to teachers; and 3) fostering discussions about the safety, reliability, and trustworthiness of LA solutions.

We propose a learning analytics-based methodology for assessing the collaborative writing of humans and generative artificial intelligence. Framed by the evidence-centered design, we used elements of knowledge-telling, knowledge transformation, and cognitive presence to identify assessment claims; we used data collected from the CoAuthor writing tool as potential evidence for these claims; and we used epistemic network analysis to make inferences from the data about the claims. Our findings revealed significant differences in the writing processes of different groups of CoAuthor users, suggesting that our method is a plausible approach to assessing human-AI collaborative writing.

Advancements in Multimodal Learning Analytics (MMLA) have the potential to enhance the development of effective teamwork skills and foster reflection on collaboration dynamics in physical learning environments. Yet, only a few MMLA studies have closed the learning analytics loop by making MMLA solutions immediately accessible to educators to support reflective practices, especially in authentic settings. Moreover, deploying MMLA solutions in authentic settings can bring new challenges beyond logistic and privacy issues. This paper reports the design and use of TeamSlides, a multimodal teamwork analytics dashboard to support teacher-guided reflection. We conducted an in-the-wild classroom study involving 11 teachers and 138 students. Multimodal data were collected from students working in team healthcare simulations. We examined how teachers used the dashboard in 22 debrief sessions to aid their reflective practices. We also interviewed teachers to discuss their perceptions of the dashboard’s value and the challenges faced during its use. Our results suggest that the dashboard effectively reinforced discussions and augmented teacher-guided reflection practices. However, teachers encountered interpretation conflicts, sometimes leading to mistrust or misrepresenting the information. We discuss the considerations needed to overcome these challenges in MMLA research.

Individual behavioral engagement is an important indicator of active learning in collaborative settings, encompassing multidimensional behaviors mediated through various interaction modes. Little existing work has explored the use of multimodal process data to understand individual behavioral engagement in face-to-face collaborative learning settings. In this study we bridge this gap, for the first time, introducing a heterogeneous tripartite network approach to analyze the interconnections among multimodal process data in collaborative learning. Students’ behavioral engagement strategies are analyzed based on their interaction patterns with various spatial locations and verbal communication types using a heterogeneous tripartite network. The multimodal collaborative learning process data were collected from 15 teams of four students. We conducted stochastic blockmodeling on a projection of the heterogeneous tripartite network to cluster students into groups that shared similar spatial and oral engagement patterns. We found two distinct clusters of students, whose characteristic behavioural engagement strategies were identified by extracting interaction patterns that were statistically significant relative to a multinomial null model. The two identified clusters also exhibited a statistically significant difference regarding students’ perceived collaboration satisfaction and teacher-assessed team performance level. This study advances collaboration analytics methodology and provides new insights into personalized support in collaborative learning.

Dashboards are increasingly used in education to provide teachers and students with insights into learning. Yet, existing dashboards are often criticised for their failure to provide the contextual information or explanations necessary to help students interpret these data. Data Storytelling (DS) is emerging as an alternative way to communicate insights providing guidance and context to facilitate students’ interpretations. However, while data stories have proven effective in prompting students’ reflections, to date, it has been necessary for researchers to craft the stories rather than enabling teachers to do this by themselves. This can make this approach more feasible and scalable while also respecting teachers’ agency. Based on the notion of DS, this paper presents a DS editor for teachers. A study was conducted in two universities to examine whether the editor could enable teachers to create stories adapted to their learning designs. Results showed that teachers appreciated how the tool enabled them to contextualise automated feedback to their teaching needs, generating data stories to support student reflection.

To meet the growing demand for future professionals who can present information to an audience and create quality written products, educators are increasingly assigning writing assignments that require students to gather information from multiple sources, reorganise and reinterpret knowledge from source materials, and plan for rhetorical structure goals in order to meet the task requirements. When evaluating an essay coherence, scorers manually look for the presence of required rhetorical categories, which takes time. Supervised Machine Learning (ML) techniques have proven to be an effective tool for automatic detection of rhetorical categories that approximate students’ cognitive engagement with source information. Previous studies that addressed this problem used relatively small datasets and reported relatively low kappa scores for accuracy, limiting the use of such models in real-world scenarios. Moreover, to empower educators to effectively evaluate the overall quality of students’ writing, the associations between the sequential patterns of rhetorical categories in students’ writing and writing performance must be examined, which remains largely unexplored in educational domain. Therefore, to fill these gaps, our study aimed to i) investigate the impact of data augmentation approaches on the performance of deep learning algorithms in classifying rhetorical categories in student essays according to Bloom‘s taxonomy ii) and explore the sequential patterns of rhetorical categories in students’ writing that can influence writing performance. Our findings showed that deep learning-based model BERT on Easy Data Augmentation (EDA) based augmented data achieved 20% higher Cohen’s kappa than normal (non-augmented) data, and we discovered that students in different performance groups were statistically different in terms of rhetorical patterns. Our proposed study is valuable in terms of building a data analytic foundation that can be used to create formative feedback on students’ writings based on the patterns of rhetorical categories to improve essay quality.

A sophisticated grasp of self-regulated learning (SRL) skills has become essential for learners in computer-based learning environment (CBLE). One aspect of SRL is the plan-making process, which, although emphasized in many SRL theoretical frameworks, has attracted little research attention. Few studies have investigated the extent to which learners complied with their planned strategies, and whether making a strategic plan is associated with actual strategy use. Limited studies have examined the role of prior knowledge in predicting planned and actual strategy use. In this study, we developed a CBLE to collect trace data, which were analyzed to investigate learners’ plan-making process and its association with planned and actual strategy use. Analysis of prior knowledge and trace data of 202 participants indicated that 1) learners tended to adopt strategies that significantly deviated from their planned strategies, 2) the level of prior knowledge was associated with planned strategies, and 3) neither the act of plan-making nor prior knowledge predicted actual strategy use. These insights bear implications for educators and educational technologists to recognise the dynamic nature of strategy adoption and to devise approaches that inspire students to continually revise and adjust their plans, thereby strengthening SRL.

Temporality in Self-Regulated Learning (SRL) has two perspectives: one as a passage of time and the other as an ordered sequence of events. Each of these conceptions is distinct and requires independent considerations. Only a single analytic method is not sufficient in adequately capturing both these facets of temporality. Yet, most research uses a single method in temporally-focused SRL research, and those that use multiple methods do not address both aspects of temporality. We propose CTAM4SRL, a consolidated temporal analytic method which combines advanced data visualisation, network analysis and pattern mining to capture both facets of temporality. We employ CTAM4SRL in a cohort of 36 learners engaged in a reading-writing activity. Using CTAM4SRL, we were able to provide a rich temporal explanation of the interplay of the self-regulatory processes of the learners. We were further able to identify differences in SRL behaviours in high and low performers in terms of their approach to learning comprising deep and surface strategies. High performers were able to more selectively and strategically combine deep and surface learning strategies when compared to low scorers– a behaviour which was only hypothesised in SRL literature previously, but now has empirical support provided by our consolidated analytic method.

Even though the engagement in self-regulated learning (SRL) has been shown to boost academic performance, SRL skills of many learners remain underdeveloped. They often struggle to productively navigate multiple cognitive, affective, metacognitive and motivational (CAMM) processes in SRL. To provide learners with the required SRL support, it is essential to understand how learners enact CAMM processes as they study. More research is needed to advance the measurement of affective and motivational processes within SRL, and investigate how these processes influence learners’ cognition and metacognition. With this in mind, we conducted a lab study involving 22 university students who worked on a 45-minute reading and writing task in digital learning environment. We used a wearable electroencephalogram device to record learner academic emotional and motivational states, and digital trace data to record learner cognitive and metacognitive processes. We harnessed time series prediction and explainable artificial intelligence methods to examine how learner’s emotional and motivational states influence their choice of cognitive and metacognitive processes. Our results indicate that emotional and motivational states can predict learners’ use of low cognitive, high cognitive and metacognitive processes with considerable classification accuracy (F1 > 0.73), and that higher values of interest, engagement and excitement promote cognitive processing.

The shift toward digitally-based education has emphasised the need for learners to have strong skills for self-regulated learning (SRL). The use of scaffolding prompts is seen as an effective way to stimulate SRL and enhance academic outcomes. A key aspect of SRL scaffolding prompts is the degree to which they are complied to by students. Compliance is a complex concept, one that is further complicated by the nature of scaffold design in the context of adaptability. These nuances notwithstanding, scaffold compliance demands specific exploration. To that end, we conducted a study in which we: 1) focused specifically on scaffolding interaction behaviour in a timed online assessment task, as opposed to the broader interaction with non-scaffolding artefacts; 2) identified distinct scaffold interaction patterns in the context of compliance and non-compliance to scaffold design; 3) analysed how groups of learners traverse compliant and non-compliant interaction behaviours and engage in SRL processes in response to a sequence of timed and personalised SRL-informed scaffold prompts. We found that scaffold interactions fell into two categories of compliance and non-compliance, and whilst there was a healthy engagement with compliance, it does ebb and flow during an online timed assessment.

Generative artificial intelligence (GenAI), exemplified by ChatGPT, Midjourney, and other state-of-the-art large language models and diffusion models, holds significant potential for transforming education and enhancing human productivity. While the prevalence of GenAI in education has motivated numerous research initiatives, integrating these technologies within the learning analytics (LA) cycle and their implications for practical interventions remain underexplored. This paper delves into the prospective opportunities and challenges GenAI poses for advancing LA. We present a concise overview of the current GenAI landscape and contextualise its potential roles within Clow’s generic framework of the LA cycle. We posit that GenAI can play pivotal roles in analysing unstructured data, generating synthetic learner data, enriching multimodal learner interactions, advancing interactive and explanatory analytics, and facilitating personalisation and adaptive interventions. As the lines blur between learners and GenAI tools, a renewed understanding of learners is needed. Future research can delve deep into frameworks and methodologies that advocate for human-AI collaboration. The LA community can play a pivotal role in capturing data about human and AI contributions and exploring how they can collaborate most effectively. As LA advances, it is essential to consider the pedagogical implications and broader socioeconomic impact of GenAI for ensuring an inclusive future.

Effective collaboration and team communication are critical across many sectors. However, the complex dynamics of collaboration in physical learning spaces, with overlapping dialogue segments and varying participant interactions, pose assessment challenges for educators and self-reflection difficulties for students. Epistemic network analysis (ENA) is a relatively novel technique that has been used in learning analytics (LA) to unpack salient aspects of group communication. Yet, most LA works based on ENA have primarily sought to advance research knowledge rather than directly aid teachers and students by closing the LA loop. We address this gap by conducting a study in which we i) engaged teachers in designing human-centred versions of epistemic networks; ii) formulated an NLP methodology to code physically distributed dialogue segments of students based on multimodal (audio and positioning) data, enabling automatic generation of epistemic networks; and iii) deployed the automatically generated epistemic networks in 28 authentic learning sessions and investigated how they can support teaching. The results indicate the viability of completing the analytics loop through the design of streamlined epistemic network representations that enable teachers to support students’ reflections.

Multimodal Learning Analytics (MMLA) innovations make use of rapidly evolving sensing and artificial intelligence algorithms to collect rich data about learning activities that unfold in physical spaces. The analysis of these data is opening exciting new avenues for both studying and supporting learning. Yet, practical and logistical challenges commonly appear while deploying MMLA innovations “in-the-wild”. These can span from technical issues related to enhancing the learning space with sensing capabilities, to the increased complexity of teachers’ tasks. These practicalities have been rarely investigated. This article addresses this gap by presenting a set of lessons learnt from a 2-year human-centred MMLA in-the-wild study conducted with 399 students and 17 educators in the context of nursing education. The lessons learnt were synthesised into topics related to (i) technological/physical aspects of the deployment; (ii) multimodal data and interfaces; (iii) the design process; (iv) participation, ethics and privacy; and (v) sustainability of the deployment.

Automatic Text Scoring (ATS) is a widely-investigated task in education. Existing approaches often stressed the structure design of an ATS model and neglected the training process of the model. Considering the difficult nature of this task, we argued that the performance of an ATS model could be potentially boosted by carefully selecting data of varying complexities in the training process. Therefore, we aimed to investigate the effectiveness of curriculum learning (CL) in scoring educational text. Specifically, we designed two types of difficulty measurers: (i) pre-defined, calculated by measuring a sample's readability, length, the number of grammatical errors or unique words it contains; and (ii) automatic, calculated based on whether a model in a training epoch can accurately score the samples. These measurers were tested in both the easy-to-hard to hard-to-easy training paradigms. Through extensive evaluations on two widely-used datasets (one for short answer scoring and the other for long essay scoring), we demonstrated that (a) CL indeed could boost the performance of state-of-the-art ATS models, and the maximum improvement could be up to 4.5%, but most improvements were achieved when assessing short and easy answers; (b) the pre-defined measurer calculated based on the number of grammatical errors contained in a text sample tended to outperform the other difficulty measurers across different training paradigms.

Abstract Peer feedback has been widely used in computer-supported collaborative learning (CSCL) setting to improve students’ engagement with massive courses. Although the peer feedback process increases students’ self-regulatory practice, metacognition, and academic achievement, instructors need to go through large amounts of feedback text data which is much more time-consuming. To address this challenge, the present study proposes an automated content analysis approach to identify relevant categories in peer feedback based on traditional and sequence-based classifiers using TF-IDF and content-independent features. We use a data set from an extensive course (N = 231 students) in the setting of engineering higher education. In particular, a total of 2,444 peer feedback messages were analyzed. The CRF classification model based on the TF-IDF features achieved the best performance. The results illustrate that the ability to scale up the automatic analysis of peer feedback provides new opportunities for student-improved learning and improved teacher support in higher education at scale.

Educational feedback has been widely acknowledged as an effective approach to improving student learning. However, scaling effective practices can be laborious and costly, which motivated researchers to work on automated feedback systems (AFS). Inspired by the recent advancements in the pre-trained language models (e.g., ChatGPT), we posit that such models might advance the existing knowledge of textual feedback generation in AFS because of their capability to offer natural-sounding and detailed responses. Therefore, we aimed to investigate the feasibility of using ChatGPT to provide students with feedback to help them learn better. Our results show that i) ChatGPT is capable of generating more detailed feedback that fluently and coherently summarizes students' performance than human instructors; ii) ChatGPT achieved high agreement with the instructor when assessing the topic of students' assignments; and iii) ChatGPT could provide feedback on the process of students completing the task, which might benefit students developing learning skills.

Traditional digital learning environments faced challenges in obtaining comprehensive user interaction data, often yielding fragmented insights without a cohesive visual representation. The emergence of metaverse platforms has enriched this landscape, enabling detailed user activity representation with multimodal data through avatars. However, how to understand the multimodal data related to teaching, social and cognitive presences underpinned by the 'Community of Inquiry' theoretical framework in the metaverse is a big challenge for educators. This study introduces a holistic visualisation solution to bridge this gap, ensuring a better understanding of avatars’ behaviours in an educational metaverse platform – Learningverse developed by our research team. The solution captures a range of multimodal data in Learningverse, such as avatar location, behaviours, emotions, and conversation. Key visualisation elements include heatmaps, points, and arrows, each with distinct informational value. In the future, integrating the solution with multimodal learning analytics is our next step work to understand teaching, social and cognitive presences.

Machine Learning (ML) techniques have been increasingly adopted to support various activities in education, including being applied in important contexts such as college admission and scholarship allocation. In addition to being accurate, the application of these techniques has to be fair, i.e., displaying no discrimination towards any group of stakeholders in education (mainly students and instructors) based on their protective attributes (e.g., gender and age). The past few years have witnessed an explosion of attention given to the predictive bias of ML techniques in education. Though certain endeavors have been made to detect and alleviate predictive bias in learning analytics, it is still hard for newcomers to penetrate. To address this, we systematically reviewed existing studies on predictive bias in education, and a total of 49 peer-reviewed empirical papers published after 2010 were included in this study. In particular, these papers were reviewed and summarized from the following three perspectives: (i) protective attributes, (ii) fairness measures and their applications in various educational tasks, and (iii) strategies for enhancing predictive fairness. These findings were summarized into recommendations to guide future endeavors in this strand of research, e.g., collecting and sharing more quality data containing protective attributes, developing fairness-enhancing approaches which do not require the explicit use of protective attributes, validating the effectiveness of fairness-enhancing on students and instructors in real-world settings.

In recent years, there has been a growing interest in creating multimodal learning analytics (LA) systems that automatically analyse students’ states that are hard to see with the "naked eye", such as cognitive load and stress levels, but that can considerably shape their learning experience. A rich body of research has focused on detecting such aspects by capturing bodily signals from students using wearables and computer vision. Yet, little work has aimed at designing end-user interfaces that visualise physiological data to support tasks deliberately designed for students to learn from stressful situations. This paper addresses this gap by designing a stress analytics dashboard that encodes students’ physiological data into stress levels during different phases of an authentic team simulation in the context of nursing education. We conducted a qualitative study with teachers to understand (i) how they made sense of the stress analytics dashboard; (ii) the extent to which they trusted the dashboard in relation to students’ cortisol data; and (iii) the potential adoption of this tool to communicate insights and aid teaching practices.

Messages exchanged between participants in online discussion forums often contain personal names and other details that need to be redacted before the data is used for research purposes in learning analytics. However, removing the names entirely makes it harder to track the exchange of ideas between individuals within a message thread and across threads, and thereby reduces the value of this type of conversational data. In contrast, the consistent use of pseudonyms allows contributions from individuals to be tracked across messages, while also hiding the real identities of the contributors. Several factors can make it difficult to identify all instances of personal names that refer to the same individual, including spelling errors and the use of shortened forms. We developed a semi-automated approach for replacing personal names with consistent pseudonyms. We evaluated our approach on a data set of over 1,700 messages exchanged during a distance-learning course, and compared it to a general-purpose pseudonymisation tool that used deep neural networks to identify names to be redacted. We found that our tailored approach out-performed the general-purpose tool in both precision and recall, correctly identifying all but 31 substitutions out of 2,888.

Essay writing has become one of the most common learning tasks assigned to students enrolled in various courses at different educational levels, owing to the growing demand for future professionals to effectively communicate information to an audience and develop a written product (i.e. essay). Evaluating a written product requires scorers who manually examine the existence of rhetorical categories, which is a time-consuming task. Machine Learning (ML) approaches have the potential to alleviate this challenge. As a result, several attempts have been made in the literature to automate the identification of rhetorical categories using Rhetorical Structure Theory (RST). However, RST do not provide information regarding students’ cognitive level, which motivates the use of Bloom’s Taxonomy. Therefore, in this research we propose to: i) investigate the extent to which classification of rhetorical categories can be automated based on Bloom’s taxonomy by comparing the traditional ML classifiers with the pre-trained language model BERT, ii) explore the associations between rhetorical categories and writing performance. Our results showed that BERT model outperformed the traditional ML-based classifiers with 18% better accuracy, indicating it can be used in future analytics tool. Moreover, we found a statistical difference between the associations of rhetorical categories in low-achiever, medium-achiever and high-achiever groups which implies that rhetorical categories can be predictive of writing performance.

Capturing data on socio-spatial behaviours is essential in obtaining meaningful educational insights into collaborative learning and teamwork in co-located learning contexts. Existing solutions, however, have limitations regarding scalability and practicality since they rely largely on costly location tracking systems, are labour-intensive, or are unsuitable for complex learning environments. To address these limitations, we propose an innovative computer-vision-based approach – Computer Vision for Position Estimation (CVPE) – for collecting socio-spatial data in complex learning settings where sophisticated collaborations occur. CVPE is scalable and practical with a fast processing time and only needs low-cost hardware (e.g., cameras and computers). The built-in privacy protection modules also minimise potential privacy and data security issues by masking individuals’ facial identities and provide options to automatically delete recordings after processing, making CVPE a suitable option for generating continuous multimodal/classroom analytics. The potential of CVPE was evaluated by applying it to analyse video data about teamwork in simulation-based learning. The results showed that CVPE extracted socio-spatial behaviours relatively reliably from video recordings compared to indoor positioning data. These socio-spatial behaviours extracted with CVPE uncovered valuable insights into teamwork when analysed with epistemic network analysis. The limitations of CVPE for effective use in learning analytics are also discussed.

Feedback is an effective way to assist students in achieving learning goals. The conceptualisation of feedback is gradually moving from feedback as information to feedback as a learner-centred process. To demonstrate feedback effectiveness, feedback as a learner-centred process should be designed to provide quality feedback content and promote student learning outcomes on the subsequent task. However, it remains unclear how instructors adopt the learner-centred feedback framework for feedback provision in the teaching practice. Thus, our study made use of a comprehensive learner-centred feedback framework to analyse feedback content and identify the characteristics of feedback content among student groups with different performance changes. Specifically, we collected the instructors’ feedback on two consecutive assignments offered by an introductory to data science course at the postgraduate level. On the basis of the first assignment, we used the status of student grade changes (i.e., students whose performance increased and those whose performance did not increase on the second assignment) as the proxy of the student learning outcomes. Then, we engineered and extracted features from the feedback content on the first assignment using a learner-centred feedback framework and further examined the differences of these features between different groups of student learning outcomes. Lastly, we used the features to predict student learning outcomes by using widely-used machine learning models and provided the interpretation of predicted results by using the SHapley Additive exPlanations (SHAP) framework. We found that 1) most features from the feedback content presented significant differences between the groups of student learning outcomes, 2) the gradient boost tree model could effectively predict student learning outcomes, and 3) SHAP could transparently interpret the feature importance on predictions.

Textual cohesion is an essential aspect of a formally written text, related to linguistic mechanisms that connect elements such as words, sentences, and paragraphs. Several studies have proposed approaches to estimate textual cohesion in essays automatically. There is limited research that aims to study the extent to which the use of machine learning approaches can predict the textual cohesion of essays written in different languages (not just English). This paper reports on the findings of a study that aimed to propose and evaluate approaches that automatically estimate the cohesion of essays in Portuguese and English. The study proposed regression-based models grounded in conventional feature-based machine learning methods and deep learning-based pre-trained language models. The study also examined the explainability of automated approaches to scrutinize their predictions. We analyzed two datasets composed of 4,570 (Portuguese) and 7,101 (English) essays. The results demonstrate that a deep learning-based model achieved the best performance on both datasets with a moderate Pearson correlation with human-rated cohesion scores. However, the explainability of the automatic cohesion estimations based on conventional machine learning models offered a stronger potential than that of the deep learning model.

There is a proliferation of learning analytics (LA) dashboards aimed at supporting teachers. Yet, teachers still find it challenging to make sense of LA dashboards, thereby making informed decisions. Two main strategies to address this are emerging: i) upskilling teachers’ data literacy; ii) improving the explanatory design features of current dashboards (e.g., adding visual cues or text) to minimise the skills required by teachers to effectively use dashboards. While each approach has its own trade-offs, no previous work has explored the interplay between the dashboard design and such "data skills". In this paper, we explore how teachers with varying visualisation literacy (VL) skills use LA dashboards enhanced with (explanatory) data storytelling elements. We conducted a quasi-experimental study with 23 teachers of varied VL inspecting two versions of an authentic multichannel dashboard enhanced with data storytelling elements. We used an eye-tracking device while teachers inspected the students’ data captured from Zoom and Google Docs, followed by interviews. Results suggest that high VL teachers adopted complex exploratory strategies and were more sensitive to subtle inconsistencies in the design; while low VL teachers benefited the most from more explicit data storytelling guidance such as accompanying complex graphs with narrative and semantic colour encoding.

Homophily is a fundamental sociological theory that describes the tendency of individuals to interact with others who share similar attributes. This theory has shown evident relevance for studying collaborative learning and classroom orchestration in learning analytics research from a social constructivist perspective. Emerging advancements in multimodal learning analytics have shown promising results in capturing interaction data and generating socio-spatial analytics in physical learning spaces through computer vision and wearable positioning technologies. Yet, there are limited ways for analysing homophily (e.g., social network analysis; SNA), especially for unpacking the temporal connections between different homophilic behaviours. This paper presents a novel analytic approach, Social-epistemic Network Analysis (SeNA), for analysing homophily by combining social network analysis with epistemic network analysis to infuse socio-spatial analytics with temporal insights. The additional insights SeNA may offer over traditional approaches (e.g., SNA) were illustrated through analysing the homophily of 98 students in open learning spaces. The findings showed that SeNA could reveal significant behavioural differences in homophily between comparison groups across different learning designs, which were not accessible to SNA alone. The implications and limitations of SeNA in supporting future learning analytics research regarding homophily in physical learning spaces are also discussed.

Embodied team learning is a form of group learning that occurs in co-located settings where students need to interact with others while actively using resources in the physical learning space to achieve a common goal. In such situations, communication dynamics can be complex as team discourse segments can happen in parallel at different locations of the physical space with varied team member configurations. This can make it hard for teachers to assess the effectiveness of teamwork and for students to reflect on their own experiences. To address this problem, we propose METS (Multimodal Embodied Teamwork Signature), a method to model team dialogue content in combination with spatial and temporal data to generate a signature of embodied teamwork. We present a study in the context of a highly dynamic healthcare team simulation space where students can freely move. We illustrate how signatures of embodied teamwork can help to identify key differences between high and low performing teams: i) across the whole learning session; ii) at different phases of learning sessions; and iii) at particular spaces of interest in the learning space.

Learning analytics (LA) employs educational data to improve the timeliness of support for learners. Apart from technical aspects, there is a need to understand social complexities brought about by different stakeholders, so as to systematise the adoption of LA in Higher Education (HE). We present an analysis of the situation, needs and challenges of LA in the context of Spanish HE, considering managers’, teachers’ and students’ perspectives. Qualitative research is employed using recursive abstraction. Results reveal that the level of institutional adoption is low and none of the analysed institutions had an LA policy. Furthermore, only two of these institutions had an initial LA strategy. While the institutions shared some commonalities in their objectives for LA, chosen tools and adoption challenges, the distinct differences in the political contexts and institutional practices among the institutions reaffirmed that LA solutions and services cannot be implemented in the same manner. Moreover, different needs for LA and concerns are identified about its adoption among managers, students and teachers. These observations lead to our conclusion that the main challenges to implement LA in Spain are not related to technological issues but to the social and cultural issues rooted in institutions and those associated with different stakeholders.

Recent years have witnessed a marked increase in both academic proposals and industrial adoptions of blockchain technology. However, a majority of the projects remain at the stage of prototype proposals and their real-world deployment has not met the anticipated level. This gap can be attributed to three major barriers - technical difficulties, human factors, and social context. Most of the existing research leans towards addressing the technical challenges, leaving the human and social aspects inadequately explored. Moreover, a lack of practical insights in the existing blockchain software engineering frameworks further exacerbates the adoption problem. To address these gaps, we introduce a Blockchain-oriented Software Engineering Approach for Higher Adoption Possibility (BOSE-HAP) . This approach emphasizes collaboration, reflective thinking, and iterative development, aiming to bolster implementation consistency and stimulate industry adoption. We have applied this approach in the design, development, and launch of a blockchain credentialing product, CValid.org , in the context of a university-level summer school. The product achieves industry-accepted System Usability Score and has seen successful real-world deployment. In addition, this study embed usability considerations throughout the process, involved a total of 112 stakeholders across different development stages, with 25 of them participating in our in-depth interviews and usability testing. Drawing from our firsthand experience and industrial-grounded findings, we deliver eight reflections and propose five best practice suggestions relevant to blockchain adoption. We believe these insights will provide invaluable guidance for both academic researchers and industry practitioners involved in the field of blockchain technology.

The past few years have witnessed an explosion of attention given to the bias displayed by Machine Learning (ML) techniques towards different groups of people (e.g., female vs. male). Although ML techniques have been widely adopted in education, it remains largely unexplored that to what extent such ML bias manifests itself in this specific setting and how it can be reduced and eliminated. Given the increasing importance of ML techniques in empowering educators to teach effectively, this study aimed to quantify the characteristics of the original datasets that might be correlated with the subsequent predictive unfairness displayed by ML models. To this end, we empirically investigated two types of data biases (i.e., distribution bias and hardness bias) towards students of different sexes and first-language backgrounds across a total of five frequently-performed predictive tasks in education. Then, to improve ML fairness, we drew inspiration from the well-established research in Class Balancing Techniques (CBTs), where samples are generated/removed to alleviate the predictive disparity between different prediction classes. We proposed two simple but effective strategies to empower class balancing techniques for alleviating data biases and improving prediction fairness. Through extensive analyses and evaluations, we demonstrated that ML models may greatly improve prediction fairness (improvement up to 66%) with only a small sacrifice (less than 1%) in prediction accuracy by balancing the training data with the use of students’ demographic information and the overall hardness bias measure. All data and code used in this study are publicly accessible via https://github.com/lsha49/FairEdu.

Pretrained Language Models (PLMs), though popular, have been diagnosed to encode bias against protected groups in the representations they learn, which may harm the prediction fairness of downstream models. Given that such bias is believed to be related to the amount of demographic information carried in the learned representations, this study aimed to quantify the awareness that a PLM (i.e., BERT) has regarding people’s protected attributes and augment BERT to improve prediction fairness of downstream models by inhibiting this awareness. Specifically, we developed a method to dynamically sample data to continue the pretraining of BERT and enable it to generate representations carrying minimal demographic information, which can be directly used as input to downstream models for fairer predictions. By experimenting on the task of classifying educational forum posts and measuring fairness between students of different gender or first-language backgrounds, we showed that, compared to a baseline without any additional pretraining, our method improved not only fairness (with a maximum improvement of 52.33%) but also accuracy (with a maximum improvement of 2.53%). Our method can be generalized to any PLM and demographic attributes. All the codes used in this study can be accessed via https://github.com/lsha49/FairBERT_deploy.

Stakeholder engagement is a key aspect for the successful implementation of Learning Analytics (LA) in Higher Education Institutions (HEIs). Studies in Europe and Latin America (LATAM) indicate that, overall, instructors and students have positive views on LA adoption, but there are differences between their ideal expectations and what they consider realistic in the context of their institutions. So far, very little has been found about stakeholders’ views on LA in Brazilian higher education. By replicating the survey conducted in other countries, in seven Brazilian HEIs, we found convergences both with Europe and LATAM, reinforcing the need for local diagnosis and indicating the risk of assuming a ”LATAM identity”. Our findings contribute to building a corpus of knowledge on stakeholders expectations with a contextualised comprehension of the gaps between ideal and predicted scenarios, which can inform institutional policies for LA implementation in Brazil.

Roles that learners assume during online discussions are an important aspect of educational experience. The roles can be assigned to learners and/or can spontaneously emerge through student-student interaction. While existing research proposed several approaches for analytics of emerging roles, there is limited research in analytic methods that can i) automatically detect emerging roles that can be interpreted in terms of higher-order constructs of collaboration; ii) analyse the extent to which students complied to scripted roles and how emerging roles compare to scripted ones; and iii) track progression of roles in social knowledge progression over time. To address these gaps in the literature, this paper propose a network-analytic approach that combines techniques of cluster analysis and epistemic network analysis. The method was validated in an empirical study discovered emerging roles that were found meaningful in terms of social and cognitive dimensions of the well-known model of communities of inquiry. The study also revealed similarities and differences between emerging and script roles played by learners and identified different progression trajectories in social knowledge construction between emerging and scripted roles. The proposed analytic approach and the study results have implications that can inform teaching practice and development techniques for collaboration analytics.

This research aimed to explore the relationship between different indicators of the depth and quality of participation in computer-mediated learning environments. By using network analyses and statistical tests, we discovered significant associations between the cognitive presence phases of the Community of Inquiry framework and speech acts, and examined the impact of two different instructional interventions on these associations. We found that there are strong associations between some speech acts and cognitive presence phases. In addition, the study revealed that the association between speech acts and cognitive presence is moderated by external facilitation, but not affected by user role assignment. The results suggest that speech acts can plausibly be used to provide feedback in relation to cognitive presence and can potentially be used to increase the generalizability of cognitive presence classification.

The benefits of incorporating scaffolds that promote strategies of self-regulated learning (SRL) to help student learning are widely studied and recognised in the literature. However, the best methods for incorporating them in educational technologies and empirical evidence about which scaffolds are most beneficial to students are still emerging. In this paper, we report our findings from conducting an in-the-field controlled experiment with 797 post-secondary students to evaluate the impact of incorporating scaffolds for promoting SRL strategies in the context of assisting students in creating novel content, also known as learnersourcing. The experiment had five conditions, including a control group that had access to none of the scaffolding strategies for creating content, three groups each having access to one of the scaffolding strategies (planning, externally-facilitated monitoring and self-assessing) and a group with access to all of the aforementioned scaffolds. The results revealed that the addition of the scaffolds for SRL strategies increased the complexity and effort required for creating content, were not positively assessed by learners and led to slight improvements in the quality of the generated content. We discuss the implications of our findings for incorporating SRL strategies in educational technologies.

Existing research indicates that students prefer to work with tutors who express politely in online human-human tutoring, but excessive polite expressions might lower tutoring efficacy. However, there is a shortage of understanding about the use of politeness in online tutoring and the extent to which the politeness of instructional strategies can contribute to students’ achievement. To address these gaps, we conducted a study on a large-scale dataset (5,165 students and 116 qualified tutors in 18,203 online tutoring sessions) of both effective and ineffective human-human online tutorial dialogues. The study made use of a well-known dialogue act coding scheme to identify instructional strategies, relied on the linguistic politeness theory to analyse the politeness levels of the tutors’ instructional strategies, and utilised Gradient Tree Boosting to evaluate the predictive power of these politeness levels in revealing students’ problem-solving performance. The results demonstrated that human tutors used both polite and non-polite expressions in the instructional strategies. Tutors were inclined to express politely in the strategy of providing positive feedback but less politely while providing negative feedback and asking questions to evaluate students’ understanding. Compared to the students with prior progress, tutors provided more polite open questions to the students without prior progress but less polite corrective feedback. Importantly, we showed that, compared to previous research, the accuracy of predicting student problem-solving performance can be improved by incorporating politeness levels of instructional strategies with other documented predictors (e.g., the sentiment of the utterances).

Brazilian universities have included essay writing assignments in the entrance examination procedure to select prospective students. The essay scorers manually look for the presence of required Rhetorical Structure Theory (RST) categories and evaluate essay coherence. However, identifying RST categories is a time-consuming task. The literature reported several attempts to automate the identification of RST categories in essays with machine learning. Still, previous studies have focused on using machine learning algorithms trained on content-dependent features that can diminish classification performance, leading to over-fitting and hindering model generalisability. Therefore, this paper proposes: (i) the analysis of state-of-the-art classifiers and content-independent features to the task of RST rhetorical moves; (ii) a new approach that considers the sequence of the text to extract features – i.e. sequential content-independent features; (iii) an empirical study about the generalisability of the machine learning models and sequential content-independent features for this context; (iv) the identification of the most predictive features for automated identification of RST categories in essays written in Portuguese. The best performing classifier, XGBoost, based on sequential content-independent features, outperformed the classifiers used in the literature and are based on traditional content-dependent features. The XGBoost classifier based on sequential content-independent features also reached promising accuracy when tested for generalisability.

One of the ultimate goals of several learning analytics (LA) initiatives is to close the loop and support students’ and teachers’ reflective practices. Although there has been a proliferation of end-user interfaces (often in the form of dashboards), various limitations have already been identified in the literature such as key stakeholders not being involved in their design, little or no account for sense-making needs, and unclear effects on teaching and learning. There has been a recent call for human-centred design practices to create LA interfaces in close collaboration with educational stakeholders to consider the learning design, and their authentic needs and pedagogical intentions. This paper addresses the call by proposing a question-driven LA design approach to ensure that end-user LA interfaces explicitly address teachers’ questions. We illustrate the approach in the context of synchronous online activities, orchestrated by pairs of teachers using audio-visual and text-based tools (namely Zoom and Google Docs). This study led to the design and deployment of an open-source monitoring tool to be used in real-time by teachers when students work collaboratively in breakout rooms, and across learning spaces.

Writing from multiple sources is a commonly administered learning task across educational levels and disciplines. In this task, learners are instructed to comprehend information from source documents and integrate it into a coherent written composition to fulfil the assignment requirements. Even though educationally potent, multi-source writing tasks are considered challenging to many learners, in particular because many learners underuse monitoring and control, critical metacognitive processes for productive engagement in multi-source writing. To understand these processes, we conducted a laboratory study involving 44 university students. They engaged in multi-source writing task hosted in digital learning environment. Adding to previous research, we unobtrusively measured metacognitive processes using learners’ trace data collected via multiple data channels and in both writing and reading space of the multi-source writing task. We further investigated how these processes affect the quality of a written product, i.e., essay score. In the analysis, we utilised both automatically and human-generated essay score. The rating performance of the essay scoring algorithm was comparable to that of human raters. Our results largely support the theoretical assumptions that engagement in metacognitive monitoring and control benefits the quality of written product. Moreover, our results can inform the development of analytics-based tools that support student writing by making use of trace data and automated essay scoring.

Self-regulated learning (SRL) skills are essential for successful learning in a technology-enhanced learning environment. Learning Analytics techniques have shown a great potential in identifying and exploring SRL strategies from trace data in various learning environments. However, these strategies have been mainly identified through analysis of sequences of learning actions, and thus interpretation of the strategies is heavily task and context dependent. Further, little research has been done on the association of SRL strategies with different influencing factors or conditions. To address these gaps, we propose an analytic method for detecting SRL strategies from theoretically supported SRL processes and applied the method to a dataset collected from a multi-source writing task. The detected SRL strategies were explored in terms of their association with the learning outcome, internal conditions (prior-knowledge, metacognitive knowledge and motivation) and external conditions (scaffolding). The study results showed our analytic method successfully identified three theoretically meaningful SRL strategies. The study results revealed small effect size in the association between the internal conditions and the identified SRL strategies, but revealed a moderate effect size in the association between external conditions and the SRL strategy use.

Academic analytics focuses on collecting, analysing and visualising educational data to generate institutional insights and improve decision-making for academic purposes. However, challenges that arise from navigating a complex organisational structure when introducing analytics systems have called for the need to engage key stakeholders widely to cultivate a shared vision and ensure that implemented systems create desired value. This paper presents a study that takes co-design steps to identify design needs and strategic approaches for the adoption of academic analytics, which serves the purpose of enhancing the measurement of educational quality utilising institutional data. Through semi-structured interviews with 54 educational stakeholders at a large research university, we identified particular interest in measuring student engagement and the performance of courses and programmes. Based on the observed perceptions and concerns regarding data use to measure or evaluate these areas, implications for adoption strategy of academic analytics, such as leadership involvement, communication, and training, are discussed.

Teacher’s in-class positioning and interaction patterns (termed spatial pedagogy) are an essential part of their classroom management and orchestration strategies that can substantially impact students’ learning. Yet, effective management of teachers’ spatial pedagogy can become increasingly challenging as novel architectural designs, such as open learning spaces, aim to disrupt teaching conventions by promoting flexible pedagogical approaches and maximising student connectedness. Multimodal learning analytics and indoor positioning technologies may hold promises to support teachers in complex learning spaces by making salient aspects of their spatial pedagogy visible for provoking reflection. This paper explores how granular x-y positioning data can be modelled into socio-spatial metrics that can contain insights about teachers’ spatial pedagogy across various learning designs. A total of approximately 172.63 million position data points were collected during 101 classes over eight weeks. The results illustrate how indoor positioning analytics can help generate a deeper understanding of how teachers use their learning spaces, such as their 1) teaching responsibilities; 2) proactive or passive interactions with students; and 3) supervisory, interactional, collaborative, and authoritative teaching approaches. Implications of the current findings to future learning analytics research and educational practices were also discussed.

Multimodal Learning Analytics (MMLA) innovations are commonly aimed at supporting learners in physical learning spaces through state-of-the-art sensing technologies and analysis techniques. Although a growing body of MMLA research has demonstrated the potential benefits of sensor-based technologies in education, whether their use can be scalable, sustainable, and ethical remains questionable. Such uncertainty can limit future research and the potential adoption of MMLA by educational stakeholders in authentic learning situations. To address this, we systematically reviewed the methodological, operational, and ethical challenges faced by current MMLA works that can affect the scalability and sustainability of future MMLA innovations. A total of 96 peer-reviewed articles published after 2010 were included. The findings were summarised into three recommendations, including i) improving reporting standards by including sufficient details about sensors, analysis techniques, and the full disclosure of evaluation metrics, ii) fostering interdisciplinary collaborations among experts in learning analytics, software, and hardware engineering to develop affordable sensors and upgrade MMLA innovations that used discontinued technologies, and iii) developing ethical guidelines to address the potential risks of bias, privacy, and equality concerns with using MMLA innovations. Through these future research directions, MMLA can remain relevant and eventually have actual impacts on educational practices.

In co-located situations, team members use a combination of verbal and visual signals to communicate effectively, among which positional forms play a key role. The spatial patterns adopted by team members in terms of where in the physical space they are standing, and who their body is oriented to, can be key in analysing and increasing the quality of interaction during such face-to-face situations. In this paper, we model the students’ communication based on spatial (positioning) and audio (voice detection) data captured from 92 students working in teams of four in the context of healthcare simulation. We extract non-verbal events (i.e., total speaking time, overlapped speech,and speech responses to team members and teachers) and investigate to what extent they can serve as meaningful indicators of students’ performance according to teachers’ learning intentions. The contribution of this paper to multimodal learning analytics includes: i) a generic method to semi-automatically model communication in a setting where students can freely move in the learning space; and ii) results from a mixed-methods analysis of non-verbal indicators of team communication with respect to teachers’ learning design.

Peer review has been recognised as a beneficial approach that promotes higher-order learning and provides students with fast and detailed feedback on their work. Still, there are some common concerns and criticisms associated with the use of peer review that limits its adoption. One of the main points of concern is that feedback provided by students may be ineffective and of low quality. Previous works supply three explanations for why students may fail to provide effective feedback: They lack (1) the ability to provide high-quality feedback, (2) the agency to monitor their work or (3) the incentive to invest the required time and effort as they think the quality of the reviews are not reviewed. To help mitigate these shortcomings, this paper presents a complementary peer review approach that integrates training, self-monitoring and AI quality-control assistance to improve peer feedback quality. In particular, informed by higher education research, we built a set of training materials and a self-monitoring checklist for students to consider while writing their reviews. Also, informed by work from natural language processing, we developed quality control functions that automatically assess feedback submitted and prompt students to improve, if necessary. A between-subjects field experiment with 374 participants was conducted to investigate the approach's efficacy. Findings suggest that offering training, self-monitoring, and quality control functionalities to students assigned to the complementary peer review approach resulted in longer feedback that was perceived as more helpful than those who utilised the regular peer review interface. However, this complementary approach does not seem to affect students judgement (leniency or harshness) or confidence in grading. Directions are suggested to further evaluate and refine peer review systems.

Zijie Zeng, Xinyu Li, Dragan Gasevic, Guanliang Chen. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

One of the ultimate goals of several learning analytics (LA) initiatives is to close the loop and support students' and teachers' reflective practices. Although there has been a proliferation of end-user interfaces (often in the form of dashboards), various limitations have already been identified in the literature such as little account for sensemaking needs. This paper addresses these limitations by proposing a question-driven LA design approach to ensure that end-user LA interfaces explicitly address teachers' questions. We illustrate this in the context of synchronous online activities orchestrated by pairs of teachers using audio-visual and text-based tools (Zoom and Google Docs). This led to the design of an open-source monitoring tool to be used in real-time by teachers when students work collaboratively in breakout rooms, and across learning spaces.

Supervised machine learning models have been widely used to address the classification of messages in online discussions. Supervised learning algorithms require a large set of annotated data to accurately create a predictive model. However, data annotation is a complex task due to three factors: (i) depends on specialists to accurately label data; (ii) it is often a time-consuming and labour-intensive work,and(iii) in educational settings, it is not always easy to collect a substantial volume of data required by the machine learning algorithms. This paper presents an active learning-based approach that can reduce the amount of annotated data required to build machine learning models for the classification of educational data. The results obtained show that with only 20% of the annotated data, the proposed approach achieved similar results to those presented in the previous works that used the complete databases to train the machine learning model.

This paper reports the findings of a study that measured the effectiveness of employing automatic text translation methods in automated classification of online discussion messages according to the categories of social and cognitive presences. Specifically, we examined the classification of 1,500 Portuguese and 1,747 English discussion messages using classifiers trained on the datasets before and after the application of text translation. While the English model generated, with the original and translated texts, achieved results (accuracy and Cohen’s κ) similar to those of the previously reported studies, the translation to Portuguese led to a decrease in the performance. The indicates the general viability of the proposed approach when converting the text to English. Moreover, this study highlighted the importance of different features and resources, and the limitations of the resources for Portuguese as reasons of the results obtained.

Peer reviews offer many learning benefits. Understanding students’ engagement in them can help design effective practices. Although learning analytics can be effective in generating such insights, its application in peer reviews is scarce. Theory can provide the necessary foundations to inform the design of learning analytics research and the interpretation of its results. In this paper, we followed a theory-based learning analytics approach to identifying students’ engagement patterns in a peer review activity facilitated via a web-based tool called Synergy. Process mining was applied on temporal learning data, traced by Synergy. The theory about peer review helped determine relevant data points and guided the top-down approach employed for their analysis: moving from the global phases to regulation of learning, and then to micro-level actions. The results suggest that theory and learning analytics should mutually relate with each other. Mainly, theory played a critical role in identifying a priori engagement patterns, which provided an informed perspective when interpreting the results. In return, the results of the learning analytics offered critical insights about student behavior that was not expected by the theory (i.e., low levels of co-regulation). The findings provided important implications for refining the grounding theory and its operationalization in Synergy.

Investigation of learning tactics and strategies has received increasing attention by the Learning Analytics (LA) community. While previous research efforts have made notable contributions towards identifying and understanding learning tactics from trace data in various blended and online learning settings, there is still a need to deepen our understanding about learning processes that are activated during the enactment of distinct learning tactics. In order to fill this gap, we propose a learning analytic approach to unveiling and comparing self-regulatory processes in learning tactics detected from trace data. Following this approach, we detected four learning tactics (Reading with Quiz Tactic, Assessment and Interaction Tactic, Short Login and Interact Tactic and Focus on Quiz Tactic) as used by 728 learners in an undergrad course. We then theorised and detected five micro-level processes of self-regulated learning (SRL) through an analysis of trace data. We analysed how these micro-level SRL processes were activated during enactment of the four learning tactics in terms of their frequency of occurrence and temporal sequencing. We found significant differences across the four tactics regarding the five micro-level SRL processes based on multivariate analysis of variance and comparison of process models. In summary, the proposed LA approach allows for meaningful interpretation and distinction of learning tactics in terms of the underlying SRL processes. More importantly, this approach shows the potential to overcome the limitations in the interpretation of LA results which stem from the context-specific nature of learning. Specifically, the study has demonstrated how the interpretation of LA results and recommendation of pedagogical interventions can also be provided at the level of learning processes rather than only in terms of a specific course design.

Asynchronous online discussions within a community of learners can improve learning outcomes through social knowledge construction, but the depth and quality of student contributions often varies widely. Approaches to assessing critical discourse typically use content analysis to identify indicators that correspond to framework constructs, that in turn serve as measures of depth and quality. Often only a single construct is addressed for performing content analysis in the literature, although recent work has used both social presence and cognitive presence constructs from the Community of Inquiry (CoI) framework. Nevertheless, there is no effective, commonly used, analytic approach to combining insights from multiple perspectives about quality and depth of online discussions. This paper addresses the gap by proposing the combined use of cognitive engagement (the ICAP framework) and cognitive presence (CoI); and by proposing a network analytic approach that quantifies the associations between the two frameworks and measures the moderation effects of two instructional interventions on those associations. The present study found that these associations were moderated by one intervention but not the other; and that messages labelled with the most common phase of cognitive presence could be usefully assigned to smaller meaningful subgroups by also considering the mode of cognitive engagement.

Researchers have been struggling with the measurement of Self-Regulated Learning (SRL) for decades. Instrumentation tools have been proposed to help capture SRL processes that are difficult to capture. The aim of the present study was to improve measurement of SRL by embedding instrumentation tools in a learning environment and validating the measurement of SRL with these instrumentation tools using think aloud. Synchronizing log data and concurrent think aloud data helped identify which SRL processes were captured by particular instrumentation tools. One tool was associated with a single SRL process: the timer co-occurred with monitoring. Other tools co-occurred with a number of SRL processes, i.e., the highlighter and note taker captured superficial writing down, organizing, and monitoring, whereas the search and planner tools revealed planning and monitoring. When specific learner actions with the tool were analyzed, a clearer picture emerged of the relation between the highlighter and note taker and SRL processes. By aligning log data with think aloud data, we showed that instrumentation tool use indeed reflects SRL processes. The main contribution is that this paper is the first to show that SRL processes that are difficult to measure by trace data can indeed be captured by instrumentation tools such as high cognition and metacognition. Future challenges are to collect and process log data real time with learning analytic techniques to measure ongoing SRL processes and support learners during learning with personalized SRL scaffolds.

Learnersourcing is emerging as a viable learner-centred and pedagogically justified approach for harnessing the creativity and evaluation power of learners as experts-in-training. Despite the increasing adoption of learnersourcing in higher education, understanding students’ behaviour while engaged in learnersourcing and best practices for the design and development of learnersourcing systems are still largely under-researched. This paper offers data-driven reflections and lessons learned from the development and deployment of a learnersourcing adaptive educational system called RiPPLE, which to date, has been used in more than 50-course offerings with over 12,000 students. Our reflections are categorised into examples and best practices on (1) assessing the quality of students’ contributions using accurate, explainable and fair approaches to data analysis, (2) incentivising students to develop high-quality contributions and (3) empowering instructors with actionable and explainable insights to guide student learning. We discuss the implications of these findings and how they may contribute to the growing literature on the development of effective learnersourcing systems and more broadly technological educational solutions that support learner-centred learning at scale.

Learning analytics (LA) has been presented as a viable solution for scaling timely and personalised feedback to support students’ self-regulated learning (SRL). Research is emerging that shows some positive associations between personalised feedback with students’ learning tactics and strategies as well as time management strategies, both important aspects of SRL. However, the definitive role of feedback on students’ SRL adaptations is under-researched; this requires an examination of students’ recalled experiences with their personalised feedback. Furthermore, an important consideration in feedback impact is the course context, comprised of the learning design and delivery modality. This mixed-methods study triangulates learner trace data from two different course contexts, with students’ qualitative data collected from focus group discussions, to more fully understand the impact of their personalised feedback and to explicate the role of this feedback on students’ SRL adaptations. The quantitative analysis showed the contextualised impact of the feedback on students’ learning and time management strategies in the different courses, while the qualitative analysis highlighted specific ways in which students used their feedback to adjust these and other SRL processes.

The conceptualisation of self-regulated learning (SRL) as a process that unfolds over time has influenced the way in which researchers approach analysis. This gave rise to the use of process mining in contemporary SRL research to analyse data about temporal and sequential relations of processes that occur in SRL. However, little attention has been paid to the choice and combinations of process mining algorithms to achieve the nuanced needs of SRL research. We present a study that 1) analysed four process mining algorithms that are most commonly used in the SRL literature – Inductive Miner, Heuristics Miner, Fuzzy Miner, and pMineR; and 2) examined how the metrics produced by the four algorithms complement each. The study looked at micro-level processes that were extracted from trace data collected in an undergraduate course (N=726). The study found that Fuzzy Miner and pMineR offered better insights into SRL than the other two algorithms. The study also found that a combination of metrics produced by several algorithms improved interpretation of temporal and sequential relations between SRL processes. Thus, it is recommended that future studies of SRL combine the use of process mining algorithms and work on new tools and algorithms specifically created for SRL research.

Video has become an essential medium for learning. However, there are challenges when using traditional methods to measure how learners attend to lecture videos in video learning analytics, such as difficulty in capturing learners’ attention at a fine-grained level. Therefore, in this paper, we propose a gaze-based metric—“with-me-ness direction” that can measure how learners’ gaze-direction changes when they listen to the instructor’s dialogues in a video-lecture. We analyze the gaze data of 45 participants as they watched a video lecture and measured both the sequences of with-me-ness direction and proportion of time a participant spent looking in each direction throughout the lecture at different levels. We found that although the majority of the time participants followed the instructor’s dialogues, their behaviour of looking-ahead, looking-behind or looking-outside differed by their prior knowledge. These findings open the possibility of using eye-tracking to measure learners’ video-watching attention patterns and examine factors that can influence their attention, thereby helping instructors to design effective learning materials.

Feedback plays a crucial role in student learning. Learning analytics (LA) has demonstrated potential in addressing prominent challenges with feedback practice, such as enabling timely feedback based on insights obtained from large data sets. However, there is insufficient research looking into relations between student expectations of feedback and their experience with LA-based feedback. This paper presents a pilot study that examined students’ experience of LA-based feedback, offered with the OnTask system, taking into consideration the factors of students’self-efficacy and self-regulation skills. Two surveys were carried out at a Brazilian university, and the results highlighted important implications for LA-based feedback practice, including leveraging the ‘partnership’ between the human teacher and the computer, and developing feedback literacy among learners.

Schools are increasingly becoming into complex learning spaces where students interact with various physical and digital resources, educators, and peers. Although the field of learning analytics has advanced in analysing logs captured from digital tools, less progress has been made in understanding the social dynamics that unfold in physical learning spaces. Among the various rapidly emerging sensing technologies, position tracking may hold promises to reveal salient aspects of activities in physical learning spaces such as the formation of interpersonal ties among students. This paper explores how granular x-y physical positioning data can be analysed to model social interactions among students and teachers. We conducted an 8-week longitudinal study in which positioning traces of 98 students and six teachers were automatically captured every day in an open-plan public primary school. Positioning traces were analysed using social network analytics (SNA) to extract a set of metrics to characterise students’ positioning behaviours and social ties at cohort and individual levels. Results illustrate how analysing positioning traces through the lens of SNA can enable the identification of certain pedagogical approaches that may be either promoting or discouraging in-class social interaction, and students who may be socially isolated.

Collaboration is considered as one of the main drivers of learning and it has been broadly studied across numerous contexts, including Massive Open Online Courses (MOOCs). The research on MOOCs has risen exponentially during the last years and there have been a number of works focused on studying collaboration. However, these previous studies have been restricted to the analysis of collaboration based on the forum and social interactions, without taking into account other possibilities such as the synchronicity in the interactions with the platform. Therefore, in this work we performed a case study with the goal of implementing a data-driven approach to detect and characterize collaboration in MOOCs. We applied an algorithm to detect synchronicity links based on their submission times to quizzes as an indicator of collaboration, and applied it to data from two large Coursera MOOCs. We found three different profiles of user accounts, that were grouped in couples and larger communities exhibiting different types of associations between user accounts. The characterization of these user accounts suggested that some of them might represent genuine online learning collaborative associations, but that in other cases dishonest behaviors such as free-riding or multiple account cheating might be present. These findings call for additional research on the study of the kind of collaborations that can emerge in online settings.

Learning Analytics is a new field in education whose adoption can bring benefits for teaching and learning processes. However, many higher education institutions may not be ready to start using learning analytics due to challenges such as organizational culture, infrastructure, and privacy. In this context, Maturity Models (MMs) can support institutions to systematize their processes, enabling them to progress successively in the learning analytics adoption. MMs are used in different fields to support the improvement of processes, describing them in terms of maturity levels, and identifying enhancements that could lead an organization to higher levels of such maturity. Thus, this paper presents an outline of a MM for Learning Analytics adoption in higher education institutions, describing its levels and areas, together with its development methodology.

Educational recommender systems (ERSs) aim to adaptively recommend a broad range of personalised resources and activities to students that will most meet their learning needs. Commonly, ERSs operate as a "black box" and give students no insight into the rationale of their choice. Recent contributions from the learning analytics and educational data mining communities have emphasised the importance of transparent, understandable and open learner models (OLMs) that provide insight and enhance learners' understanding of interactions with learning environments. In this paper, we aim to investigate the impact of complementing ERSs with transparent and understandable OLMs that provide justification for their recommendations. We conduct a randomised control trial experiment using an ERS with two interfaces ("Non-Complemented Interface" and "Complemented Interface") to determine the effect of our approach on student engagement and their perception of the effectiveness of the ERS. Overall, our results suggest that complementing an ERS with an OLM can have a positive effect on student engagement and their perception about the effectiveness of the system despite potentially making the system harder to navigate. In some cases, complementing an ERS with an OLM has the negative consequence of decreasing engagement, understandability and sense of fairness.

This paper presents a study that examined automated cross-language classification of online discussion messages for the levels of cognitive presence, a key construct from the widely used Community of Inquiry (CoI) model of online learning. Specifically, we examined the classification of 1,500 Portuguese language discussion messages using a classifier trained on a corpus of the 1,747 English language discussion messages. In the study, a random forest classifier was developed using a small set of 108 validated indicators of psychological processes, linguistic coherence, and online discussion structure. The classifier obtained 67% accuracy and Cohen's κ of 0.32, showing a moderate level of inter-rater agreement above chance and the general viability of the proposed approach. Most importantly, the findings suggest that certain aspects of cognitive presence construct are highly generalizable and transfer across different languages. Finally, the paper also presents a novel method for addressing class imbalance problem using a generic algorithm heuristic technique, which provided substantial improvements over the use of imbalanced dataset. Results and practical implications are further discussed.

Feedback is a crucial element in helping students identify gaps and assess their learning progress. In online courses, feedback becomes even more critical as it is one of the resources where the teacher interacts directly with the student. However, with the growing number of students enrolled in online learning, it becomes a challenge for instructors to provide good quality feedback that helps the student self-regulate. In this context, this paper proposed a content analysis of feedback text provided by instructors based on different indicators of good feedback. A random forest classifier was trained and evaluated at different feedback levels. The results achieved outcomes up to 87% and 0.39 of accuracy and Cohen's κ, respectively. The paper also provides insights into the most influential textual features of feedback that predict feedback quality.

Learning Analytics and Knowledge (LAK) and Educational Data Mining (EDM) are two of the most popular venues for researchers and practitioners to report and disseminate discoveries in data-intensive research on technology-enhanced education. After the development of about a decade, it is time to scrutinize and compare these two venues. By doing this, we expected to inform relevant stakeholders of a better understanding of the past development of LAK and EDM and provide suggestions for their future development. Specifically, we conducted an extensive comparison analysis between LAK and EDM from four perspectives, including (i) the topics investigated; (ii) community development; (iii) community diversity; and (iv) research impact. Furthermore, we applied one of the most widely-used language modeling techniques (Word2Vec) to capture words used frequently by researchers to describe future works that can be pursued by building upon suggestions made in the published papers to shed light on potential directions for future research.

Several tools to support learning processes based on educational data have emerged from research on Learning Analytics (LA) in the last few years. These tools aim to support students and instructors in daily activities, and academic managers in making institutional decisions. Although the adoption of LA tools is spreading, the field still needs to deepen the understanding of the contexts where learning takes place, and of the views of the stakeholders involved in implementing and using these tools. In this sense, the SHEILA framework proposes a set of instruments to perform a detailed analysis of the expectations and needs of different stakeholders in higher education institutions, regarding the adoption of LA. Moreover, there is a lacuna in research on stakeholders' expectations from LA outside the Global North. Therefore, this paper reports on the findings of the application of interviews and focus groups, based on the SHEILA framework, with students and teaching staff from a Brazilian public university, to investigate their perceptions of the potential benefits and risks of using LA in higher education in the country. Findings indicate that there is a high interest in using LA for improving the learning experience, in particular, being able to provide personalized feedback, to adapt teaching practices to students' needs, and to make evidence-based pedagogical decisions. From the analysis of these perspectives, we point to opportunities for using LA in Brazilian higher education.

This paper describes work in progress to answer the question of how we can identify and model the depth and quality of student participation in class discussion forums using the content of the discussion forum messages. We look at two widely-studied frameworks for assessing critical discourse and cognitive engagement: the ICAP and Community of Inquiry (CoI) frameworks. Our goal is to discover where they agree and where they offer complementary perspectives on learning.

This paper presents an approach to automatic labeling of the content of messages in online discussion according to the categories of social presence. To achieve this goal, the proposed approach is based on a combination of traditional text mining features and word counts extracted with the use of established linguistic frameworks (i.e., LIWC and Coh-metrix). The best performing classifier obtained 0.95 and 0.88 for accuracy and Cohen's kappa, respectively. This paper also provides some theoretical insights into the nature of social presence by looking at the classification features that were most relevant for distinguishing between the different categories. Finally, this study adopted epistemic network analysis to investigate the structural construct validity of the automatic classification approach. Namely, the analysis showed that the epistemic networks produced based on messages manually and automatically coded produced nearly identical results. This finding thus produced evidence of the structural validity of the automatic approach.

Studying online requires well-developed self-regulated learning skills to properly manage one's learning strategies. Learning analytics research has proposed novel methods for extracting theoretically meaningful learning strategies from trace data originating from formal learning settings (online, blended, or flipped classroom). Thus identified strategies proved to be associated with academic achievement. However, automated extraction of theoretically meaningful learning strategies from trace data in the context of massive open online courses (MOOCs) is still under-explored. Moreover, there is a lacuna in research on the relations between automatically detected strategies and the established psychological constructs. The paper reports on a study that (a) applied a state-of-the-art analytic method that combines process and sequence mining techniques to detect learning strategies from the trace data collected in a MOOC (N=1,397), and (b) explored associations of the detected strategies with academic performance and personality traits (Big Five). Four learning strategies detected with the adopted analytics method were shown to be theoretically interpretable as the well-known approaches to learning. The results also revealed that the four detected learning strategies were predicted by conscientiousness, emotional instability, and agreeableness and were associated with academic performance. Implications for theoretical validity and practical application of analytics-detected learning strategies are also provided.

The temporal and sequential nature of learning is receiving increasing focus in Learning Analytics circles. The desire to embed studies in recognised theories of self-regulated learning (SRL) has led researchers to conceptualise learning as a process that unfolds and changes over time. To that end, a body of research knowledge is growing which states that traditional frequency-based correlational studies are limited in narrative impact. To further explore this, we analysed trace data collected from online activities of a sample of 239 computer engineering undergraduate students enrolled on a course that followed a flipped class-room pedagogy. We employed SRL categorisation of micro-level processes based on a recognised model of learning, and then analysed the data using: 1) simple frequency measures; 2) epistemic network analysis; 3) temporal process mining; and 4) stochastic process mining. We found that a combination of analyses provided us with a richer insight into SRL behaviours than any one single method. We found that better performing learners employed more optimal behaviours in their navigation through the course's learning management system.

Performance and consistency play a large role in learning. This study analyzes the relation between consistency in students' online work habits and academic performance in a blended course. We utilize the data from logs recorded by a learning management system (LMS) in two information technology courses. The two courses required the completion of monthly asynchronous online discussion tasks and weekly assignments, respectively. We measure consistency by using Data Time Warping (DTW) distance for two successive tasks (assignments or discussions), as an appropriate measure to assess similarity of time series, over 11-day timeline starting 10 days before and up to the submission deadline. We found meaningful clusters of students exhibiting similar behavior and we use these to identify three distinct consistency patterns: highly consistent, incrementally consistent, and inconsistent users. We also found evidence of significant associations between these patterns and learner's academic performance.

Learning analytics promises to support adaptive learning in higher education. However, the associated issues around privacy protection, especially their implications for students as data subjects, has been a hurdle to wide-scale adoption. In light of this, we set out to understand student expectations of privacy issues related to learning analytics and to identify gaps between what students desire and what they expect to happen or choose to do in reality when it comes to privacy protection. To this end, an investigation was carried out in a UK higher education institution using a survey (N=674) and six focus groups (26 students). The study highlight a number of key implications for learning analytics research and practice: (1) purpose, access, and anonymity are key benchmarks of ethics and privacy integrity; (2) transparency and communication are key levers for learning analytics adoption; and (3) information asymmetry can impede active participation of students in learning analytics.

This paper reports on the findings of a study that proposed a novel learning analytics methodology that combines three complimentary techniques - agglomerative hierarchical clustering, epistemic network analysis, and process mining. The methodology allows for identification and interpretation of self-regulated learning in terms of the use of learning strategies. The main advantage of the new technique over the existing ones is that it combines the time management and learning tactic dimensions of learning strategies, which are typically studied in isolation. The new technique allows for novel insights into learning strategies by studying the frequency of, strength of connections between, and ordering and time of execution of time management and learning tactics. The technique was validated in a study that was conducted on the trace data of first-year undergraduate students who were enrolled into two consecutive offerings (N2017 = 250 and N2018 = 232) of a course at an Australian university. The application of the proposed technique identified four strategy groups derived from three distinct time management tactics and five learning tactics. The tactics and strategies identified with the technique were correlated with academic performance and were interpreted according to the established theories and practices of self-regulated learning.

Learning design research has predominately relied upon survey- and interview-based methodologies, both of which are subject to limitations of social desirability and recall. An alternative approach is offered in this manuscript, whereby physical and online learning activity data is analysed using Epistemic Network Analysis. Using a sample of 6,040 course offerings from 10 faculties across a four year period (2016--2019), the utility of networks to understand learning design is illustrated. Specifically, through the adoption of a network analytic approach, the following was found: universities are clearly committed to blended learning, but there are considerable differences both between and within disciplines.

Adaptive learning systems (ALSs) aim to provide an efficient, effective and customised learning experience for students by dynamically adapting learning content to suit their individual abilities or preferences. Despite consistent evidence of their effectiveness and success in improving student learning over the past three decades, the actual impact and adoption of ALSs in education remain restricted to mostly research projects. In this paper, we provide a brief overview of reflections and lessons learned from developing and piloting an ALS in a course on relational databases. While our focus has been on adaptive learning, many of the presented lessons are also applicable to the development and adoption of educational tools and technologies in general. Our aim is to provide insight for other instructors, educational researchers and developers that are interested in adopting ALSs or are involved in the implementation of educational tools and technologies.

Interest in learning analytics (LA) has grown rapidly among higher education institutions (HEIs). However, the maturity levels of HEIs in terms of being ‘student data-informed’ are only at early stages. There often are barriers that prevent data from being used systematically and effectively. To assist higher education institutions to become more mature users and custodians of digital data collected from students during their online learning activities, the SHEILA framework, a policy development framework that supports systematic, sustainable and responsible adoption of LA at an institutional level, was recently built. This paper presents a mix-method study using a group concept mapping (GCM) approach that was conducted with LA experts to explore essential features of LA policy in HEI in contribution the development of the framework. The study identified six clusters of features that an LA policy should include, provided ratings based on ease of implementation and importance for each of the six themes, and offered suggestions to HEIs how they can proceed with the development of LA policies.

Feedback is an essential component of any learning experience. It allows students to identify gaps in their learning and improve their self-regulation. However, providing useful feedback is a challenging and time-consuming task. In digital learning environments, this challenge is even more significant due to a large number of students. Thus, this paper reports on the findings of an analysis of the quality of feedback provided by instructors in an online course. The paper also proposes a supervised machine learning algorithm that can identify the presence of good practices in feedback messages sent to students in a digital learning environment. The results reveal the most commonly used kinds of feedback and how to identify them automatically. The results of the study could potentially be used to improve the quality of the feedback provided by instructors in online education.

In recent years, several tools to support learning processes have emerged from the research on Learning Analytics (LA). However, little attention has been given to the contexts where learning takes place, and to the stakeholders involved in implementing and using these tools. Following the SHEILA Framework, we present the results of interviews and focus groups with students from a Brazilian public university on their perceptions of the potential and risks of using LA in higher education. Findings indicate great difficulties to use the Learning Management System; lack of effective communication with teaching staff; lack of quality and timely feedback; high perceived value of personalization; rejection of competition and rankings; issues with self-confidence; and trustworthiness of the educational institution. From the analysis of students' perspectives, we point to opportunities to use LA in the context of a Brazilian public university.

Student retention and learning in STEM disciplines is a growing problem. The 2012 report by the US President's Council of Advisors on Science and Technology (PCAST) predicts a future deficit in science, engineering, and mathematics (STEM) in the following decade and emphasizes the importance of addressing this issue. With this as a motivating factor, the OSBLE+ Social Programming Environment (SPE) was used to leverage social and programming data for the basis of automatically generated prompts inserted into the SPE. These prompts were designed to stimulate help-seeking, help-giving, and social interaction in the learning environment. A social network analysis was performed in order to determine whether exposure to the automated interventions would positively affect the relationship among students over time. Results of this study suggest that students in the experimental treatment who were presented with automated prompts developed more connected and social networks than those in the control treatment.

This paper presents an approach to studying relationships between students' social presence and course topics from transcripts of asynchronous discussions in online learning environments. Specifically, the paper uses topic modeling and epistemic network analysis to investigate how students' social presence is expressed across different course topics. Finally, we show how this method can be adopted to examine how students' social presence changed due to an instructional intervention. The results of this study and its implications are further discussed.

The widespread use of online discussion forums in educational settings provides a rich source of data for researchers interested in how collaboration and interaction can foster effective learning. Such online behaviour can be understood through the Community of Inquiry framework, and the cognitive presence construct in particular can be used to characterise the depth of a student's critical engagement with course material. Automated methods have been developed to support this task, but many studies used small data sets, and there have been few replication studies.

Student engagement is often considered an overarching construct in educational research and practice. Though frequently employed in the learning analytics literature, engagement has been subjected to a variety of interpretations and there is little consensus regarding the very definition of the construct. This raises grave concerns with regards to construct validity: namely, do these varied metrics measure the same thing? To address such concerns, this paper proposes, quantifies, and validates a model of engagement which is both grounded in the theoretical literature and described by common metrics drawn from the field of learning analytics. To identify a latent variable structure in our data we used exploratory factor analysis and validated the derived model on a separate sub-sample of our data using confirmatory factor analysis. To analyze the associations between our latent variables and student outcomes, a structural equation model was fitted, and the validity of this model across different course settings was assessed using MIMIC modeling. Across different domains, the broad consistency of our model with the theoretical literature suggest a mechanism that may be used to inform both interventions and course design.

The use of learning trace data together with various analytical methods has proven successful in detecting patterns in learning behaviour, identifying student profiles, and clustering learning resources. However, interpretation of the findings is often difficult and uncertain due to a lack of contextual data (e.g., data on student motivation, emotion or curriculum design). In this study we explored the integration of student self-reports about cognitive load and self-efficacy into the learning process and collection of relevant students' perceptions as learning traces. Our objective was to examine the association of traced measures of relevant learning constructs (cognitive load and self-efficacy) with i) indicators of the students' learning behaviour derived from trace data, and ii) the students' academic performance. The results indicated the presence of association between some indicators of students' engagement with learning activities and traced measures of cognitive load and self-efficacy. Correlational analysis demonstrated significant positive correlation between the students' course performance and traced measures of cognitive load and self-efficacy.

Learning Analytics Dashboards (LAD) are becoming an increasingly popular way to provide students with personalised feedback. Despite the number of LADs being developed, significant research gaps exist around the student perspective, especially how students make sense of graphics provided in LADs, and how they intend to act on the feedback provided therein. This study employed a randomized-controlled trial to examine students' sense-making of LADs showing four different frames of reference, and to what extent the impact of LADs was mediated by baseline self-regulation. Using a mix of quantitative and qualitative data analysis, the results revealed rather distinct patterns in students' sense-making across the four LADs. These patterns involved the intersection of visual salience and planned learning actions. However, collectively, across all four LADs a consistent theme emerged around students planned learning actions. This theme was classified as time and study environment management. A key finding of the study is that the use of LADs as a primary feedback process should be personalized and include training and support to aid student sensemaking.

Learning analytics has the potential to detect and explain characteristics of learning strategies through analysis of trace data and communicate the findings via feedback. However, the role of learning analytics-based feedback in selection and regulation of learning strategies is still insufficiently explored and understood. This research aims to examine the sequential and temporal characteristics of learning strategies and investigate their association with feedback. Three years of trace data were collected from online pre-class activities of a flipped classroom, where different types of feedback were employed in each year. Clustering, sequence mining, and process mining were used to detect and interpret learning tactics and strategies. Inferential statistics were used to examine the association of feedback with the learning performance and the detected learning strategies. The results suggest a positive association between the personalised feedback and the effective strategies.

With increasing abundance and ubiquity of mobile phones, desktop PCs, and tablets in the last decade, we are seeing students intermixing these modalities to learn and regulate their learning. However, the role of these modalities in educational settings is still largely under-researched. Similarly, little attention has been paid to the research on the extension of learning analytics to analyze the learning processes of students adopting various modalities during a learning activity. Traditionally, research on how modalities affect the way in which activities are completed has mainly relied upon self-reported data or mere counts of access from each modality. We explore the use of technological modalities in regulating learning via learning management systems (LMS) in the context of blended courses. We used data mining techniques to analyze patterns in sequences of actions performed by learners (n = 120) across different modalities in order to identify technological modality profiles of sequences. These profiles were used to detect the technological modality strategies adopted by students. We found a moderate effect size (∈2 = 0.12) of students' adopted strategies on the final course grade. Furthermore, when looking specifically at online discussion engagement and performance, students' adopted technological modality strategies explained a large amount of variance (η2 = 0.68) in their engagement and quality of contributions. The result implications and further research are discussed.

Research on learner behaviors and course completion within Massive Open Online Courses (MOOCs) has been mostly confined to single courses, making the findings difficult to generalize across different data sets and to assess which contexts and types of courses these findings apply to. This paper reports on the development of the MOOC Replication Framework (MORF), a framework that facilitates the replication of previously published findings across multiple data sets and the seamless integration of new findings as new research is conducted or new hypotheses are generated. In the proof of concept presented here, we use MORF to attempt to replicate 15 previously published findings across 29 iterations of 17 MOOCs. The findings indicate that 12 of the 15 findings replicated significantly across the data sets, and that two findings replicated significantly in the opposite direction. MORF enables larger-scale analysis of MOOC research questions than previously feasible, and enables researchers around the world to conduct analyses on huge multi-MOOC data sets without having to negotiate access to data.

Despite strong interest in learning analytics (LA), adoption at a large-scale organizational level continues to be problematic. This may in part be due to the lack of acknowledgement of existing conceptual LA models to operationalize how key adoption dimensions interact to inform the realities of the implementation process. This paper proposes the framing of LA adoption in complexity leadership theory (CLT) to study the overarching system dynamics. The framing is empirically validated in a study analysing interviews with senior staff in Australian universities (n=32). The results were coded for several adoption dimensions including leadership, governance, staff development, and culture. The coded data were then analysed with latent class analysis. The results identified two classes of universities that either i) followed an instrumental approach to adoption - typically top-down leadership, large scale project with high technology focus yet demonstrating limited staff uptake; or ii) were characterized as emergent innovators - bottom up, strong consultation process, but with subsequent challenges in communicating and scaling up innovations. The results suggest there is a need to broaden the focus of research in LA adoption models to move on from small-scale course/program levels to a more holistic and complex organizational level.

Reflective writing has been widely recognized as one of the most effective activities for fostering students' reflective and critical thinking. The analysis of students' reflective writings has been the focus of many research studies. However, to date this has been typically a very labor-intensive manual process involving content analysis of student writings. With recent advancements in the field of learning analytics, there have been several attempts to use text analytics to examine student reflective writings. This paper presents the results of a study examining the use of theoretically-sound linguistic indicators of different psychological processes for the development of an analytics system for assessment of reflective writing. More precisely, we developed a random-forest classification system using linguistic indicators provided by the LIWC and Coh-Metrix tools. We also examined what particular indicators are representative of the different types of student reflective writings.

This paper introduces a learning analytics policy development framework developed by a cross-European research project team - SHEILA (Supporting Higher Education to Integrate Learning Analytics), based on interviews with 78 senior managers from 51 European higher education institutions across 16 countries. The framework was developed using the RAPID Outcome Mapping Approach (ROMA), which is designed to develop effective strategies and evidence-based policy in complex environments. This paper presents three case studies to illustrate the development process of the SHEILA policy framework, which can be used to inform strategic planning and policy processes in real world environments, particularly for large-scale implementation in higher education contexts.

The role of alignment between interlocutors in second language learning is different to that in fluent conversational dialogue.Learners gain linguistic skill through increased alignment, yet the extent to which they can align will be constrained by their ability.Tutors may use alignment to teach and encourage the student, yet still must push the student and correct their errors, decreasing alignment.To understand how learner ability interacts with alignment, we measure the influence of ability on lexical priming, an indicator of alignment.We find that lexical priming in learner-tutor dialogues differs from that in conversational and task-based dialogues, and we find evidence that alignment increases with ability and with word complexity.

In our days, the interaction of Behaviour and Information Technology is challenged by emerging technologies, including cognitive computing and business intelligence. The learning domain has many fe...

In this poster, we describe the aim and current activities of the EARLI-Centre for Innovative Research (E-CIR) "Measuring and Supporting Student's Self-Regulated Learning in Adaptive Educational Technologies" which is funded by the European Association for Research on Learning and Instruction (EARLI) from 2015 to 2019. The aim is to develop our understanding of multimodal data that unobtrusively capture cognitive, meta-cognitive, affective and motivational states of learners over time. This demands for a concerted interdisciplinary dialogue combining findings from psychology and educational sciences with advances in computer sciences and artificial intelligence. The participants in this E-CIR are leading international researchers who have articulated different emerging perspectives and methodologies to measure cognition, metacognition, motivation, and emotions during learning. The participants recognize the need for intensive collaboration to accelerate progress with new interdisciplinary methods including learning analytics to develop more powerful adaptive educational technologies.

Learning analytics research has often been touted as a means to address concerns regarding student retention outcomes. However, few research studies to date, have examined the impact of the implemented intervention strategies designed to address such retention challenges. Moreover, the methodological rigor of some of the existing studies has been challenged. This study evaluates the impact of a pilot retention program. The study contrasts the findings obtained by the use of different methods for analysis of the effect of the intervention. The pilot study was undertaken between 2012 and 2014 resulting in a combined enrolment of 11,160 students. A model to predict attrition was developed, drawing on data from student information system, learning management system interactions, and assessment. The predictive model identified some 1868 students as academically at-risk. Early interventions were implemented involving learning and remediation support. Common statistical methods demonstrated a positive association between the intervention and student retention. However, the effect size was low. The use of more advanced statistical methods, specifically mixed-effect methods explained higher variability in the data (over 99%), yet found the intervention had no effect on the retention outcomes. The study demonstrates that more data about individual differences is required to not only explain retention but to also develop more effective intervention approaches.

In 2011, the phenomenon of MOOCs had swept the world of education and put online education in the focus of the public discourse around the world. Although researchers were excited with the vast amounts of MOOC data being collected, the benefits of this data did not stand to the expectations due to several challenges. The analyses of MOOC data are very time-consuming and labor-intensive, and require and require a highly advanced set of technical skills, often not available to the education researchers. Because of this MOOC data analyses are rarely done before the courses end, limiting the potential of data to impact the student learning outcomes and experience.

In this poster, we present the results of the study which examined the relationship between student differences in their use of the available technology and their perceived levels of cognitive presence within the MOOC context. The cognitive presence is a construct used to measure the level of practical inquiry in the Communities of Inquiry model. Our results revealed the existence of three clusters based on student technology use. The clusters significantly differed in terms of their levels of cognitive presence, most notably they differed on the levels of problem resolution.

The amount of data extracted from learning experiences has grown at an astonishing pace both in depth due to the increasing variety of data sources, and in breath with courses now being offered to massive student cohorts. However, in this emerging scenario instructors are now facing the challenge of connecting the knowledge emerging from data analysis with the provision of meaningful support actions to students within the context of an instructional design.

This paper presents the results of a review of eight policies for learning analytics of relevance for higher education, and discusses how these policies have tried to address prominent challenges in the adoption of learning analytics, as identified in the literature. The results show that more considerations need to be given to establishing communication channels among stakeholders and adopting pedagogy-based approaches to learning analytics. It also reveals the shortage of guidance for developing data literacy among end-users and evaluating the progress and impact of learning analytics. Moreover, the review highlights the need to establish formalised guidelines to monitor the soundness, effectiveness, and legitimacy of learning analytics. As interest in learning analytics among higher education institutions continues to grow, this review will provide insights into policy and strategic planning for the adoption of learning analytics.

This workshop aims to promote strategic planning for learning analytics in higher education through developing institutional policies. While adoption of learning analytics is predominantly seen in small-scale and bottom-up patterns, it is believed that a systemic implementation can bring the widest impact to the education system and lasting benefits to learners. However, the success of it highly depends on the adopted strategy that meets the needs of various stakeholders and systematically pushes the institution towards achieving its targets. It is imperative to develop a learning analytics policy that ensures a practice that is valid, effective and ethical.

Quality assurance in any organization is important for ensuring that service users are satisfied with the service offered. For higher education institutes, the use of service quality measures allows for ideological gaps to be both identified and resolved. The learning analytic community, however, has rarely addressed the concept of service quality. A potential outcome of this is the provision of a learning analytics service that only meets the expectations of certain stakeholders (e.g., managers), whilst overlooking those who are most important (e.g., students). In order to resolve this issue, we outline a framework and our current progress towards developing a scale to assess student expectations and perceptions of learning analytics as a service.

There is an emerging trend in higher education for the adoption of massive open online courses (MOOCs). However, despite this interest in learning at scale, there has been limited work investigating how MOOC participants have changed over time. In this study, we explore the temporal changes in MOOC learners' language and discourse characteristics. In particular, we demonstrate that there is a clear trend within a course for language in discussion forums to be of both more on-topic and reflective of deep learning in subsequent offerings of a course. We measure this in two ways, and demonstrate this trend through several repeated analyses of different courses in different domains. While not all courses show an increase beyond statistical significance, the majority do, providing evidence that MOOC learner populations are changing as the educational phenomena matures.

In this paper we present a learning analytics conceptual framework that supports enquiry-based evaluation of learning designs. The dimensions of the proposed framework emerged from a review of existing analytics tools, the analysis of interviews with teachers, and user scenarios to understand what types of analytics would be useful in evaluating a learning activity in relation to pedagogical intent. The proposed framework incorporates various types of analytics, with the teacher playing a key role in bringing context to the analysis and making decisions on the feedback provided to students as well as the scaffolding and adaptation of the learning design. The framework consists of five dimensions: temporal analytics, tool-specific analytics, cohort dynamics, comparative analytics and contingency. Specific metrics and visualisations are defined for each dimension of the conceptual framework. Finally the development of a tool that partially implements the conceptual framework is discussed.

An ongoing challenge for Learning Analytics research has been the scalable derivation of user interaction data from multiple technologies. The complexities associated with this challenge are increasing as educators embrace an ever growing number of social and content-related technologies. The Experience API (xAPI) alongside the development of user specific record stores has been touted as a means to address this challenge, but a number of subtle considerations must be made when using xAPI in Learning Analytics. This paper provides a general overview to the complexities and challenges of using xAPI in a general systemic analytics solution - called the Connected Learning Analytics (CLA) toolkit. The importance of design is emphasised, as is the notion of common vocabularies and xAPI Recipes. Early decisions about vocabularies and structural relationships between statements can serve to either facilitate or handicap later analytics solutions. The CLA toolkit case study provides us with a way of examining both the strengths and the weaknesses of the current xAPI specification, and we conclude with a proposal for how xAPI might be improved by using JSON-LD to formalise Recipes in a machine readable form.

When designing learning analytics tools for use by learners we have an opportunity to provide tools that consider a particular learner's situation and the learner herself. To afford actual impact on learning, such tools have to be informed by theories of education. Particularly, educational research shows that individual differences play a significant role in explaining students' learning process. However, limited empirical research in learning analytics has investigated the role of theoretical constructs, such as motivational factors, that are underlying the observed differences between individuals. In this work, we conducted a field experiment to examine the effect of three designed learning analytics visualizations on students' participation in online discussions in authentic course settings. Using hierarchical linear mixed models, our results revealed that effects of visualizations on the quantity and quality of messages posted by students with differences in achievement goal orientations could either be positive or negative. Our findings highlight the methodological importance of considering individual differences and pose important implications for future design and research of learning analytics visualizations.

An important research problem in learning analytics is to expedite the cycle of data leading to the analysis of student progress and the improvement of student support. For this goal in the context of social learning, we propose a pipeline that includes data infrastructure, learning analytics, and intervention, along with computational models for individual components. Next, we describe an example of applying this pipeline to real data in a case study, whose goal is to investigate the positive effects that goal-setting students have on their peers, which suggests ways in which we might foster these social benefits through intervention.

As the field of learning analytics continues to mature, there is a corresponding evolution and sophistication of the associated analytical methods and techniques. In this regard social network analysis (SNA) has emerged as one of the cornerstones of learning analytics methodologies. However, despite the noted importance of social networks for facilitating the learning process, it remains unclear how and to what extent such network measures are associated with specific learning outcomes. Motivated by Simmel's theory of social interactions and building on the argument that social centrality does not always imply benefits, this study aimed to further contribute to the understanding of the association between students' social centrality and their academic performance. The study reveals that learning analytics research drawing on SNA should incorporate both - descriptive and statistical methods to provide a more comprehensive and holistic understanding of a students' network position. In so doing researchers can undertake more nuanced and contextually salient inferences about learning in network settings. Specifically, we show how differences in the factors framing students' interactions within two instances of a MOOC affect the association between the three social network centrality measures (i.e., degree, closeness, and betweenness) and the final course outcome.

This demonstration introduces the Connected Learning Analytics (CLA) Toolkit. The CLA toolkit harvests data about student participation in specified learning activities across standard social media environments, and presents information about the nature and quality of the learning interactions.

In this paper, we present the results of an exploratory study that examined the problem of automating content analysis of student online discussion transcripts. We looked at the problem of coding discussion transcripts for the levels of cognitive presence, one of the three main constructs in the Community of Inquiry (CoI) model of distance education. Using Coh-Metrix and LIWC features, together with a set of custom features developed to capture discussion context, we developed a random forest classification system that achieved 70.3% classification accuracy and 0.63 Cohen's kappa, which is significantly higher than values reported in the previous studies. Besides improvement in classification accuracy, the developed system is also less sensitive to overfitting as it uses only 205 classification features, which is around 100 times less features than in similar systems based on bag-of-words features. We also provide an overview of the classification features most indicative of the different phases of cognitive presence that gives an additional insights into the nature of cognitive presence learning cycle. Overall, our results show great potential of the proposed approach, with an added benefit of providing further characterization of the cognitive presence coding scheme.

The pervasive collection of data has opened the possibility for educational institutions to use analytics methods to improve the quality of the student experience. However, the adoption of these methods faces multiple challenges particularly at the course level where instructors and students would derive the most benefit from the use of analytics and predictive models. The challenge lies in the knowledge gap between how the data is captured, processed and used to derive models of student behavior, and the subsequent interpretation and the decision to deploy pedagogical actions and interventions by instructors. Simply put, the provision of learning analytics alone has not necessarily led to changing teaching practices. In order to support pedagogical change and aid interpretation, this paper proposes a model that can enable instructors to readily identify subpopulations of students to provide specific support actions. The approach was applied to a first year course with a large number of students. The resulting model classifies students according to their predicted exam scores, based on indicators directly derived from the learning design.

Massive Open Online Courses represent a fertile ground for examining student behavior. However, due to their openness MOOC attract a diverse body of students, for the most part, unknown to the course instructors. However, a certain number of students enroll in the same course multiple times, and there are records of their previous learning activities which might provide some useful information to course organizers before the start of the course. In this study, we examined how student behavior changes between subsequent course offerings. We identified profiles of returning students and also interesting changes in their behavior between two enrollments to the same course. Results and their implications are further discussed.

Research on self-regulated learning has taken main two paths: self-regulated learning as aptitudes and more recently, self-regulated learning as events. This paper proposes the use of the Fuzzy miner process mining technique to examine the relationship between students' self-reported aptitudes (i.e., achievement goal orientation and approaches to learning) and strategies followed in self-regulated learning. A pilot study is conducted to probe the method and the preliminary results are reported.

Connections established between learners via interactions are seen as fundamental for connectivist pedagogy. Connections can also be viewed as learning outcomes, i.e. learners' social capital accumulated through distributed learning environments. We applied linear mixed effects modeling to investigate whether the social capital accumulation interpreted through learners' centrality to course interaction networks, is influenced by the language learners use to express and communicate in two connectivist MOOCs. Interactions were distributed across the three social media, namely Twitter, blog and Facebook. Results showed that learners in a cMOOC connect easier with the individuals who use a more informal, narrative style, but still maintain a deeper cohesive structure to their communication.

Creating meaning from a wide variety of available information and being able to choose what to learn are highly relevant skills for learning in a connectivist setting. In this work, various approaches have been utilized to gain insights into learning processes occurring within a network of learners and understand the factors that shape learners' interests and the topics to which learners devote a significant attention. This study combines different methods to develop a scalable analytic approach for a comprehensive analysis of learners' discourse in a connectivist massive open online course (cMOOC). By linking techniques for semantic annotation and graph analysis with a qualitative analysis of learner-generated discourse, we examined how social media platforms (blogs, Twitter, and Facebook) and course recommendations influence content creation and topics discussed within a cMOOC. Our findings indicate that learners tend to focus on several prominent topics that emerge very quickly in the course. They maintain that focus, with some exceptions, throughout the course, regardless of readings suggested by the instructor. Moreover, the topics discussed across different social media differ, which can likely be attributed to the affordances of different media. Finally, our results indicate a relatively low level of cohesion in the topics discussed which might be an indicator of a diversity of the conceptual coverage discussed by the course participants.

All forms of learning take time. There is a large body of research suggesting that the amount of time spent on learning can improve the quality of learning, as represented by academic performance. The wide-spread adoption of learning technologies such as learning management systems (LMSs), has resulted in large amounts of data about student learning being readily accessible to educational researchers. One common use of this data is to measure time that students have spent on different learning tasks (i.e., time-on-task). Given that LMS systems typically only capture times when students executed various actions, time-on-task measures are estimated based on the recorded trace data. LMS trace data has been extensively used in many studies in the field of learning analytics, yet the problem of time-on-task estimation is rarely described in detail and the consequences that it entails are not fully examined.

The higher education sector has seen a shift in teaching approaches over the past decade with an increase in the use of video for delivering lecture content as part of a flipped classroom or blended learning model. Advances in video technologies have provided opportunities for students to now annotate videos as a strategy to support their achievement of the intended learning outcomes. However, there are few studies exploring the relationship between video annotations, student approaches to learning, and academic performance. This study seeks to narrow this gap by investigating the impact of students' use of video annotation software coupled with their approaches to learning and academic performance in the context of a flipped learning environment. Preliminary findings reveal a significant positive relationship between annotating videos and exam results. However, negative effects of surface approaches to learning, cognitive strategy use and test anxiety on midterm grades were also noted. This indicates a need to better promote and scaffold higher order cognitive strategies and deeper learning with the use of video annotation software.

To be considered secure, a payment system needs to address a number of security issues. Besides fundamental security requirements, like confidentiality, data integrity, authentication and non-repudiation, another important requirement for a secure payment system is fair exchange. Many existing payment protocols require that customers must pay for products before their delivery (in the case of delivery of digital goods) or the delivery of the receipt (in the case of delivery of physical goods). This unfair situation should be eliminated afterward; that is, it is necessary to rebalance fairness for customers. To address these issues, we propose the Fair Exchange Internet Payment Protocol (FEIPS). The FEIPS protocol is designed for the payment of physical goods and falls into the category that uses a trusted third party for ensuring fair exchange. Although FEIPS has a strong emphasis on fair exchange, it still guarantees strong security properties, including confidentiality, data integrity, authentication and non-repudiation. The FEIPS protocol is designed to be simple and practical, unlike other similar protocols designed for the payment of physical goods. To demonstrate that FEIPS satisfies the desired properties, we perform a formal verification using the HLPSL language and the AVISPA tool.

Abstract The combination of Agile methods and distributed software development via remote teams represents an emerging approach to address the challenges such as late feedback, slow project timelines, and high cost, typically associated with software development projects. However, when projects are implemented using an Agile model with distributed human resources, there are a number of challenges that need to be considered and mitigated. The objectives of our work are multifold. First, we would like to understand the reasons and conditions that lead to the adoption of distributed Agile software engineering (DASE) practices. Second, we would like to investigate and find out the most important risks that threaten a DASE approach and what mitigation strategies exist to address them. Finally, we would like to highlight which of the available approaches among the existing Agile methodologies has been successfully adopted by the community. We intend to solidify our findings by exploring the strength of the evidence that has been reported in the literature. We carried out a systematic literature review of DASE techniques and approaches. This systematic literature review found time zone difference, knowledge of resources, lack of infrastructure, missing roles, and responsibilities as being the primary challenges that needed to be addressed. Copyright © 2015 John Wiley & Sons, Ltd.

The concept of social goals refers to organizational goals that are defined in an open and transparent manner; they serve as social objects that incite both formal and informal collaboration around shared interests/objectives. Our objective is to facilitate the comprehension of social goals and examine the role of social goals as scaffolds of social learning in an organization. To this end, we followed an approach based on the visualization of social goals and explored how different presentations of goals, specifically, faceted goal browsing, graph-based visualization, and timeline-based visualization, contribute to the realization of the stated objective. To assess this approach, we conducted a between subjects study where each participant performed a set of goal comprehension tasks with one of the examined presentations of goals. The study demonstrated that our visualizations of goals increase the accuracy of the overall comprehension of an organization's goals; this positive effect is also present when the comprehension of relationships-either explicit or implicit ties-among social goals is needed. The results also confirmed that our graph-based visualization of social goals could serve as a facilitator of social learning in an organization.

This paper provides an evaluation of the current state of the field of learning analytics through analysis of articles and citations occurring in the LAK conferences and identified special issue journals. The emerging field of learning analytics is at the intersection of numerous academic disciplines, and therefore draws on a diversity of methodologies, theories and underpinning scientific assumptions. Through citation analysis and structured mapping we aimed to identify the emergence of trends and disciplinary hierarchies that are influencing the development of the field to date. The results suggest that there is some fragmentation in the major disciplines (computer science and education) regarding conference and journal representation. The analyses also indicate that the commonly cited papers are of a more conceptual nature than empirical research reflecting the need for authors to define the learning analytics space. An evaluation of the current state of learning analytics provides numerous benefits for the development of the field, such as a guide for under-represented areas of research and to identify the disciplines that may require more strategic and targeted support and funding opportunities.

Although video annotation software is no longer considered as a new innovation, its application in promoting student self-regulated learning and reflection skills has only begun to emerge in the research literature. Advances in text and video analytics provide the capability of investigating students' use of the tool and the psychometrics and linguistic processes evident in their written annotations. This paper reports on a study exploring students' use of a video annotation tool when two different instructional approaches were deployed -- graded and non-graded self-reflection annotations within two courses in the performing arts. In addition to counts and temporal locations of self-reflections, the Linguistic Inquiry and Word Counts (LIWC) framework was used for the extraction of variables indicative of the linguistic and psychological processes associated with self-reflection annotations of videos. The results indicate that students in the course with graded self-reflections adopted more linguistic and psychological related processes in comparison to the course with non-graded self-reflections. In general, the effect size of the graded reflections was lower for students who took both courses in parallel. Consistent with prior research, the study identified that students tend to make the majority of their self-reflection annotations early in the video time line. The paper also provides several suggestions for future research to better understand the application of video annotations in facilitating student learning.

Learning analytics (LA) as a field remains in its infancy. Many of the techniques now prominent from practitioners have been drawn from various fields, including HCI, statistics, computer science, and learning sciences. In order for LA to grow and advance as a discipline, two significant challenges must be met: 1) development of analytics methods and techniques that are native to the LA discipline, and 2) practitioners in LA to develop algorithms and models that reflect the social and computational dimensions of analytics. This workshop introduces researchers in learning analytics to machine learning (ML) and the opportunities that ML can provide in building next generation analysis models.

One of the most important prerequisites for achieving the Semantic Web vision is semantic annotation of data/resources. Semantic annotation enriches unstructured and/or semistructured content with a context that is further linked to the structured domain-specific knowledge. In particular, ontologybased semantic annotators enable the selection of a specific ontology to annotate content. This paper presents results of an empirical study of recent ontology-based annotators, namely Stanbol, KIM, and SDArch. Specifically, we evaluated the robustness of these annotators with respect to specific features of ontology concepts such as the length of concepts? labels and their linguistic categories (e.g., prepositions and conjunctions). Our results show that although significantly correlated according to most of the conducted evaluations, tools still exhibit their unique features that could be a topic of new research.

Source code plagiarism is an easy to do task, but very difficult to detect without proper tool support. Various source code similarity detection systems have been developed to help detect source code plagiarism. Those systems need to recognize a number of lexical and structural source code modifications. For example, by some structural modifications (e.g. modification of control structures, modification of data structures or structural redesign of source code) the source code can be changed in such a way that it almost looks genuine. Most of the existing source code similarity detection systems can be confused when these structural modifications have been applied to the original source code. To be considered effective, a source code similarity detection system must address these issues. To address them, we designed and developed the source code similarity system for plagiarism detection. To demonstrate that the proposed system has the desired effectiveness, we performed a well-known conformism test. The proposed system showed promising results as compared with the JPlag system in detecting source code similarity when various lexical or structural modifications are applied to plagiarized code. As a confirmation of these results, an independent samples t-test revealed that there was a statistically significant difference between average values of F-measures for the test sets that we used and for the experiments that we have done in the practically usable range of cut-off threshold values of 35–70%.

This paper describes LePress, a WordPress plug-in that enhances blog-based personal learning environment (PLE) with features and semantics that facilitate planning, implementation, and analysis of learning flows. The paper introduces learning flows in LePress, and then explains learning semantics the LePress supports. In order to demonstrate the advantages of LePress for learning analytics, we describe how it can facilitate explicit data collection and analysis of learning activities in blog-based PLEs. In order to demonstrate the advantages of LePress for learning analytics, we describe how it can facilitate explicit data collection and analysis of learning activities in blog-based PLEs.

In this design briefing, we introduce the Learn-B environment, our attempt in designing and implementing a research prototype to address some of the challenges inherent in workplace learning: the informal aspect of workplace learning requires knowledge workers to be supported in their self-regulatory learning (SRL) processes, whilst its social nature draws attention to the role of collective in those processes. Moreover, learning at workplace is contextual and on-demand, thus requiring organizations to recognize and motivate the learning and knowledge building activities of their employees, where individual learning goals are harmonized with those of the organization. In particular, we focus on the analytics-based features of Learn-B, illustrate their design and current implementation, and discuss how each of them is hypothesized to target the above challenges.

Requirements engineering is recognized as a critical stage in software development lifecycle. Given the nature of Software Product Lines (SPL), the importance of requirements engineering is more pronounced as SPLs pose more complex challenges than development of a 'single' product. Several methods have been proposed in the literature, which encompass activities for capturing requirements, their variability and commonality. To investigate the maturity and effectiveness of the current requirements engineering approaches in software product lines, we develop an evaluation framework containing a set of evaluation criteria and assess feature oriented requirements engineering methods based on the proposed criteria. As a result of this initial study, we find out the majority of approaches lacks proper techniques for supporting the validation of family requirements models as well as dealing with delta requirements. Additionally, capturing stakeholders' preferences and applying them during the course of software feature configuration have not been taken into account and addressed in the proposed approaches.

Feature modeling is one of the main techniques used in Software Product Line Engineering to manage the variability within the products of a family. Concrete products of the family can be generated through a configuration process. The configuration process selects and/or removes features from the feature model according to the stakeholders' requirements. Selecting the right set of features for one product from amongst all of the available features in the feature model is a complex task because: 1) the multiplicity of stakeholders' functional requirements; 2) the positive or negative impact of features on non-functional properties; and 3) the stakeholders' preferences w.r.t. the desirable non-functional properties of the final product. Many configurations techniques have already been proposed to facilitate automated product derivation. However, most of the current proposals are not designed to consider stakeholders' preferences and constraints especially with regard to non-functional properties. We address the software product line configuration problem and propose a framework, which employs an artificial intelligence planning technique to automatically select suitable features that satisfy both the stakeholders' functional and non-functional preferences and constraints. We also provide tooling support to facilitate the use of our framework. Our experiments show that despite the complexity involved with the simultaneous consideration of both functional and non-functional properties our configuration technique is scalable.

SUMMARY Feature modeling an attractive technique for capturing commonality as well as variability within an application domain for generative programming and software product line engineering. Feature models symbolize an overarching representation of the possible application configuration space, and can hence be customized based on specific domain requirements and stakeholder goals. Most interactive or semi‐automated feature model customization processes neglect the need to have a holistic approach towards the integration and satisfaction of the stakeholder's soft and hard constraints, and the application‐domain integrity constraints. In this paper, we will show how the structure and constraints of a feature model can be modeled uniformly through Propositional Logic extended with concrete domains, called Pscr (𝒩). Furthermore, we formalize the representation of soft constraints in fuzzy 𝒫(𝒩) and explain how semi‐automated feature model customization is performed in this setting. The model configuration derivation process that we propose respects the soundness and completeness properties. Copyright © 2011 John Wiley & Sons, Ltd.

The six papers in this special issue depict the state of the art and practice of the impact of semantic technologies in the field of Software Engineering.

In the paper, we investigate principles for modeling flexible business processes enhanced by business rules. In our work, we start from a set of rule patterns, which are identified in the literature as a mean for increasing flexibility of business processes. The previous work on these patterns only considered the implementation level, but not the implications on the modeling level. Moreover, the potential for business process flexibility have not been fully leveraged due to some limitations in externalization of business logic into business rules. In this work, we report on the experience in modeling the set of rule patterns by using a rule-enhanced business process modeling language (rBPMN), and demonstrate the applicability of our findings on a business process case study.

Self-regulated learning processes have a potential to enhance the motivation of knowledge workers to take part in learning and knowledge building activities, and thus contribute to the resolution of an important research challenge in workplace learning. An equally important research challenge for successful completion of each step of a self-regulatory process is to enable learners to be aware of characteristics of their organizationally embedded learning context. In this paper, we describe how a combination of pedagogy and Semantic Web-based technologies can be utilized to address the above two challenges. Specifically, we demonstrate the proposed solution through the Learning Pal tool which leverages ontologies to support self-regulation in organizational learning.

In Software Product Line Engineering, concrete products of a family can be generated through a configuration process over a feature model. The configuration process selects features from the feature model according to the stakeholders' requirements. Selecting the right set of features for one product from all the available features in the feature model is a cumbersome task because 1) the stakeholders may have diverse business concerns and limited resources that they can spend on a product and 2) features may have negative and positive contributions on different business concern. Many configurations techniques have been proposed to facilitate software developers' tasks through automated product derivation. However, most of the current proposals for automatic configuration are not devised to cope with business oriented requirements and stakeholders' resource limitations. We propose a framework, which employs an artificial intelligence planning technique to automatically select suitable features that satisfy the stakeholders' business concerns and resource limitations. We also provide tooling support to facilitate the use of our framework.

The research community has so far mainly focused on the problem of modeling of service orchestrations in the domain of service composition, while modeling of service choreographies has attracted less attention. The following challenges in choreography modeling are tackled in this paper: i) choreography models are not well-connected with the underlying business vocabulary models. ii) there is limited support for decoupling parts of business logic from complete choreography models. This reduces dynamic changes of choreographies, iii) choreography models contain redundant elements of shared business logic, which might lead to an inconsistent implementation and incompatible behavior. Our proposal - rBPMN - is an extension of a business process modeling language with rule and choreography modeling support. rBPMN is defined by weaving the metamodels of the Business Process Modeling Notation and REWERSE Rule Markup Language. To evaluate our proposal, we use service-interaction patterns and compare our approach with related solutions.

Semantic Web technologies have been applied to many aspects of learning content authoring including annotation, dynamic assembly, and personalization of learning content as well as authors' collaborative activities. Whether Semantic Web technologies improved the authoring process and to what extend they make authors' life easier, however, remains an open question that we try to address in this paper. We report on the results of an empirical study based on the experiments that we conducted with the prototype of a novel document architecture called SDArch. Semantic Web technologies and social networking are two pillars of SDArch, thus potential benefits of SDArch naturally extend to them. Results of the study show that the utilization of SDArch in authoring improves user' performances compared to the authoring with conventional tools. In addition, the users' satisfaction collected from their subjective feedback was also highly positive.

Since the introduction in the early nineties, feature models receive a great deal of attention in industry and academia. Industrial success stories in applying feature models for modeling software product lines, and using them for configuring software-intensive systems motivate academia to discover ways to integrate different feature dependencies into the feature model, and automate verified feature configuration. In this paper we demonstrate how ontologies and Semantic Web technologies facilitate seamless integration of required external services and deployment platform capabilities into the feature model. Furthermore, we also contribute with an algorithm for automating staged configuration using Semantic Web reasoners to discover unfeasible features of the feature model.

Having the notion of services brought into the domain of component based software design has enriched the domain of modular self-contained components with support for loose coupling, dynamic runtime discovery, and late binding. However, loose coupling, runtime discovery, and composition for a service-oriented component model in its current scale mostly applies to homogeneous services that admit to the specifications of the original service component model. This design is in conflict with the initial outlook of SOA in which reusability of existing services is considered a primary goal. In this paper, we discuss ReCoIn, a remote service composition and integration framework, which aims to bring functionalities of traditional Web services into the local repository of a service oriented component model. We discuss how ReCoIn enables remote Web services to be properly discovered, encapsulated, and used in a service oriented component model framework, offering better service reuse and fault tolerance. In this paper, we layout the overall architectural design for ReCoIn and present its initial prototype.

Business process modeling is a commonlyused approach in the development of serviceorientedarchitectures. The previous research onthis topic demonstrated that process-oriented modelsmight be too rigid for dynamic adaptations ofthe business logic. Rule-based approaches are consideredan alternative, which offers more flexibilitythanks to the declarative nature of rules and theirunderlying reasoning algorithms. However, modelinga business process through rules is a tediousprocess for developers in terms of the overall businessprocess comprehension. In this paper, we proposea hybrid solution – a modeling language thatintegrates both rule- and process-oriented modelingperspectives. The language (Rule-based BPMN –rBPMN) is based on the integration of the BusinessProcess Modeling Notation with the REWERSERule Markup Language. In this paper, after introducingrBPMN, we report on the experience inmodeling of Service-Oriented Architectures (SOA)from the perspective of message exchange patterns.

Teaching and learning software design patterns (DPs) is not an easy task. Apart from learning individual DPs and the principle behind them, students should learn how to apply them in real-life situations. Therefore, to make the learning process of DPs effective, it is necessary to include a project component in which students, usually in small teams, develop a medium-sized software application. Following this paradigm, and using active learning techniques, project-based learning (PBL) and collaborative learning (CL), we have developed a learning environment for software DPs which leverages semantic technologies to integrate several existing learning systems and tools.

Software development in highly variable domains constrained by tight regulations and with many business concepts involved results in hard to deliver and maintain applications, due to the complexity of dealing with the large number of concepts provided by the different parties and system involved in the process. One way to tackle these problems is thru combining software product lines and model-driven software development supported by ontologies. Software product lines and model-driven approaches would promote reuse on the software artifacts and, if supported by an ontological layer, those artifacts would be domain-validated. We intend to create a new conceptual framework for software development with domain validated models in highly variable domains. To define such a framework we will propose a model that relates several dimensions and areas of software development thru time and abstraction levels. This model would guarantee to the software house traceability of components, domain validated artifacts, easy to maintain and reusable components, due to the relations and mappings we propose to establish in the conceptual framework, between the software artifacts and the ontology.

Abstract The paper covers the problem of bridging the gap between abstract and textual concrete syntaxes of software languages in the model‐driven engineering (MDE) context. This problem has been well studied in the context of programming languages, but due to the obvious difference in the definitions of abstract syntax, MDE requires a new set of engineering principles. We first explore different approaches to defining abstract and concrete syntaxes in the MDE context. Next, we investigate the current state of languages and techniques used for bridging between textual concrete and abstract syntaxes in the context of MDE. Finally, we report on lessons learned in experimenting with the current technologies. In order to provide a comprehensive coverage of the problem under study, we have selected a case of Web rule languages. Web rule languages leverage various types of syntax specification languages; and they are complex in nature and large in terms of the language elements. Thus, they provide us with a realistic analysis framework based on which we can draw general conclusions. Based on the series of experiments that we conducted with the analyzed languages, we propose a method for approaching such problems and report on the empirical results obtained from the data collected during our experiments. Copyright © 2009 John Wiley & Sons, Ltd.

The six articles in this special section are devoted to software language engineering.

Today, it is almost impossible to imagine distance education without the use of numerous software systems, technologies, specifications and standards. Industry, to be able to develop and maintain "learning" systems effectively and yet to have commercially successful business models, limits the overall learning landscape, by fixing learning to a predetermined set of activities, communities, sources of information, and educational services. Simply, one solution does not fit all the problems and no one can predict the needs that may rise in different learning contexts.

Current approaches to learning software patterns are based on individual use of different learning systems and tools. With this ‘fragmented’ approach it is very hard to provide support for context-aware learn-ing and offer personalized learning experience to students. In this paper, we propose a new approach to learning software patterns that integrates existing Learning Management Systems, domain specific tools for software modeling and relevant online repositories of software patterns into a complex learning framework that supports collaborative learning. This framework is based on the semantic web technologies.

In this paper, we propose the use of semantic documents as learning objects. The core part of our solution is a semantic document model that allows for unique identification of document content units (CUs) and their annotation with different types of metadata. On the top this model, we have developed the semantic document management system (SDMS), which enables efficient collaborative authoring of learning objects within a social network of content authors, by reusing document CUs based on the accumulated metadata.

The social semantic Web has recently emerged as a paradigm in which ontologies (aimed at defining, structuring and sharing information) and collaborative software (used for creating and sharing knowledge) have been merged together. Ontologies provide an effective means of capturing and integrating knowledge for feedback provisioning, while using collaborative activities can support pedagogical theories, such as social constructivism. Both technologies have developed separately in the e-learning domain; representing respectively a teacher-centered and a learner-centered approach for learning environments. In this paper we bridge the gap between these two approaches by leveraging the social semantic Web paradigm, and propose a collaborative semantic-rich learning environment in which folksonomies created from studentspsila collaborative tags contribute to ontology maintenance, and teacher-directed feedback.

In this paper, we present an extension to MS Office that enables users to search and retrieve document content units (e.g., paragraphs, images, tables, slides, etc.) from documents, which are stored on user’s indi-vidual desktops organized in a peer-to-peer fashion. We first introduce the Semantic Document Model (SDM) that turns MS Office documents (i.e., MS Word and MS PowerPoint) into Semantic Web resources, making document content to be accessible and query-able as RDF data. Then we describe the developed tools, which extend Office applications with support for ontology-based, distributed search of semantic documents stored in local RDF repositories over Se-mantic Web protocols.

This paper gives a brief overview of the International Workshop on Semantic Technologies in System Maintenance. It describes a number of semantic technologies (e.g., ontologies, text mining, and knowledge integration techniques) and identifies diverse tasks in software maintenance where the use of semantic technologies can be beneficial, such as traceability, system comprehension, software artifact analysis, and information integration.

In this paper, we show how spreadsheets, an end-user development paradigm proven to be highly productive and simple to learn and use, can be used for complex service compositions. We identify the requirements for spreadsheet-based service composition, and present our framework that implements these requirements. Our framework enables spreadsheets to send requests and retrieve results from various local and remote services. We show how our tools support different composition patterns, and how the style of declarative dependencies of spreadsheets can facilitate service composition. We also discuss novel issues identified by using the framework in several projects and education.

A business process usually includes multiple busi- ness partners that use systems with their business log- ics represented in different rule (or policy) languages. The integration of business processes is a major goal followed by business enterprises and involves the in- terchange of business rules and policies between part- ners. However, the variety of business rules employed by the partners presents a significant burden to the integration of collaborating business systems. In this paper, we propose a solution that enables translating rules and policies defined in different rule languages into a single general rule language (REWERSE I1 Rule Markup Language - R2ML) and processing them in a uniform manner. This provides a unified view of different partners' business rules in an integration process, while permitting the partners to continue to leverage their own business rules without changing their technologies. We show how the concepts of the KAoS policy language can be transformed to R2ML and then from R2ML to the other rule (or policy) lan- guages.

In our previous work, we developed the LOCO ontology framework which formalizes the notion of learning object context as a complex interplay of learning activities, learning objects, and learners. We now use that framework in conjunction with semantic annotation to generate different kinds of feedback (which we had identified by interviewing several Web educators) for educators to help them improve the learning process in Web-based settings. To test the feasibility of the proposed approach for feedback provision we developed a tool named LOCO-Analyst. Here we report on our experiences in developing LOCO-Analyst and using it to generate feedback out of the real data obtained from the iHelp Courses Learning Content Management System. Finally, we present evaluation results.

In this paper, we propose the use of Semantic Web technologies to bridge the gap between authoring systems and authors. The core part of our solution is the ontology-based framework that captures the information about the interaction between learning objects (LOs) and four main roles in the educational process (content author, instructional designer, teacher, and learner), and then according to this information proposes the most relevant learning content to the author. The central part of the framework is the Request- Recommendation ontology that formalizes the author's request along with a set of learning content proposals (recommendation), as a response to that request. Furthermore, we propose the use of a weighting scheme to calculate the weight of the content proposals and thus enable their ranking within the recommendation.

The paper presents Abstract Compound Content Model (ACCM), a generic content model which we have devel-oped aiming to facilitate interoperability, repurposing and integration of diverse platform specific content models. Based on this model we have developed the ACCM ontology in order to turn the ACCM's elements (i.e., content units and content aggregations) into resources that can be directly accessed and thus reused. The paper also presents our current work on the implementation of an ACCM-based content management system that enables efficient storage, indexing, search and retrieval of content units as they are defined in the ACCM ontology.

The paper presents an ontology-based framework for capturing learning context related information important for personalization of both learning objects (LOs) and learning designs (LDs). The central part of the framework is the LO Context ontology, that bridges a learning content ontology and a LD ontology. The LO context ontology is aimed at capturing information about the actual usage of a LO inside a LD, such as the learning activity the LO was used in, the pedagogical role assumed by the LO (e.g. exercise), the learner’s features (represented in the form of the learner model) and the like. Furthermore we suggest the architecture of an adaptive educational system that leverages the proposed approach to enable personalization and reuse of LOs and LDs.

In this paper we deal with knowledge representation in the area of learning design and adaptive learning. Specification of concrete instances is usually context-dependent and does not support reusability very well, thus we need to represent the knowledge that could help us in generating the instances dynamically.

This paper presents an ontology-based approach to automatic annotation of learning objects’ (LOs) content units that we tested in TANGRAM, an integrated learning environment for the domain of Intelligent Information Systems. The approach does not primarily focus on automatic annotation of entire LOs, as other relevant solutions do. Instead, it provides a solution for automatic metadata generation for LOs’ components (i.e., smaller, potentially reusable, content units). Here we mainly report on the content-mining algorithms and heuristics applied for determining values of certain metadata elements used to annotate content units. Specifically, the focus is on the following elements: title, description, unique identifier, subject (based on a domain ontology), and pedagogical role (based on an ontology of pedagogical roles). Additionally, as TANGRAM is grounded on an LO content structure ontology that drives the process of an LO decomposition into its constituent content units, each thus generated content unit is implicitly semantically annotated with its role/position in the LO’s structure. Employing such semantic annotations, TANGRAM allows assembling content units into new LOs personalized to the users’ goals, preferences, and learning styles. In order to provide the evaluation of the proposed solution, we describe our experiences with automatic annotation of slide presentations, one of the most common LO types.

In this paper we present PatternGuru an educational system for learning software patterns. Software patterns capture proven solutions of common problems in software development. Those solutions are general, but again they solve a problem in a particular context. PatternGuru is designed with the goal to be used for teaching software patterns within both undergraduate and graduate software engineering courses. The basic idea of PatternGuru is to provide learning of software patterns in collaborative manner and to present them as an integral part of software development. The tool is developed by extending ArgoUML, an open source project, so it can be used for both software engineering and education.

The paper proposes a framework for building learning object (LO) content using ontologies. In the previous work on using ontologies to describe LOs, researchers employed ontologies exclusively for describing LOs' metadata. Although such an approach is useful for searching for LOs in LO Repositories, it does not provide us with features to reuse components of LOs, nor to incorporate an explicit specification of domain semantics into LO content. We propose the use of two kinds of ontologies as a solution to this problem: content structure ontologies and domain ontologies.

This paper gives a proposal to enhance learning object (LO) content using ontologies and semantic Web languages. In the previous work on using ontologies to describe LOs, researchers have built ontologies for description of metadata. However, these ontologies do not improve an LO's content. We suggest creating LOs that have content marked up in accordance with domain ontologies. Accordingly, LOs can be used not only as learning materials, but they can also be used in real-world applications.

P3 is an application designed for teaching Petri nets within a course on architecture and organization of computers (AOC). Existing Petri net software implements different Petri net concepts, but does not give full support for learning their basic postulates. The idea of P3 is to enable learning Petri nets in a more obvious and quicker way in order to use them for hardware modeling. Therefore, P3 has a more suitable graphical user interface and implements model simulation with dynamic model modification, interactive model analysis, and copying of the results to the graph of the observed model.