In educational assessment, rubrics are a key tool because they define clear criteria for evaluating learner responses and specify the evidence required. Yet, in research on automatic short-answer scoring, rubrics are seldom employed as explicit scoring references and, when they are, they are typically treated only as supplementary inputs. This study contrasts this by exploring the usage of rubrics as explicit scoring anchors in automatic short-answer scoring. It introduces a task definition for rubric-based automatic short-answer scoring, a version of short-answer scoring that uses scoring rubrics as scoring references. As an approach for implementing rubric-based short answer scoring, we introduce the idea of guided rubric answer alignment and provide two concrete architectures based on this, GRAASP and ToLeGRAA. Both are novel transformer-based architectures that use attention mechanisms to predict the alignment between student answers and rubric criteria. We compare them to multiple baselines using ALICE-LP, a novel German short-answer scoring dataset collected from formative assessments in authentic German school contexts, and the widely used ASAP-SAS dataset, collected at American schools. For both datasets, GRAASP and ToLeGRAA achieve highly competitive performance. By explicitly aligning learner responses with rubric criteria when assigning scores, these models demonstrate the feasibility of rubric-based short-answer scoring. This finding underscores the high potential of integrating rubric-driven scoring models into educational assessment.

When reading long and complex texts, students may disengage and miss out on relevant content. In order to prevent disengaged behavior or to counteract it by means of an intervention, it is ideally detected an early stage. In this paper, we present a method for early disengagement detection that relies only on the classification of scrolling data. The presented method transforms scrolling data into a time series representation, where each point of the series represents the vertical position of the viewport in the text document. This time series representation is then classified using time series classification algorithms. We evaluated the method on a dataset of 565 university students reading eight different texts. We compared the algorithm performance with different time series lengths, data sampling strategies, the texts that make up the training data, and classification algorithms. The method can classify disengagement early with up to 70% accuracy. However, we also observe differences in the performance depending on which of the texts are included in the training dataset. We discuss our results and propose several possible improvements to enhance the method.

In recent years, Learning Analytics (LA) has become a very heterogeneous research field due to the diversity in the data generated by the Learning Management Systems (LMS) as well as the researchers in a variety of disciplines, who analyze this data from a range of perspectives. In this paper, we present the evaluation of a LA tool that helps course designers, teachers, students and educational researchers to make informed decisions about the selection of learning activities and LA indicators for their course design or LA dashboard. The aim of this paper is to present Open Learning Analytics Indicator Repository (OpenLAIR) and provide a first evaluation with key stakeholders (N=41). Moreover, it presents the results of the prevalence of indicators that have been used over the past ten years in LA. Our results show that OpenLAIR can support course designers in designing LA-based learning activities and courses. Furthermore, we found a significant difference between the relevance and usage of LA indicators between educators and learners. The top rated LA indicators by researchers and educators were not perceived as equally important from students' perspectives.

Collaboration is one of the four important 21st-century skills. With the pervasive use of sensors, interest on co-located collaboration (CC) has increased lately. Most related literature used the audio modality to detect indicators of collaboration (such as total speaking time and turn taking). CC takes place in physical spaces where group members share their social (i.e., non-verbal audio indicators like speaking time, gestures) and epistemic space (i.e., verbal audio indicators like the content of the conversation). Past literature has mostly focused on the social space to detect the quality of collaboration. In this study, we focus on both social and epistemic space with an emphasis on the epistemic space to understand different evolving collaboration patterns and collaborative convergence and quantify collaboration quality. We conduct field trials by collecting audio recordings in 14 different sessions in a university setting while the university staff and students collaborate over playing a board game to design a learning activity. This collaboration task consists of different phases with each collaborating member having been assigned a pre-fixed role. We analyze the collected group speech data to do role-based profiling and visualize it with the help of a dashboard.

Learning analytics dashboards (LADs) are designed as feedback tools for learners, but until recently, learners rarely have had a say in how LADs are designed and what information they receive through LADs. To overcome this shortcoming, we have developed a customisable LAD for Coursera MOOCs on which learners can set goals and choose indicators to monitor. Following a mixed-methods approach, we analyse 401 learners’ indicator selection behaviour in order to understand the decisions they make on the LAD and whether learner goals and self-regulated learning skills influence these decisions. We found that learners overwhelmingly chose indicators about completed activities. Goals are not associated with indicator selection behaviour, while help-seeking skills predict learners’ choice of monitoring their engagement in discussions and time management skills predict learners’ interest in procrastination indicators. The findings have implications for our understanding of learners’ use of LADs and their design.

In order to obtain a holistic perspective on learning, technical infrastructure at an institutional level can be advantageous for Learning Analytics (LA). If personal data is collected and processed in such infrastructure, legal requirements are of crucial importance. Recent studies have examined various aspects of LA infrastructure, such as ethical trade-offs and stakeholder needs. However, strategies for designing technical LA infrastructure at the institutional level and strategies for dealing with data protection regulations have been lacking so far, although they are crucial for the adoption of LA in Europe. The purpose of this paper is to examine three research questions: (RQ1) which strategies are currently used to design LA infrastructure; (RQ2) how data protection and privacy affect the design of LA infrastructure; and (RQ3) how could technical measures support the adoption of LA. These research questions were investigated by conducting eleven interviews with LA infrastructure developers representing eight different higher education institutions and ten different infrastructures. According to RQ1, the paper first examines the domain specificity of LA infrastructure and the four design strategies used. The paper also examines, according to RQ2, the interviewees’ awareness of data protection, the conflict with users’ consent, and the data subject rights. Finally, the paper presents, in line with RQ3, the results of the strategies in dealing with trust, stakeholder expectations, and engagement. Researchers and infrastructure developers can use and adopt these findings to improve their strategies for developing technical LA infrastructure with regard to data protection, privacy, and trust.

We report the results of a study that investigated the usage of a consumer-grade smartwatch for associative media learning. In a study with 29 participants, we investigated the effects of learning the Morse code alphabet purely visual versus learning with an associated vibration pattern. Our results show a trend that participants who received the vibration stimulus in addition to the visual stimulus learned faster and retained more knowledge than the group who learned using only the visualization, as measured by two post-tests one and three days after the learning took place.

Interest in learning analytics (LA) has grown rapidly among higher education institutions (HEIs). However, the maturity levels of HEIs in terms of being ‘student data-informed’ are only at early stages. There often are barriers that prevent data from being used systematically and effectively. To assist higher education institutions to become more mature users and custodians of digital data collected from students during their online learning activities, the SHEILA framework, a policy development framework that supports systematic, sustainable and responsible adoption of LA at an institutional level, was recently built. This paper presents a mix-method study using a group concept mapping (GCM) approach that was conducted with LA experts to explore essential features of LA policy in HEI in contribution the development of the framework. The study identified six clusters of features that an LA policy should include, provided ratings based on ease of implementation and importance for each of the six themes, and offered suggestions to HEIs how they can proceed with the development of LA policies.

This paper introduces the Visual Inspection Tool (VIT) which supports researchers in the annotation of multimodal data as well as the processing and exploitation for learning purposes. While most of the existing Multimodal Learning Analytics (MMLA) solutions are tailor-made for specific learning tasks and sensors, the VIT addresses the data annotation for different types of learning tasks that can be captured with a customisable set of sensors in a flexible way. The VIT supports MMLA researchers in 1) triangulating multimodal data with video recordings; 2) segmenting the multimodal data into time-intervals and adding annotations to the time-intervals; 3) downloading the annotated dataset and using it for multimodal data analysis. The VIT is a crucial component that was so far missing in the available tools for MMLA research. By filling this gap we also identified an integrated workflow that characterises current MMLA research. We call this workflow the Multimodal Learning Analytics Pipeline, a toolkit for orchestration, the use and application of various MMLA tools.

Collaboration is an important 21st century skill; it can take place in a remote or co-located setting. Co-located collaboration (CC) is a very complex process which involves subtle human interactions that can be described with multimodal indicators (MI) like gaze, speech and social skills. In this paper, we first give an overview of related work that has identified indicators during CC. Then, we look into the state-of-the-art studies on feedback during CC which also make use of MI. Finally, we describe a Wizard of Oz (WOz) study where we design a privacy-preserving research prototype with the aim to facilitate real-time collaboration in-the-wild during three co-located group PhD meetings (of 3–7 members). Here, human observers stationed in another room act as a substitute for sensors to track different speech-based cues (like speaking time and turn taking); this drives a real-time visualization dashboard on a public shared display. With this research prototype, we want to pave way for design-based research to track other multimodal indicators of CC by extending this prototype design using both humans and sensors.

Learning analytics can bridge the gap between learning sciences and data analytics, leveraging the expertise of both fields in exploring the vast amount of data generated in online learning environments. A typical learning analytics intervention is the learning dashboard, a visualisation tool built with the purpose of empowering teachers and learners to make informed decisions about the learning process. Related work has investigated learning dashboards, yet none have explored the theoretical foundation that should inform the design and evaluation of such interventions. In this systematic literature review, we analyse the extent to which theories and models from learning sciences have been integrated into the development of learning dashboards aimed at learners. Our analysis revealed that very few dashboard evaluations take into account the educational concepts that were used as a theoretical foundation for their design. Furthermore, we report findings suggesting that comparison with peers, a common reference frame for contextualising information on learning analytics dashboards, was not perceived positively by all learners. We summarise the insights gathered through our literature review in a set of recommendations for the design and evaluation of learning analytics dashboards for learners.

Learning Analytics is a melting pot for a multitude of research fields and origin of many developments about learning and its environment. There is a serious hype over the concepts of learning analytics, however, concrete solutions and applications are comparably scarce. Of course, data rich environments, such as MOOCs, come with statistical analytics dashboards, although the educational value is often limited. Practical solutions for scenarios in data-lean environments or for small-scale organizations are rarely adopted. The LA4S project is dedicated to gather practical solutions, provide a tool box for practitioners, and publish a cook book with concrete learning analytics recipes for everyone.

Learning Pulse explores whether using a machine learning approach on multimodal data such as heart rate, step count, weather condition and learning activity can be used to predict learning performance in self-regulated learning settings. An experiment was carried out lasting eight weeks involving PhD students as participants, each of them wearing a Fitbit HR wristband and having their application on their computer recorded during their learning and working activities throughout the day. A software infrastructure for collecting multimodal learning experiences was implemented. As part of this infrastructure a Data Processing Application was developed to pre-process, analyse and generate predictions to provide feedback to the users about their learning performance. Data from different sources were stored using the xAPI standard into a cloud-based Learning Record Store. The participants of the experiment were asked to rate their learning experience through an Activity Rating Tool indicating their perceived level of productivity, stress, challenge and abilities. These self-reported performance indicators were used as markers to train a Linear Mixed Effect Model to generate learner-specific predictions of the learning performance. We discuss the advantages and the limitations of the used approach, highlighting further development points.

The collaborative learning processes of students in online learning environments can be supported by providing learning analytics-based visualisations that foster awareness and reflection about an individual's as well as the team's behaviour and their learning and collaboration processes. For this empirical study we implemented an activity widget into the online learning environment of a live five-months Master course and investigated the predictive power of the widget indicators towards the students' grades and compared the results to those from an exploratory study with data collected in previous runs of the same course where the widget had not been in use. Together with information gathered from a quantitative as well as a qualitative evaluation of the activity widget during the course, the findings of this current study show that there are indeed predictive relations between the widget indicators and the grades, especially those regarding responsiveness, and indicate that some of the observed differences in the last run could be attributed to the implemented activity widget.

This short paper presents the experiences of several Dutch projects in their application of the xAPI standard and different design patterns including the deployment of Learning Record Stores. In this paper we share insights and argue for the formation of an international Special Interest Group on interoperability issues to contribute to the Open Analytics Framework as envisioned by SoLAR and enacted by the Apereo Learning Analytics Initiative. Therefore, we provide an overview of the advantages and disadvantages of implementing the current xAPI standard by presenting projects that applied xAPI in very different ways followed by the lessons learned.

Almost all studies on course recommenders in online platforms target closed online platforms that belong to a University or other provider. Recently, a demand has developed that targets open platforms. Such platforms lack rich user profiles with content metadata. Instead they log user interactions. We report on how user interactions and activities tracked in open online learning platforms may generate recommendations. We use data from the OpenU open online learning platform in use by the Open University of the Netherlands to investigate the application of several state-of-the-art recommender algorithms, including a graph-based recommender approach. It appears that user-based and memory-based methods perform better than model-based and factorization methods. Particularly, the graph-based recommender system outperforms the classical approaches on prediction accuracy of recommendations in terms of recall.

We present the collected experiences since 2012 of the Dutch Special Interest Group (SIG) for Learning Analytics in the application of the xAPI standard. We have been experimenting and exchanging best practices around the application of xAPI in various contexts. The practices include different design patterns centered around Learning Record Stores. We present three projects that apply xAPI in very different ways and publish a consistent set of xAPI recipes.

Collaborative learning is considered a critical 21st century skill. Much is known about its contribution to learning, but still investigating a process of collaboration remains a challenge. This paper approaches the investigation on collaborative learning from a psychophysiological perspective. An experiment was set up to explore whether biosensors can play a role in analysing collaborative learning. On the one hand, we identified five physiological coupling indices (PCIs) found in the literature: 1) Signal Matching (SM), 2) Instantaneous Derivative Matching (IDM), 3) Directional Agreement (DA), 4) Pearson's correlation coefficient (PCC) and the 5) Fisher's z-transform (FZT) of the PCC. On the other hand, three collaborative learning measurements were used: 1) collaborative will (CW), 2) collaborative learning product (CLP) and 3) dual learning gain (DLG). Regression analyses showed that out of the five PCIs, IDM related the most to CW and was the best predictor of the CLP. Meanwhile, DA predicted DLG the best. These results play a role in determining informative collaboration measures for designing a learning analytics, biofeedback dashboard.

The widespread adoption of Learning Analytics (LA) and Educational Data Mining (EDM) has somewhat stagnated recently, and in some prominent cases even been reversed following concerns by governments, stakeholders and civil rights groups. In this ongoing discussion, fears and realities are often indistinguishably mixed up, leading to an atmosphere of uncertainty among potential beneficiaries of Learning Analytics, as well as hesitations among institutional managers who aim to innovate their institution’s learning support by implementing data and analytics with a view on improving student success. In this paper, we try to get to the heart of the matter, by analysing the most common views and the propositions made by the LA community to solve them. We conclude the paper with an eight-point checklist named DELICATE that can be applied by teachers, researchers, policy makers and institutional managers to facilitate a trusted implementation of Learning Analytics.

Issues related to Ethics and Privacy have become a major stumbling block in application of Learning Analytics technologies on a large scale. Recently, the learning analytics community at large has more actively addressed the EP4LA issues, and we are now starting to see learning analytics solutions that are designed not only as an afterthought, but also with these issues in mind. The 2nd EP4LA@LAK16 workshop will bring the discussion on ethics and privacy for learning analytics to a the next level, helping to build an agenda for organizational and technical design of LA solutions, addressing the different processes of a learning analytics workflow.

The LAK Data Challenge 2015 continues the research efforts of the previous data competitions in 2013 and 2014 by stimulating research on the evolving fields Learning Analytics (LA) and Educational Data Mining (EDM). Building on a series of activities of the LinkedUp project, the challenge aims to generate new insights and analysis on the LA & EDM disciplines and is supported through the LAK Dataset - a unique corpus of LA & EDM literature, exposed in structured and machine-readable formats.

The large-scale production, collection, aggregation, and processing of information from various learning platforms and online environments have led to ethical and privacy concerns regarding potential harm to individuals and society. In the past, these types of concern have impacted on areas as diverse as computer science, legal studies and surveillance studies. Within a European consortium that brings together the EU project LACE, the SURF SIG Learning Analytics, the Apereo Foundation and the EATEL SIG dataTEL, we aim to understand the issues with greater clarity, and to find ways of overcoming the issues and research challenges related to ethical and privacy aspects of learning analytics practice. This interactive workshop aims to raise awareness of major ethics and privacy issues. It will also be used to develop practical solutions to advance the application of learning analytics technologies.

Since the emergence of learning analytics in North America, researchers and practitioners have worked to develop an international community. The organization of events such as SoLAR Flares and LASI Locals, as well as the move of LAK in 2013 from North America to Europe, has supported this aim. There are now thriving learning analytics groups in North American, Europe and Australia, with smaller pockets of activity emerging on other continents. Nevertheless, much of the work carried out outside these forums, or published in languages other than English, is still inaccessible to most people in the community. This panel, organized by Europe's Learning Analytics Community Exchange (LACE) project, brings together researchers from five European countries to examine the field from European perspectives. In doing so, it will identify the benefits and challenges associated with sharing and developing practice across national boundaries.

This paper presents results from the continuous process of developing an evaluation framework of quality indicators for learning analytics (LA). Building on a previous study, a group concept mapping approach that uses multidimensional scaling and hierarchical clustering, the study presented here applies the framework to a collection of LA tools in order to evaluate the framework. Using the quantitative and qualitative results of this study, the first version of the framework was revisited so as to allow work towards an improved version of the evaluation framework of quality indicators for LA.

This paper reports about experiences and lessons learned from a recently conducted pilot study about sustainability education with mobile inquiry-based learning in a secondary school in the Netherlands. In the pilot study learners were involved in a mobile location-based game that was conducted in reserved time-slots over 5 weeks. Mobile devices of the school and of learners have been combined with a smart-energy meter network to support learning activities covering the full inquiry cycle. Analysis of data collected shows that the implementation was beneficial for learners with a low prior knowledge level and that future designs should take into account gender aspects in the design phase.

The LAK Data Challenge 2014 continues the research efforts of the second edition by stimulating research on the evolving fields Learning Analytics (LA) and Educational Data Mining (EDM). Building on a series of activities of the LinkedUp project, the challenge aims to generate new insights and analysis on the LA & EDM disciplines and is supported through the LAK Dataset - a unique corpus of LA & EDM literature, exposed in structured and machine-readable formats.

The article reports the findings of a Group Concept Mapping study that was conducted within the framework of the Learning Analytics Summer Institute (LASI) in the Netherlands. Learning Analytics are expected to be beneficial for students and teacher empowerment, personalization, research on learning design, and feedback for performance. The study depicted some management and economics issues and identified some possible treats. No differences were found between novices and experts on how important and feasible are changes in education triggered by Learning Analytics.

Incorporating social trust in Matrix Factorization (MF) methods demonstrably improves accuracy of rating prediction. Such approaches mainly use the trust scores explicitly expressed by users. However, it is often challenging to have users provide explicit trust scores of each other. There exist quite a few works, which propose Trust Metrics (TM) to compute and predict trust scores between users based on their interactions. In this paper, we first evaluate several TMs to find out which one can best predict trust scores compared to the actual trust scores explicitly expressed by users. And, second, we propose to incorporate these trust scores inferred from the candidate TMs into social matrix factorization (MF). We investigate if incorporating the implicit trust scores in MF can make rating prediction as accurate as the MF on explicit trust scores. The reported results support the idea of employing implicit trust into MF whenever explicit trust is not available, since the performance of both models is similar.

The main objective of the 1st International Workshop on Learning Analytics and Linked Data (#LALD2012) is to connect the research efforts on Linked Data and Learning Analytics in order to create visionary ideas and foster synergies between the two young research fields. Therefore, the workshop will collect, explore, and present datasets, technologies and applications for Technology Enhanced Learning (TEL) to discuss Learning Analytics approaches that make use of educational data or Linked Data sources. During the workshop, an overview of available educational datasets and related initiatives will be given. The participants will have the opportunity to present their own research with respect to educational datasets, technologies and applications and discuss major challenges to collect, reuse, and share these datasets.

While there is currently much buzz about the new field of learning analytics [19] and the potential it holds for benefiting teaching and learning, the impression one currently gets is that there is also much uncertainty and hesitation, even extending to scepticism. A clear common understanding and vision for the domain has not yet formed among the educator and research community. To investigate this situation, we distributed a stakeholder survey in September 2011 to an international audience from different sectors of education. The findings provide some further insights into the current level of understanding and expectations toward learning analytics among stakeholders. The survey was scaffolded by a conceptual framework on learning analytics that was developed based on a recent literature review. It divides the domain of learning analytics into six critical dimensions. The preliminary survey among 156 educational practitioners and researchers mostly from the higher education sector reveals substantial uncertainties in learning analytics.

The Recommender System Challenge 2012 invited participants to work on two tracks with real-world datasets and to submit their contributions that would be related to specific problem contexts. First of all, it asked participants to develop new algorithms and to compare them to other algorithms in given settings; in addition, it asked participants to explore with new recommendation methods, services, as well as added-value services related to recommendation.

In the world of recommender systems, it is a common practice to use public available datasets from different application environments (e.g. MovieLens, Book-Crossing, or Each-Movie) in order to evaluate recommendation algorithms. These datasets are used as benchmarks to develop new recommendation algorithms and to compare them to other algorithms in given settings. In this paper, we explore datasets that capture learner interactions with tools and resources. We use the datasets to evaluate and compare the performance of different recommendation algorithms for learning. We present an experimental comparison of the accuracy of several collaborative filtering algorithms applied to these TEL datasets and elaborate on implicit relevance data, such as downloads and tags, that can be used to improve the performance of recommendation algorithms.

This workshop presents the current status related to the design, development and evaluation of recommender systems in educational settings. It emphasizes the importance of recommender systems for Technology Enhanced Learning (TEL) to support learners with personalized learning resources and suitable peer learners to improve their learning process. Moreover, it proposes a dataTEL challenge to obtain data sets from TEL applications that can be used to benchmark algorithms specifically for the TEL context.

This paper offers an extended abstract of a PhD project that focuses on supporting learners in finding most suitable learning activities in informal learning environments. For this purpose we aim to develop a personal recommender system, which will recommend most suitable learning activities to learners regarding their personal needs and preferences.