Student ability to succeed in collaborative problem solving (CPS) is increasingly important. However, identifying aspects of CPS that individuals can act on to improve it remains a challenge. This study explores listening behaviour as a lever for effective CPS, given that listening is at once cognitive and social, and that individuals can enact it. Despite its importance in communication, listening is rarely systematically examined in CPS. To address this gap, we propose a framework for identifying listening behaviours in the text of student exchanges, and apply it to analyse patterns of listening behaviours of 34 K-12 students aged 11-14, working in nine groups on CPS activities. By combining content analysis, k-means cluster analysis, and correlation-based coupling network analysis, we identify ten distinct patterns of listening behaviours and their coupling over time, across groups with different success outcomes. We found that listening patterns varied by performance, under the assumption of moderate temporal dependence among interaction segments. Higher performing groups engaged in more counterarguments and constructive listening. Groups with lower CPS successes exhibited two ineffective patterns: questioning with instrumental listening, and counterarguments and questioning without encouraging listening. These findings pose questions about the relationship between listening and learning processes and have implications for multimodal research in learning analytics.

Abstract Over the last few decades, machine learning (ML) applications have grown exponentially, yielding several benefits to society. However, these benefits are tempered with concerns of discriminatory behaviours exhibited by ML models. In this regard, fairness in machine learning has emerged as a priority research area. Consequently, several fairness metrics and algorithms have been developed to mitigate against discriminatory behaviours that ML models may possess. Yet still, very little attention has been paid to the problem of naturally occurring changes in data patterns (aka data distributional drift), and its impact on fairness algorithms and metrics. In this work, we study this problem comprehensively by analyzing 4 fairness-unaware baseline algorithms and 7 fairness-aware algorithms, carefully curated to cover the breadth of its typology, across 5 datasets including public and proprietary data, and evaluated them using 3 predictive performance and 10 fairness metrics. In doing so, we show that (1) data distributional drift is not a trivial occurrence, and in several cases can lead to serious deterioration of fairness in so-called fair models; (2) contrary to some existing literature, the size and direction of data distributional drift is not correlated to the resulting size and direction of unfairness; and (3) choice of, and training of fairness algorithms is impacted by the effect of data distributional drift which is largely ignored in the literature. Emanating from our findings, we synthesize several policy implications of data distributional drift on fairness algorithms that can be very relevant to stakeholders and practitioners.

Mounting evidence indicates that modern citation practices contribute to inequalities in who receives citations. In response to this evidence, our paper investigates citation practices in learning analytics (LA). We analyse citations in papers published over ten years at the Learning Analytics and Knowledge conference (LAK). Our analysis examines the gender composition of authored and cited papers in LA, estimating various factors that explain why one paper cites another, and if the citation rates differ across different author teams. Results indicate an overall increase in the number of women authors at LAK, while the ratio of men to women remains stable. Citation patterns in LAK are influenced by the seniority of authors, paper age, topic, and team size. We found that LAK papers with women as the last author are under-cited, but papers where the first author is a woman and the last author is a man are over-cited. Author teams with different gender composition also vary in who they over- and under-cite. Upon presenting the empirical results, the paper reflects on the role of mindful citation practices and reviews existing measures proposed to promote diversity in citations.

Complex problem solving (CPS) is a critical skill with far-reaching implications for personal success and professional development. While CPS research has made extensive progress, additional investigation is needed to explore CPS processes beyond online contexts and performance outcomes. This study, conducted with Year 9 students aged between thirteen and fourteen, focuses on collaborative CPS. It utilises audio and video recordings to capture group communications during a CPS classroom activity. To analyse these interactions, we introduce a novel CPS framework as a dynamic, cognitive and social process involving interrelated main skills, sub-skills, and indicators. Through sequential pattern mining, we identify recurring subskill patterns that reflect CPS processes in an educational environment. Our research underscores the importance of employing diverse patterns before plan execution, particularly building shared knowledge, planning, and negotiation. We uncover patterns related to groups going off-task and highlight the significance of effective communication and maintaining focus for keeping groups on track. Furthermore, we indicate patterns following the detection of emergent issues, recognising the value of cultivating clarity and adaptability among team members. Our CPS framework, combined with our research results, offers practical implications for teaching, learning, and assessment approaches in educational, professional and industry sectors.

Machine Learning (ML) techniques have been increasingly adopted to support various activities in education, including being applied in important contexts such as college admission and scholarship allocation. In addition to being accurate, the application of these techniques has to be fair, i.e., displaying no discrimination towards any group of stakeholders in education (mainly students and instructors) based on their protective attributes (e.g., gender and age). The past few years have witnessed an explosion of attention given to the predictive bias of ML techniques in education. Though certain endeavors have been made to detect and alleviate predictive bias in learning analytics, it is still hard for newcomers to penetrate. To address this, we systematically reviewed existing studies on predictive bias in education, and a total of 49 peer-reviewed empirical papers published after 2010 were included in this study. In particular, these papers were reviewed and summarized from the following three perspectives: (i) protective attributes, (ii) fairness measures and their applications in various educational tasks, and (iii) strategies for enhancing predictive fairness. These findings were summarized into recommendations to guide future endeavors in this strand of research, e.g., collecting and sharing more quality data containing protective attributes, developing fairness-enhancing approaches which do not require the explicit use of protective attributes, validating the effectiveness of fairness-enhancing on students and instructors in real-world settings.

Industry supervisors play a pivotal role in ongoing learner support and guidance within a work-integrated learning context. Effective provisional feedback from industry supervisors in work-integrated learning environments is essential for increasing a team’s metacognitive awareness and ability to evaluate their performance. However, research that examines the usefulness and type of feedback from industry supervisors for teams remains limited. In this study, we investigate the quality of provisional feedback by comparing the teams’ helpfulness rating of the feedback from two types of industry supervisors (i.e., clients and mentors), based on the feedback type (task, process, regulatory and self-level oriented) using learning analytics. The results show that teams rated the perceived helpfulness scores of clients and mentors as very useful, with mentors providing slightly more helpful feedback. We also found that mentors provide more co-occurrences of feedback classifications than clients. The overall results show that teams perceive mentor feedback as more helpful than clients and that the mentor targets feedback that is more beneficial to the teams learning than the clients. Our findings can aid in developing guidelines that aim to validate and improve existing or new feedback quality frameworks by leveraging backward evaluation data.

In recent years, predictive models have been increasingly used by education practitioners and stakeholders to leverage actionable insights to support student success. Usually, model selection (i.e., the decision of which predictive model to use) is largely based on the predictive performance of the models. Nevertheless, it has become important to consider fairness as an integral part of the criteria for model selection. Might a model be unfair towards certain demographic groups? Might it systematically perform poorly for certain demographic groups? Indeed, prior studies affirm this. Which model out of the lot should we choose then? Additionally, prior studies suggest demographic group imbalance in the training dataset to be a source of such unfairness. If so, would the fairness of the predictive models improve if the demographic group distribution in the training dataset becomes balanced? This study seeks to answer these questions. Firstly, we analyze the fairness of 4 of the commonly used state-of-the-art models to predict course success for 3 IT courses in a large public Australian university. Specifically, we investigate if the models serve different demographic groups equally. Secondly, to address the identified unfairnes---supposedly caused by the demographic group imbalance---we train the models on 3 types ofbalanced data and investigate again if the unfairness was mitigated. We found that none of the predictive models wasconsistently fair in all 3 courses. This suggests that model selection decision should be carefully made by both the researchers and stakeholders as the per the requirement of the domain of application. Furthermore, we found that balancing demographic groups (and class labels) is not enough---albeit can be an initial step---to ensure fairness of predictive models in education. An implication of this is that sometimes, the source of unfairness may not be immediately apparent. Therefore, "blindly" attributing the unfairness to demographic group imbalance may cause the unfairness to persist even when the data becomes balanced. We hope that our findings can guide practitioners and relevant stakeholders in making well-informed decisions.

This research aimed to explore the relationship between different indicators of the depth and quality of participation in computer-mediated learning environments. By using network analyses and statistical tests, we discovered significant associations between the cognitive presence phases of the Community of Inquiry framework and speech acts, and examined the impact of two different instructional interventions on these associations. We found that there are strong associations between some speech acts and cognitive presence phases. In addition, the study revealed that the association between speech acts and cognitive presence is moderated by external facilitation, but not affected by user role assignment. The results suggest that speech acts can plausibly be used to provide feedback in relation to cognitive presence and can potentially be used to increase the generalizability of cognitive presence classification.

Collaboration is considered as one of the main drivers of learning and it has been broadly studied across numerous contexts, including Massive Open Online Courses (MOOCs). The research on MOOCs has risen exponentially during the last years and there have been a number of works focused on studying collaboration. However, these previous studies have been restricted to the analysis of collaboration based on the forum and social interactions, without taking into account other possibilities such as the synchronicity in the interactions with the platform. Therefore, in this work we performed a case study with the goal of implementing a data-driven approach to detect and characterize collaboration in MOOCs. We applied an algorithm to detect synchronicity links based on their submission times to quizzes as an indicator of collaboration, and applied it to data from two large Coursera MOOCs. We found three different profiles of user accounts, that were grouped in couples and larger communities exhibiting different types of associations between user accounts. The characterization of these user accounts suggested that some of them might represent genuine online learning collaborative associations, but that in other cases dishonest behaviors such as free-riding or multiple account cheating might be present. These findings call for additional research on the study of the kind of collaborations that can emerge in online settings.

Models and processes developed in learning analytics research are increasing in sophistication and predictive power. However, the ability to translate analytic findings to practice remains problematic. This study aims to address this issue by establishing a model of learner behaviour that is both predictive of student course performance, and easily interpreted by instructors. To achieve this aim, we analysed fine grained trace data (from 3 offerings of an undergraduate online course, N=1068) to establish a comprehensive set of behaviour indicators aligned with the course design. The identified behaviour patterns, which we refer to as observed study strategies, proved to be associated with the student course performance. By examining the observed strategies of high and low performers throughout the course, we identified prototypical pathways associated with course success and failure. The proposed model and approach offers valuable insights for the provision of process-oriented feedback early in the course, and thus can aid learners in developing their capacity to succeed online.

Learning Analytics (LA) studies the learning process in order to optimize learning opportunities for students. Although LA has quickly risen to prominence, there remain questions regarding the impact LA has made to date. To evaluate the extent that LA has impacted our understanding of learning and produced insights that have been translated to mainstream practice or contributed to theory, we reviewed the research published in 2011-2018 LAK conferences and Journal of Learning Analytics. The reviewed studies were coded according to five dimensions: study focus, data types, purpose, institutional setting, and scale of research and implementation. The coding and subsequent epistemic network analysis indicates that while LA research has developed in the areas of focus and sophistication of analyses, the impact on practice, theory and frameworks have been limited. We hypothesize that this finding is due to a continuing predominance of small-scale techno-centric exploratory studies that to date have not fully accounted for the multi-disciplinarity that comprises education. For the field to reach its potential in understanding and optimizing learning and learning environments, there must be a purposeful shift to move from exploratory models to more holistic and integrative systems-level research. This necessitates greater effort applied to understanding the research cycles that emerge when multiple knowledge domains coalesce into new fields of research.

Student engagement is often considered an overarching construct in educational research and practice. Though frequently employed in the learning analytics literature, engagement has been subjected to a variety of interpretations and there is little consensus regarding the very definition of the construct. This raises grave concerns with regards to construct validity: namely, do these varied metrics measure the same thing? To address such concerns, this paper proposes, quantifies, and validates a model of engagement which is both grounded in the theoretical literature and described by common metrics drawn from the field of learning analytics. To identify a latent variable structure in our data we used exploratory factor analysis and validated the derived model on a separate sub-sample of our data using confirmatory factor analysis. To analyze the associations between our latent variables and student outcomes, a structural equation model was fitted, and the validity of this model across different course settings was assessed using MIMIC modeling. Across different domains, the broad consistency of our model with the theoretical literature suggest a mechanism that may be used to inform both interventions and course design.

Learning Analytics Dashboards (LAD) are becoming an increasingly popular way to provide students with personalised feedback. Despite the number of LADs being developed, significant research gaps exist around the student perspective, especially how students make sense of graphics provided in LADs, and how they intend to act on the feedback provided therein. This study employed a randomized-controlled trial to examine students' sense-making of LADs showing four different frames of reference, and to what extent the impact of LADs was mediated by baseline self-regulation. Using a mix of quantitative and qualitative data analysis, the results revealed rather distinct patterns in students' sense-making across the four LADs. These patterns involved the intersection of visual salience and planned learning actions. However, collectively, across all four LADs a consistent theme emerged around students planned learning actions. This theme was classified as time and study environment management. A key finding of the study is that the use of LADs as a primary feedback process should be personalized and include training and support to aid student sensemaking.

Research on learner behaviors and course completion within Massive Open Online Courses (MOOCs) has been mostly confined to single courses, making the findings difficult to generalize across different data sets and to assess which contexts and types of courses these findings apply to. This paper reports on the development of the MOOC Replication Framework (MORF), a framework that facilitates the replication of previously published findings across multiple data sets and the seamless integration of new findings as new research is conducted or new hypotheses are generated. In the proof of concept presented here, we use MORF to attempt to replicate 15 previously published findings across 29 iterations of 17 MOOCs. The findings indicate that 12 of the 15 findings replicated significantly across the data sets, and that two findings replicated significantly in the opposite direction. MORF enables larger-scale analysis of MOOC research questions than previously feasible, and enables researchers around the world to conduct analyses on huge multi-MOOC data sets without having to negotiate access to data.

Reflective writing has been widely recognized as one of the most effective activities for fostering students' reflective and critical thinking. The analysis of students' reflective writings has been the focus of many research studies. However, to date this has been typically a very labor-intensive manual process involving content analysis of student writings. With recent advancements in the field of learning analytics, there have been several attempts to use text analytics to examine student reflective writings. This paper presents the results of a study examining the use of theoretically-sound linguistic indicators of different psychological processes for the development of an analytics system for assessment of reflective writing. More precisely, we developed a random-forest classification system using linguistic indicators provided by the LIWC and Coh-Metrix tools. We also examined what particular indicators are representative of the different types of student reflective writings.

In 2011, the phenomenon of MOOCs had swept the world of education and put online education in the focus of the public discourse around the world. Although researchers were excited with the vast amounts of MOOC data being collected, the benefits of this data did not stand to the expectations due to several challenges. The analyses of MOOC data are very time-consuming and labor-intensive, and require and require a highly advanced set of technical skills, often not available to the education researchers. Because of this MOOC data analyses are rarely done before the courses end, limiting the potential of data to impact the student learning outcomes and experience.

In this poster, we present the results of the study which examined the relationship between student differences in their use of the available technology and their perceived levels of cognitive presence within the MOOC context. The cognitive presence is a construct used to measure the level of practical inquiry in the Communities of Inquiry model. Our results revealed the existence of three clusters based on student technology use. The clusters significantly differed in terms of their levels of cognitive presence, most notably they differed on the levels of problem resolution.

There is an emerging trend in higher education for the adoption of massive open online courses (MOOCs). However, despite this interest in learning at scale, there has been limited work investigating how MOOC participants have changed over time. In this study, we explore the temporal changes in MOOC learners' language and discourse characteristics. In particular, we demonstrate that there is a clear trend within a course for language in discussion forums to be of both more on-topic and reflective of deep learning in subsequent offerings of a course. We measure this in two ways, and demonstrate this trend through several repeated analyses of different courses in different domains. While not all courses show an increase beyond statistical significance, the majority do, providing evidence that MOOC learner populations are changing as the educational phenomena matures.

When designing learning analytics tools for use by learners we have an opportunity to provide tools that consider a particular learner's situation and the learner herself. To afford actual impact on learning, such tools have to be informed by theories of education. Particularly, educational research shows that individual differences play a significant role in explaining students' learning process. However, limited empirical research in learning analytics has investigated the role of theoretical constructs, such as motivational factors, that are underlying the observed differences between individuals. In this work, we conducted a field experiment to examine the effect of three designed learning analytics visualizations on students' participation in online discussions in authentic course settings. Using hierarchical linear mixed models, our results revealed that effects of visualizations on the quantity and quality of messages posted by students with differences in achievement goal orientations could either be positive or negative. Our findings highlight the methodological importance of considering individual differences and pose important implications for future design and research of learning analytics visualizations.

As the field of learning analytics continues to mature, there is a corresponding evolution and sophistication of the associated analytical methods and techniques. In this regard social network analysis (SNA) has emerged as one of the cornerstones of learning analytics methodologies. However, despite the noted importance of social networks for facilitating the learning process, it remains unclear how and to what extent such network measures are associated with specific learning outcomes. Motivated by Simmel's theory of social interactions and building on the argument that social centrality does not always imply benefits, this study aimed to further contribute to the understanding of the association between students' social centrality and their academic performance. The study reveals that learning analytics research drawing on SNA should incorporate both - descriptive and statistical methods to provide a more comprehensive and holistic understanding of a students' network position. In so doing researchers can undertake more nuanced and contextually salient inferences about learning in network settings. Specifically, we show how differences in the factors framing students' interactions within two instances of a MOOC affect the association between the three social network centrality measures (i.e., degree, closeness, and betweenness) and the final course outcome.

In this paper, we present the results of an exploratory study that examined the problem of automating content analysis of student online discussion transcripts. We looked at the problem of coding discussion transcripts for the levels of cognitive presence, one of the three main constructs in the Community of Inquiry (CoI) model of distance education. Using Coh-Metrix and LIWC features, together with a set of custom features developed to capture discussion context, we developed a random forest classification system that achieved 70.3% classification accuracy and 0.63 Cohen's kappa, which is significantly higher than values reported in the previous studies. Besides improvement in classification accuracy, the developed system is also less sensitive to overfitting as it uses only 205 classification features, which is around 100 times less features than in similar systems based on bag-of-words features. We also provide an overview of the classification features most indicative of the different phases of cognitive presence that gives an additional insights into the nature of cognitive presence learning cycle. Overall, our results show great potential of the proposed approach, with an added benefit of providing further characterization of the cognitive presence coding scheme.

Massive Open Online Courses represent a fertile ground for examining student behavior. However, due to their openness MOOC attract a diverse body of students, for the most part, unknown to the course instructors. However, a certain number of students enroll in the same course multiple times, and there are records of their previous learning activities which might provide some useful information to course organizers before the start of the course. In this study, we examined how student behavior changes between subsequent course offerings. We identified profiles of returning students and also interesting changes in their behavior between two enrollments to the same course. Results and their implications are further discussed.

Connections established between learners via interactions are seen as fundamental for connectivist pedagogy. Connections can also be viewed as learning outcomes, i.e. learners' social capital accumulated through distributed learning environments. We applied linear mixed effects modeling to investigate whether the social capital accumulation interpreted through learners' centrality to course interaction networks, is influenced by the language learners use to express and communicate in two connectivist MOOCs. Interactions were distributed across the three social media, namely Twitter, blog and Facebook. Results showed that learners in a cMOOC connect easier with the individuals who use a more informal, narrative style, but still maintain a deeper cohesive structure to their communication.

Creating meaning from a wide variety of available information and being able to choose what to learn are highly relevant skills for learning in a connectivist setting. In this work, various approaches have been utilized to gain insights into learning processes occurring within a network of learners and understand the factors that shape learners' interests and the topics to which learners devote a significant attention. This study combines different methods to develop a scalable analytic approach for a comprehensive analysis of learners' discourse in a connectivist massive open online course (cMOOC). By linking techniques for semantic annotation and graph analysis with a qualitative analysis of learner-generated discourse, we examined how social media platforms (blogs, Twitter, and Facebook) and course recommendations influence content creation and topics discussed within a cMOOC. Our findings indicate that learners tend to focus on several prominent topics that emerge very quickly in the course. They maintain that focus, with some exceptions, throughout the course, regardless of readings suggested by the instructor. Moreover, the topics discussed across different social media differ, which can likely be attributed to the affordances of different media. Finally, our results indicate a relatively low level of cohesion in the topics discussed which might be an indicator of a diversity of the conceptual coverage discussed by the course participants.

All forms of learning take time. There is a large body of research suggesting that the amount of time spent on learning can improve the quality of learning, as represented by academic performance. The wide-spread adoption of learning technologies such as learning management systems (LMSs), has resulted in large amounts of data about student learning being readily accessible to educational researchers. One common use of this data is to measure time that students have spent on different learning tasks (i.e., time-on-task). Given that LMS systems typically only capture times when students executed various actions, time-on-task measures are estimated based on the recorded trace data. LMS trace data has been extensively used in many studies in the field of learning analytics, yet the problem of time-on-task estimation is rarely described in detail and the consequences that it entails are not fully examined.

This paper provides an evaluation of the current state of the field of learning analytics through analysis of articles and citations occurring in the LAK conferences and identified special issue journals. The emerging field of learning analytics is at the intersection of numerous academic disciplines, and therefore draws on a diversity of methodologies, theories and underpinning scientific assumptions. Through citation analysis and structured mapping we aimed to identify the emergence of trends and disciplinary hierarchies that are influencing the development of the field to date. The results suggest that there is some fragmentation in the major disciplines (computer science and education) regarding conference and journal representation. The analyses also indicate that the commonly cited papers are of a more conceptual nature than empirical research reflecting the need for authors to define the learning analytics space. An evaluation of the current state of learning analytics provides numerous benefits for the development of the field, such as a guide for under-represented areas of research and to identify the disciplines that may require more strategic and targeted support and funding opportunities.

One of the most important prerequisites for achieving the Semantic Web vision is semantic annotation of data/resources. Semantic annotation enriches unstructured and/or semistructured content with a context that is further linked to the structured domain-specific knowledge. In particular, ontologybased semantic annotators enable the selection of a specific ontology to annotate content. This paper presents results of an empirical study of recent ontology-based annotators, namely Stanbol, KIM, and SDArch. Specifically, we evaluated the robustness of these annotators with respect to specific features of ontology concepts such as the length of concepts? labels and their linguistic categories (e.g., prepositions and conjunctions). Our results show that although significantly correlated according to most of the conducted evaluations, tools still exhibit their unique features that could be a topic of new research.