As information technology advanced, accuracy of technology-driven assessment is being improved. In order to assess learning performance and awareness, assessment of metacognition level with technology can be useful to understand learner’s learning comprehension and awareness. Metacognition is one of the most important elements for successful learning. However, current way to evaluate metacognition level focuses on psychological method such as questionnaire and interview. The recent growth of learning analytics research has demonstrated the relationships between metacognition, learning awareness, and learning behaviors. This study aims to investigate metacognitive learning behaviors using small grain data on eBook and learning analytics dashboard (LAD) over eight weeks in a university course. To do so, we determined high and low metacognitive learner groups using the Metacognitive Awareness Inventory and investigated the differences between the two groups in eBook and LAD. The findings suggest that four learning behaviors eBook and LAD were detected as metacognitive learning behaviors, and contribute to the improvement of technology-driven assessment.

Student-facing learning analytics dashboards (LADs) provide visualizations of course-related information to help students understand and personalize their educational practices. As such, they can be viewed as a meta-cognitive tool that enables awareness, self-reflection and sensemaking of academic performance. While student-facing LADs are becoming a standard feature in educational software, questions have been raised about students’ willingness to adopt LADs and their ability to interpret feedback provided by student-facing LADs. The extent to which student-facing LADs can broadly improve educational outcomes depends, in part, on students’ ability to readily incorporate LAD usage in their educational workflows.This study investigates the use of a student-facing LAD, My Learning Analytics (MyLA), over the span of one semester in a university introductory science course. MyLA draws data from the campus learning management system (Canvas) and displays three visualizations designed to provide students with actionable information. Adoption and use of MyLA was voluntary. As an exploratory study of MyLA’s use in an introductory science course, this work addresses three research questions: i) What are the characteristics of students that use MyLA?, ii) How do students make use of MyLA in their coursework?, and iii) What patterns of use are exhibited by more frequent MyLA users? The results indicate that given the opportunity to use a student-facing LAD, 33% of students made repeated use of the tool. Demographic data (e.g., gender, domestic/international student) did not predict MyLA usage but significant differences in mean cumulative GPA were found between non-MyLA users and MyLA users. Broad patterns of MyLA use were aligned with major assessments in the course (e.g., MyLA was used more often around exam dates) and the grade distribution view was the most commonly accessed. Among the most highly active MyLA users, two distinct profiles were identified: aware and sensemakers. Aware users made use of the dashboard on more than 12 distinct days across the course, primarily around exam dates, and stated that they accessed the dashboard to compare their performance with others. Sensemakers made frequent use of all three MyLA views multiple times over the semester to monitor their own progress, compare their grades to others, and check what materials other students had viewed.LADs such as MyLA allow students to leverage what they already know about course assessment in their interpretation of the data presented, easing adoption and deployment of a student-facing LAD in higher education. As MyLA does not require that students have any additional training to interpret the visualizations they provide, LADs can readily be employed by students in introductory computing and engineering courses to provide them with feedback to help them plan for, monitor, and evaluate their academic progress.

Research has emphasized that self-regulated learning (SRL) is critically important for learning. However, students have different capabilities of regulating their learning processes and individual needs. To help students improve their SRL capabilities, we need to identify students’ current behaviors. Specifically, we applied instructional design to create visible and meaningful markers of student learning at different points in time in LMS logs. We adopted knowledge engineering to develop a framework of proximal indicators representing SRL phases and evaluated them in a quasi-experiment in two different learning activities. A comparison of two sources of collected students’ SRL data, self-reported and trace data, revealed a relatively high agreement between our classifications (weighted kappa, κ = .74 and κ = .68). However, our indicators did not always discriminate adjacent SRL phases, particularly for enactment and adapting phases, compared with students’ real-time self-reported behaviors. Our behavioral indicators also were comparably successful at classifying SRL phases for different self-regulatory engagement levels. This study demonstrated how the triangulation of various sources of students’ self-regulatory data could help to unravel the complex nature of metacognitive processes.

The aim of student-facing dashboards is to support learning by providing students with actionable information and promoting self-regulated learning. We created a new dashboard design aligned with SRL theory, called MyLA, to better understand how students use a learning analytics tool. We conducted sequence analysis on students' interactions with three different visualizations in the dashboard, implemented in a LMS, for a large number of students (860) in ten courses representing different disciplines. To evaluate different students' experiences with the dashboard, we computed chi-squared tests of independence on dashboard users (52%) to find frequent patterns that discriminate students by their differences in academic achievement and self-regulated learning behaviors. The results revealed discriminating patterns in dashboard use among different levels of academic achievement and self-regulated learning, particularly for low achieving students and high self-regulated learners. Our findings highlight the importance of differences in students' experience with a student-facing dashboard, and emphasize that one size does not fit all in the design of learning analytics tools.

There has been limited research on how perceptions of socioeconomic status (SES) and opinion difference could influence peer feedback in Massive Open Online Courses (MOOCs). Using social comparison theory [12], we investigated the influence of ability and opinion-related factors on peer feedback text in a data science MOOC. Perceived SES of peers and the formality of written responses were used as the ability-related factor, while agreement between learners represented the opinion-related factor. We focused on understanding the behaviors of those learners who are most prevalent in MOOCs; those from high socioeconomic countries. Through two studies, we found a strong and repeated influence of agreement on affect and formality in feedback to peers. While a mediation effect of perceived SES was found, a significant effect of formality was not. This work contributes to an understanding of how social comparison theory can be operationalized in online peer writing environments.

In this study, we develop and test three measures for conceptualizing the potential impact of co-enrollment in different courses on students' changing risk for academic difficulty in a focal course. Two of these measures, concurrent enrollment in at least one difficult course and academic difficulty in the prior week in courses other than the focal course, significantly increase students' odds of academic difficulty in the focal course in our models. Our results have implications for the designs of Early Warning Systems and suggest that academic planners consider the relationship between course co-enrollment and students' academic success.

The scholarly investigation of discourse in teaching and learning is multi-disciplinary, theoretically rich, and highly technical. Researchers with backgrounds in education, cognitive psychology, computer science, and the social sciences apply a diverse set of techniques to understand how student discussions affect learning. Aided by big data coming from learning content management and massive open online course systems, these researchers have an unparalleled opportunity for insight into the teaching and learning process. In this paper we summarize some of the challenges and opportunities arising out of three workshops on Educational Discourse. These workshops convened both expert and emerging scholars to discuss the (i) ethical, (ii) technical, and (iii) infrastructure barriers to building a research community focused on computer-mediated educational discourse. Of particular note is that while computational infrastructure exists for storing and manipulating educational discourse, there is a need for a sociotechnical infrastructure upon which community members can come together to engage in joint work.

Recent research using learning analytics data to explore student performance over the course of a term suggests that a substantial percentage of students who are classified as academically struggling manage to recover. In this study, we report the result of a hazard analysis based on students' behavioral engagement with different digital instructional technologies over the course of a semester. We observe substantially different adoption and use behavior between students who did and did not experience academic difficulty in the course. Students who experienced moderate academic difficulty benefited the most from using tools that helped them plan their study behaviors. Students who experienced more severe academic difficulty benefited from tools that helped them prepare for exams. We observed that students adopted most tools and system features before they experienced academic difficulty, and students who adopted early were more likely to recover.

Learning Analytics courses and degree programs both on-and offline have begun to proliferate over the last three years. As a result of this growth in interest from students, university administrators, researchers and instructors we believe it is a good time to review how these educational efforts are impacting the field, how synergy between instructors might be developed to greater serve the field and what kinds of best practices could be developed.

In this paper we discuss the results of a study of students' academic performance in first year general education courses. Using data from 566 students who received intensive academic advising as part of their enrollment in the institution's pre-major/general education program, we investigate individual student, organizational, and disciplinary factors that might predict a students' potential classification in an Early Warning System as well as factors that predict improvement and decline in their academic performance. Disciplinary course type (based on Biglan's [7] typology) was significantly related to a student's likelihood to enter below average performance classifications. Students were the most likely to enter a classification in fields like the natural science, mathematics, and engineering in comparison to humanities courses. We attribute these disparities in academic performance to disciplinary norms around teaching and assessment. In particular, the timing of assessments played a major role in students' ability to exit a classification. Implications for the design of Early Warning analytics systems as well as academic course planning in higher education are offered.

In this paper we examine the issue of selection bias in quasi-experimental (non-randomly controlled) educational studies. We provide background about common sources of selection bias and the issues involved in evaluating the outcomes of quasi-experimental studies. We describe two methods, matched sampling and propensity score matching, that can be used to overcome this bias. Using these methods, we describe their application through one case study that leverages large educational datasets drawn from higher education institutional data warehouses. The contribution of this work is the recommendation of a methodology and case study that educational researchers can use to understand, measure, and reduce selection bias in real-world educational interventions.

As courses become bigger, move online, and are deployed to the general public at low cost (e.g. through Massive Open Online Courses, MOOCs), new methods of predicting student achievement are needed to support the learning process. This paper presents a novel method for converting educational log data into features suitable for building predictive models of student success. Unlike cognitive modelling or content analysis approaches, these models are built from interactions between learners and resources, an approach that requires no input from instructional or domain experts and can be applied across courses or learning environments.

Demographics factors have been used successfully as predictors of student success in traditional higher education systems, but their relationship to achievement in MOOC environments has been largely untested. In this work we explore the predictive power of user demographics compared to learner interaction trace data generated by students in two MOOCs. We show that demographic information offers minimal predictive power compared to activity models, even when compared to models created very early on in the course before substantial interaction data has accrued.

This paper reports findings from the implementation of a learning analytics-powered Early Warning System (EWS) by academic advisors who were novice users of data-driven learning analytics tools. The information collected from these users sheds new light on how student analytic data might be incorporated into the work practices of advisors working with university students. Our results indicate that advisors predominantly used the EWS during their meetings with students---despite it being designed as a tool to provide information to prepare for meetings and identify students who are struggling academically. This introduction of an unintended audience brings significant design implications to bear that are relevant for learning analytics innovations.

Massive Open Online Courses (MOOCs) are seen as an opportunity for individuals to gain access to education, develop new skills to prepare for high-paying jobs, and achieve upward mobility without incurring the increasingly high debt that comes with a university degree. Despite this perception, few studies have examined whether populations with the most to gain do leverage these resources. We analyzed student demographic information from course surveys and performance data of MOOC participation in a single course. We targeted students who stated that they were motivated to take the course because they "cannot afford to pursue a formal education," and compared them to the group of all other students. Our three key findings are that 1) a higher percentage of non-traditional enrolled students are in this population than the comparison population, 2) in an independent t-test, a statistically significant portion (28%) of this group has less than a 4-year college degree versus 15% of the comparison group, and 3) the completion rate between both groups are relatively equal.

Student Explorer is an early warning system designed to support academic advising that uses learning analytics to categorize students' ongoing academic performance and effort. Advisors use this tool to provide just-in-time assistance to students at risk of underperforming in their classes. Student Explorer is designed to eventually support targeted advising for thousands of undergraduate students.

Research in CSCW has demonstrated that people use technology in inventive ways, yet little work investigates the adoption and adaptation of collaborative technologies by unanticipated users. In this paper, we present a study investigating an unanticipated user group's appropriation of a leaning management system, CTools. This group of users, staff at a large research university, has adapted the system, which was designed to support student-content-faculty interactions at the University of Michigan. We present the User/Use Technology Appropriation Matrix (UTAM) as a way to frame our understanding of users and their system use. Based on findings from system log data and surveys, we show that staff use the system similarly to students and faculty, though they value the tools and work affordances differently in their varied work contexts. We discuss these findings, how UTAM can be used to frame these findings, and suggestions for future research.

The International Workshop on Analytics on Video-based Learning (WAVe2013) aims to connect research efforts on Video-based Learning with Learning Analytics to create visionary ideas and foster synergies between the two fields. The main objective of WAVe is to build a research community around the topical area of Analytics on video-based learning. In particular, WAVe aims to develop a critical discussion about the next generation of analytics employed on video learning tools, the form of these analytics and the way they can be analyzed in order to help us to better understand and improve the value of video-based learning. WAVe is based on the rationale that combining and analyzing learners' interactions with other available data obtained from learners, new avenues for research on video-based learning have emerged.

This paper describes an intra-institutional partnership between a research team and a technology service group that was established to facilitate the scaling up of a learning analytics intervention. Our discussion focuses on the benefits and challenges that arose from this partnership in order to provide useful information for similar partnerships developed to support scaling up learning analytics interventions.

This paper presents current findings from an ongoing design-based research project aimed at developing an early warning system (EWS) for academic mentors in an undergraduate engineering mentoring program. This paper details our progress in mining Learning Management System data and translating these data into an EWS for academic mentors. We focus on the role of mentors and advisors, and elaborate on their importance in learning analytics-based interventions developed for higher education.

Real-time, character-by-character transmission of messages in synchronous forms of text-based communications has seen a recent resurgence in CMC. We evaluated the impact of real-time text display on the usability of an instant messaging (IM) client. Participants were randomly assigned to dyads to participate in two discussion tasks using IM with both real-time text and enhanced message-by-message display (i.e., line-by-line display with additional cues to show when the remote party is typing). We found that real-time text helped users better coordinate turns and lead to less self-editing of messages, but had no overall influence on users' typing ability and provided minimal support for collaborative completion of sentences. Users who typed less or had less experience with IM tended to prefer real-time text. These findings have significance for several forms text-based CMC, including IM, chat, text telephony, and collaborative document editing.

This paper presents a qualitative study of attitudes towards participation and contribution in an online creative community. The setting of the work is an online community of practice focused on the use and development of a user-customizable music software package called Reaktor. Findings from the study highlight four emergent topics in the discourse related to user contributions to the community: contribution assessment, support for learning, perceptions of audience and tensions about commercialization. Our analysis of these topics frames discussion about the value and challenges of attending to amateur and professional users in online creative communities.

HCI and CSCW researchers have begun to call for greater and more explicit support of creative endeavors. Current theories of creativity suggest that it is an inherently collaborative activity, situated and highly contextualized. This work argues that a contextualized view of creativity calls in turn for assessment and technological support to be considered in situ.This poster presents a case study of the creative collaboration in a multimedia arts collective, with the goal of describing their current practices to inform appropriate information system design. We found that even a small and cohesive collaborative arts group contained a multitude of artistic practices and production tool choices, several distinct but interdependent work tracks and a variety of attitudes about the individual members' collaborative roles. Such heterogeneity, evidenced even within a self-selected and self-organized group, suggests challenges for future technological support of creative practices.

Heavy use of an online collaboration and learning environment (CLE) at a large research university led the graduate school to consider how a CLE might support dissertation committees. The project team conducted focus groups with 38 student, faculty, and administrative staff to determine system requirements. Results showed that users would benefit from a tool designed to facilitate the dissertation process, especially if social norms and work-benefit disparity issues were directly addressed. The development team designed and built a "dissertation navigator" in our CLE. 645 users have adopted Grad Tools, suggesting that some traditional groupware design challenges have been overcome.

In a field study conducted at a leading Fortune 100 company, we examined how having development teams reside in their own large room (an arrangement called radical collocation) affected system development. The collocated projects had significantly higher productivity and shorter schedules than both the industry benchmarks and the performance of past similar projects within the firm. The teams reported high satisfaction about their process, and both customers and project sponsors were similarly highly satisfied. The analysis of questionnaire, interview and observational data from these teams showed that being "at hand," i.e. both visible and available, helped them to coordinate their work better and learn from each other. Radical collocation seems to be one of the factors leading to high productivity in these teams.

Companies are experimenting with putting teams into warrooms, hoping for some productivity enhancement. We conducted a field study of six such teams, tracking their activity, attitudes, use of technology and productivity. Teams in these warrooms showed a doubling of productivity. Why? Among other things, teams had easy access to each other for both coordination of their work and for learning, and the work artifacts they posted on the walls remained visible to all. These results imply that if we are to truly support remote teams, we should provide constant awareness and easy transitions in and out of spontaneous meetings.

Current research on CSC W for remote groups focuses on one technology at a time: shared editing on the desktop, video conferencing, glancing at others' offices, email, etc.When a real group sets out to work remotely, however, they need to consider all aspects of work, synchronous, asynchronous, and the transitions to and tkom.This paper explores the planning, implementation, and use of a suite of groupware tools over the course of a year in a real group with remote members.We found that groupware atlkted people's commitments and the nature of the work distribution.