Massive Open Online Courses (MOOCs) are a popular form of online education that often attracts a huge and heterogeneous group of learners with diverse interests and backgrounds. However, most MOOCs follow a one-size-fits-all approach, providing a fixed order of learning materials and expecting all learners to follow this recommended path. Thus, they neither motivate nor support their learners in adapting the courses to their individual preferences. In the work at hand, we tackle this issue by introducing and evaluating the concept of flexible learning paths in MOOCs. We, therefore, establish a network of dependencies between course content, omit intermediate deadlines, and thereby rethink the way learners interact with the course. By presenting learners with a non-linear course format, we encourage them to create their individual learning paths based on instructor-defined dependencies and their personal interests. Our evaluation of flexible learning paths within a programming MOOC shows that learners chose many different learning paths. Despite achieving similar results in individual tasks compared to learners using the traditional course structure, they engaged with less course content, resulting in a slight decrease in their overall performance. This may indicate a lack of self-regulatory learning skills, with learners struggling to organise their work without instructor-given deadlines. However, the flexible course format significantly increased the motivation of learners. By introducing and evaluating the concept of flexible learning paths in MOOCs, this work provides valuable insights into the individualisation of online education.

The Hasso Plattner Institute (HPI) has been successfully delivering courses on several MOOC (Massive Open Online Course) platforms for the last 10 years, offering courses on various topics in the context of Artificial Intelligence (AI), Machine Learning (ML), and Data Science. In recent years, Jupyter Notebooks have become one of the most widely used tools for data science applications, a platform for learning and practicing various programming languages. We want to integrate JupyterHub into our learning platform in order to provide students with hands-on experience in AI. We have conducted a survey with a series of research questions in order to understand the needs of instructors in their courses at different institutions. In this paper, we present a detailed analysis of our survey results and we discuss our future approach to using JupyterHub as an infrastructure to solve hands-on programming exercises on our platform. We propose the idea of creating a tool to automate server and environment creation for students to work on. This tool would give instructors a platform to operate from and allow them to customize their courses. Moreover, it would help them automate assignment submissions, grading, and provide feedback to their students.

In 2017, Geoffrey West published his book "Scale" in which he examined universal laws of scale in different contexts. Inspired by his keynote in 2021's [email protected] conference, we investigated the applicability of these laws in the context of Massive Open Online Courses and learners' behavior. We tested these laws on different learning platforms from academic, enterprise and social, and research contexts. In this paper, we examine course characteristics, such as course size, the completion rate, and the forum activity. We observed that the number of issued certificates scales almost identically on all examined platforms, while forum participation scales slightly different on each of the platforms. In the future, we will perform a deeper analysis on the forum behavior that exceeds a mere quantitative analysis.

Peer assessment has become a regular feature of many MOOC1platforms and also has potential for other contexts where learning and teaching are required to scale because of growing numbers of students. Where manual grading is not possible due to the large number of submissions and the tasks to be assessed are to complex or open-ended to be assessed by machines, peer assessment offers a valuable alternative. However, particularly in the context of MOOCS, courses featuring peer assessments often have lower completion rates. Furthermore, participants with negative expectations and opinions about this form of assessment are generally ‘louder’ in their communication with the teaching teams than their counterparts who respond more positively. We have, therefore, set out to establish a broader understanding how the participants perceive the appropriateness and effectiveness of peer assessed tasks, and the quality of the received reviews on the X1, X2, and X32MOOC platforms. For this purpose, we have conducted post-course surveys in a large number of courses that included peer assessments. Additionally, we analyzed the discussions in the forums of these courses as the post-course surveys often are biased due to the low proportion of unsuccessful participants that are still around at the end of a course.1Massive Open Online Course2The platforms’ names have been obscured for double-blind review

Massive Open Online Courses (MOOCs) have opened new educational possibilities for learners around the world. Most of the research and spotlight has been concentrated on a handful of global, English-language providers, but there are a growing number of regional providers of MOOCS in languages other than English. In this work, we have partnered with thirteen MOOC providers from around the world. We apply a multi-platform approach generating a joint and comparable analysis with data from millions of learners. This allows us to examine learning analytics trends at a macro level across various MOOC providers, with a goal of understanding which MOOC trends are globally universal and which of them are context-dependent. The analysis reports preliminary results on the differences and similarities of trends based on the country of origin, level of education, gender and age of their learners across global and regional MOOC providers. This study exemplifies the potential of macro learning analytics in MOOCs to understand the ecosystem and inform the whole community, while calling for more large scale studies in learning analytics through partnerships among researchers and institutions.

This workshop proposes specifically soliciting contributions and presentations from initiatives, programs, and platforms around the world. While many of these may already be presented at the full conference, we are also interested in more casual experience reports, case studies, and background presentations from individuals more closely acquainted with how learning at scale initiatives-including MOOCs, for-credit degree programs, informal learning environments, government initiatives, and so on-have unique needs and opportunities based on their local context. We refer to this as Global Learning @ Scale. For the purposes of this workshop, we take two views of Global Learning @ Scale.

Teamwork and graded team assignments in MOOCs are still largely under-researched. Nevertheless, the topic is enormously important as the ability to work and solve problems in teams is becoming increasingly common in modern work environments. The paper at hand discusses the reliability of a system to detect free-riders in peer assessed team tasks.

This paper explores the development of perceived learner skill confidence in a programming MOOC by applying and analyzing a new Skill Confidence Rating (SCR) survey. After cleaning datasets, we analyze a sample of n = 1689 for the first course module and n = 1147 for the second course module. Results show that on average, learners perceive their skills more confidently after taking a module. The initial confidence per module differs. We could not find a correlation between perceived learner confidence and learner performance in this course.

Computer science education in German schools is often less than optimal. It is only mandatory in a few of the federal states and there is a lack of qualified teachers. As a MOOC (Massive Open Online Course) provider with a German background, we developed the idea to implement a MOOC addressing pupils in secondary schools to fill this gap. The course targeted high school pupils and enabled them to learn the Python programming language. In 2014, we successfully conducted the first iteration of this MOOC with more than 7000 participants. However, the share of pupils in the course was not quite satisfactory. So we conducted several workshops with teachers to find out why they had not used the course to the extent that we had imagined. The paper at hand explores and discusses the steps we have taken in the following years as a result of these workshops.

This Research to Practice Full Paper presents the results of a survey among the participants of our MOOC platform about the benefits of what they have learned in our courses and the benefits of the certificates that they have earned for their daily life. We often hear about the high dropout rates in Massive Open Online Courses (MOOCs). On the other hand, there is a lot of movement towards micro-credentials, online master's degrees based on MOOCs, and formal MOOC degrees. However, what is the classic MOOC clientele actually doing with their certificates? What are the reasons why successful MOOC learners put a lot of time and effort in exams and exercises? Do employers accept these certificates in application portfolios? Do they allow their employees to participate in a MOOC during their working hours? Do they pay for the certificates? Are there any differences concerning gender because women are still in the minority in science, technology, engineering and mathematics (STEM) and their career paths often hit a plateau? These questions have been on our mind since we started offering courses on our MOOC platform in 2012. We have had anecdotal evidence, for example, that a participant was helped to get a new job with a certificate from one of our courses-yet this is ultimately just hearsay. Therefore, to find out what is really going on we conducted a survey among the 187,000 registered users of our platform.

The ability to work in teams is an important skill in today's work environments. In MOOCs, however, team work, team tasks, and graded team-based assignments play only a marginal role. To close this gap, we have been exploring ways to integrate graded team-based assignments in MOOCs. Some goals of our work are to determine simple criteria to match teams in a volatile environment and to enable a frictionless online collaboration for the participants within our MOOC platform. The high dropout rates in MOOCs pose particular challenges for team work in this context. By now, we have conducted 15 MOOCs containing graded team-based assignments in a variety of topics. The paper at hand presents a study that aims to establish a solid understanding of the participants in the team tasks. Furthermore, we attempt to determine which team compositions are particularly successful. Finally, we examine how several modifications to our platform's collaborative toolset have affected the dropout rates and performance of the teams.

This Research-to-Practice paper examines the practical application of various forms of collaborative learning in MOOCs. Since 2012, about 60 MOOCs in the wider context of Information Technology and Computer Science have been conducted on our self-developed MOOC platform. The platform is also used by several customers, who either run their own platform instances or use our white label platform. We, as well as some of our partners, have experimented with different approaches in collaborative learning in these courses. Based on the results of early experiments, surveys amongst our participants, and requests by our business partners we have integrated several options to offer forms of collaborative learning to the system. The results of our experiments are directly fed back to the platform development, allowing to fine tune existing and to add new tools where necessary. In the paper at hand, we discuss the benefits and disadvantages of decisions in the design of a MOOC with regard to the various forms of collaborative learning. While the focus of the paper at hand is on forms of large group collaboration, two types of small group collaboration on our platforms are briefly introduced.

This work presents insights about the long-term effects and retention rates of knowledge acquired within MOOCs. In 2015 and 2017, we conducted two introductory MOOCs on object-oriented programming in Java with each over 10,000 registered participants. In this paper, we analyze course scores, quiz results and self-stated skill levels of our participants. The aim of our analysis is to uncover factors influencing the retention of acquired knowledge, such as time passed or knowledge application, in order to improve long-term success. While we know that some participants learned the programming basics within our course, we lack information on whether this knowledge was applied and fortified after the course's end. To fill this knowledge gap, we conducted a survey in 2018 among all participants of our 2015 and 2017 programming MOOCs. The first part of the survey elicits responses on whether and how MOOC knowledge was applied and gives participants opportunity to voice individual feedback. The second part of the survey contains several questions of increasing difficulty and complexity regarding course content in order to learn about the consolidation of the acquired knowledge. We distinguish three programming knowledge areas in the survey: First, understanding of concepts, such as loops and boolean algebra. Second, syntax knowledge, such as specific keywords. Third, practical skills including debugging and coding. We further analyzed the long-term effects separately per participant skill group. While answer rates were low, the collected data shows a decrease of knowledge over time, relatively unaffected by skill level. Application of the acquired knowledge improves the memory retention rates of MOOC participants across all skill levels.

Teamwork and collaborative learning are considered superior to learning individually by many instructors and didactical theories. Particularly, in the context of e-learning and Massive Open Online Courses (MOOCs) we see great benefits but also great challenges for both, learners and instructors. We discuss our experience with six team based assignments on the openHPI and openSAP1 MOOC platforms.

A typical problem in MOOCs is the missing opportunity for course conductors to individually support students in overcoming their problems and misconceptions. This paper presents the results of automatically intervening on struggling students during programming exercises and offering peer feedback and tailored bonus exercises. To improve learning success, we do not want to abolish instructionally desired trial and error but reduce extensive struggle and demotivation. Therefore, we developed adaptive automatic just-in-time interventions to encourage students to ask for help if they require considerably more than average working time to solve an exercise. Additionally, we offered students bonus exercises tailored for their individual weaknesses. The approach was evaluated within a live course with over 5,000 active students via a survey and metrics gathered alongside. Results show that we can increase the call outs for help by up to 66% and lower the dwelling time until issuing action. Learnings from the experiments can further be used to pinpoint course material to be improved and tailor content to be audience specific.

Massive Open Online Courses (MOOCs) have left their mark on the face of education during the recent years. At the Hasso Plattner Institute (HPI) in Potsdam, Germany, we are actively developing a MOOC platform, which provides our research with a plethora of e-learning topics, such as learning analytics, automated assessment, peer assessment, team-work, online proctoring, and gamification. We run several instances of this platform. On openHPI, we provide our own courses from within the HPI context. Further instances are openSAP, openWHO, and mooc.HOUSE, which is the smallest of these platforms, targeting customers with a less extensive course portfolio. In 2013, we started to work on the gamification of our platform. By now, we have implemented about two thirds of the features that we initially have evaluated as useful for our purposes. About a year ago we activated the implemented gamification features on mooc.HOUSE. Before activating the features on openHPI as well, we examined, and re-evaluated our initial considerations based on the data we collected so far and the changes in other contexts of our platforms.

Teamwork is an an important topic in education. It fosters deep learning and allows educators to assign interesting tasks, which would be too complex to be solved by single participants due to the time restrictions defined by the context of a course.Furthermore, today's jobs require an increasing amount of team skills. On the other hand, teamwork comes with a variety of issues of its own. Particularly in large scale settings, such as MOOCs, teamwork is challenging. Courses often end with dysfunctional teams due to drop-outs or insufficient matching. The paper at hand presents a set of three tools that we have recently added to our system to enable teamwork in our courses. This toolset consists of the TeamBuilder, a tool to match successful teams based on a variable set of parameters, CollabSpaces, providing teams with a secluded area to communicate and collaborate within the course context, and a TeamPeerAssessment tool, which allows to provide teams with complex tasks and which allows assessment that suffiiently scales for the MOOC context. The presented tools are evaluated in terms of success rates of the created teams and workload reduction for the courses' teaching teams.

The paper at hand introduces CodeOcean, a web-based platform to provide practical programming exercises. CodeOcean is designed to be used in Massive Open Online Courses (MOOCs) to teach programming to beginners. Its concept and implementation are discussed with regard to tools provided to students and teachers, sandboxed and scalable code execution, scalable assessment, and interoperability. MOOCs bear a tremendous potential for teaching programming to a large and diverse audience. Learning to program, however, is a hands-on effort; watching videos and solving multiple choice tests will not be sufficient. A platform, such as CodeOcean, to work on practical programming exercises and to solve actual programming tasks is required. Due to the massiveness of the courses, teaching teams cannot check, give feedback, or assess the submissions of the participants manually. CodeOcean provides the participants with proper automated feedback in a timely manner and is able to assess the given programming tasks in an automated way.

In this paper we propose a method to evaluate the importance of lecture video segments in online courses. The video will be first segmented based on the slide transition. Then we evaluate the importance of each segment based on our analysis of the teacher's focus. This focus is mainly identified by exploring features in the slide and the speech. Since the whole analysis process is based on multimedia materials, it could be done before the official start of the course. By setting survey questions and collecting forum statistics in the MOOC "Web Technologies", the proposed method is evaluated. Both the general trend and the high accuracy of selected key segments (over 70%) prove the effectiveness of the proposed method.

In recent years, Massive Open Online Courses (MOOCs) have become a phenomenon offering the possibility to teach thousands of participants simultaneously. In the same time the platforms used to deliver these courses are still in their fledgling stages. While course content and didactics of those massive courses are the primary key factors for the success of courses, still a smart platform may increase or decrease the learners experience and his learning outcome. The paper at hand proposes the usage of an A/B testing framework that is able to be used within an micro-service architecture to validate hypotheses about how learners use the platform and to enable data-driven decisions about new features and settings. To evaluate this framework three new features (Onboarding Tour, Reminder Mails and a Pinboard Digest) have been identified based on a user survey. They have been implemented and introduced on two large MOOC platforms and their influence on the learners behavior have been measured. Finally this paper proposes a data driven decision workflow for the introduction of new features and settings on e-learning platforms.

This paper at hand describes the design and implementation of an analytics service to retrieve live usage data from students enrolled in a service-oriented MOOC platform for the purpose of learning analytics (LA) research. A real-time and extensible architecture for consolidating and processing data in versatile analytics stores is introduced.

Massive Open Online Courses (MOOCs) have revolutionized higher education by offering university-like courses for a large amount of learners via the Internet. The paper at hand takes a closer look on peer assessment as a tool for delivering individualized feedback and engaging assignments to MOOC participants. Benefits, such as scalability for MOOCs and higher order learning, and challenges, such as grading accuracy and rogue reviewers, are described. Common practices and the state-of-the-art to counteract challenges are highlighted. Based on this research, the paper at hand describes a peer assessment workflow and its implementation on the openHPI and openSAP MOOC platforms. This workflow combines the best practices of existing peer assessment tools and introduces some small but crucial improvements.

The paper at hand evaluates the Massive Open Online Course (MOOC) Spielend Programmieren Lernen (Playfully learning to program), an effort to scale the youth development program at the Hasso Plattner Institute (HPI) for a larger audience. The HPI has a strong tradition in attracting children and adolescents to take their first steps towards a career in IT at an early age. The Schülerakademie, the Schülerkolleg, the Schülerklub, and the support for the CoderDojo in Potsdam are some of the regular activities in this context to take youngsters by the hand and supply them with material and guidance in their mother tongue. With the emergence of MOOCs and the success of HPI's own MOOC Platform - openHPI - it was a natural step to develop a course to address an audience that is only marginally represented in openHPI's regular courses: school children and adolescents. A further novelty for openHPI in this course was the focus on teaching programming with a high percentage of obligatory hands-on tasks. Particularly for this course, a standalone tool allowing participants to write and evaluate code directly in the browser - without the need to install additional software - has been developed. We will compare this tool to a small selection of similar approaches on other platforms. As it will be shown, the course attracted a far more diverse audience than expected, and therefore, also needs to be seen in the context of spreading digital literacy amongst wider parts of society. In this context we also will discuss the significant differences in the usage of the forum between the course Spielend Programmieren Lernen and the course In-Memory Databases, a more traditional openHPI course.

The process of assessment grading does not end on publishing the grades and assessment results. Often, discussions about single grades arise in post-exam reviews. While in traditional teaching and smaller online classes this can be handled on a per-user based approach, in Massive Open Online Courses (MOOCs) with thousands of e-assessment submissions this approach cannot be followed without investing huge resources. Therefore, a new approach of handling re-gradings is needed. In this paper, we will discuss the experience on re-gradings based on six courses offered on openHPI. We will introduce our concept of automated re-grading following an approach of fairness and demonstrate the importance of transparent communication. This paper provides a blueprint for handling re-grading issues in large scale learning environments.

There is a great demand for hands-on training in engineering education. In the context of a Massive Open Online Course (MOOC), assessing these experiments manually by teaching assistants is not possible owed to the high number of participants and the resulting workload for the teaching team. Systems for machine-based assessment of coding tasks are existing, but not necessarily available publicly, or not prepared to handle the massive amount of users in a MOOC. Definitely, they are not available “ad hoc”, but require a certain amount of effort to be integrated in the MOOC platform or to be made available for the students in another way. Time and money to provide the required effort is not always available. This work presents a lightweight solution for the assessment of practical programming exercises, based on third party online coding tools. The solution was introduced as a part of openHPI's Web-Technologies course. The basic idea is to prepare a task in an available online tool, along with a piece of code that is able to evaluate the participant's solution. In case of success the participant is provided with a password, which in return serves as the answer for a fill-in-the-gap question in a standard quiz as provided by the openHPI MOOC platform, and thus allows for automatic online assessment based on practical coding exercises.