International audience

Abstract In MOOCs, identifying urgent comments on discussion forums is an ongoing challenge. Whilst urgent comments require immediate reactions from instructors, to improve interaction with their learners, and potentially reducing drop-out rates—the task is difficult, as truly urgent comments are rare. From a data analytics perspective, this represents a highly unbalanced (sparse) dataset . Here, we aim to automate the urgent comments identification process, based on fine-grained learner modelling —to be used for automatic recommendations to instructors. To showcase and compare these models, we apply them to the first gold standard dataset for U rgent i N structor I n TE rvention (UNITE) , which we created by labelling FutureLearn MOOC data. We implement both benchmark shallow classifiers and deep learning. Importantly, we not only compare, for the first time for the unbalanced problem, several data balancing techniques , comprising text augmentation, text augmentation with undersampling, and undersampling, but also propose several new pipelines for combining different augmenters for text augmentation . Results show that models with undersampling can predict most urgent cases; and 3X augmentation + undersampling usually attains the best performance. We additionally validate the best models via a generic benchmark dataset (Stanford). As a case study, we showcase how the naïve Bayes with count vector can adaptively support instructors in answering learner questions/comments, potentially saving time or increasing efficiency in supporting learners. Finally, we show that the errors from the classifier mirrors the disagreements between annotators. Thus, our proposed algorithms perform at least as well as a ‘super-diligent’ human instructor (with the time to consider all comments).

Abstract Massive Online Open Course (MOOC) platforms are considered a distinctive way to deliver a modern educational experience, open to a worldwide public. However, student engagement in MOOCs is a less explored area, although it is known that MOOCs suffer from one of the highest dropout rates within learning environments in general, and in e-learning in particular. A special challenge in this area is finding early, measurable indicators of engagement. This paper tackles this issue with a unique blend of data analytics and NLP and machine learning techniques together with a solid foundation in psychological theories. Importantly, we show for the first time how Self-Determination Theory (SDT) can be mapped onto concrete features extracted from tracking student behaviour on MOOCs. We map the dimensions of Autonomy, Relatedness and Competence, leading to methods to characterise engaged and disengaged MOOC student behaviours, and exploring what triggers and promotes MOOC students’ interest and engagement. The paper further contributes by building the Engage Taxonomy, the first taxonomy of MOOC engagement tracking parameters, mapped over 4 engagement theories: SDT, Drive, ET, Process of Engagement. Moreover, we define and analyse students’ engagement tracking, with a larger than usual body of content (6 MOOC courses from two different universities with 26 runs spanning between 2013 and 2018) and students (initially around 218.235). Importantly, the paper also serves as the first large-scale evaluation of the SDT theory itself, providing a blueprint for large-scale theory evaluation. It also provides for the first-time metrics for measurable engagement in MOOCs, including specific measures for Autonomy, Relatedness and Competence; it evaluates these based on existing (and expanded) measures of success in MOOCs: Completion rate, Correct Answer ratio and Reply ratio. In addition, to further illustrate the use of the proposed SDT metrics, this study is the first to use SDT constructs extracted from the first week, to predict active and non-active students in the following week.

User eXperience (UX) evaluations play an essential role in the software development process. As the results from such evaluations can drive future releases, it is necessary to identify which factors can substantially change users’ judgments about their experience to have more precise results and understand UX better. This article investigates how interaction sequencing, previous experience, and the number of problems could affect overall satisfaction and the two main UX dimensions: pragmatic and hedonic. We employed three different evaluation methods to evaluate a chatbot-based mobile shopping application. The results revealed that participants with previous experience with similar apps tended to give lower ratings. We also found that as inspectors identify more problems, they tend to rate the pragmatic dimension lower. Finally, we did not identify a significant influence of interaction sequencing on UX evaluation. We discuss the reasons for these results, the implications for practitioners and researchers, and research opportunities.

The difficulty in reading and interpreting the text has been a common problem in several types of intellectual disability, such as autism and dyslexia. Moreover, it triggers numerous damages to students throughout their academic life. However, this work presents an approach to the inclusive learning of reading, using beauty technology, a playful activities box, and an emotional assessment tool. To verify the approach’s applicability, we tarried out a learning workshop with six students with some literacy difficulty in elementary school. The results showed that the playful experience of nails with the activity box and the tangible evaluation of use could make students feel good when practicing a challenging discipline.

Stakeholder engagement is a key aspect for the successful implementation of Learning Analytics (LA) in Higher Education Institutions (HEIs). Studies in Europe and Latin America (LATAM) indicate that, overall, instructors and students have positive views on LA adoption, but there are differences between their ideal expectations and what they consider realistic in the context of their institutions. So far, very little has been found about stakeholders’ views on LA in Brazilian higher education. By replicating the survey conducted in other countries, in seven Brazilian HEIs, we found convergences both with Europe and LATAM, reinforcing the need for local diagnosis and indicating the risk of assuming a ”LATAM identity”. Our findings contribute to building a corpus of knowledge on stakeholders expectations with a contextualised comprehension of the gaps between ideal and predicted scenarios, which can inform institutional policies for LA implementation in Brazil.

Students face several difficulties in introductory programming courses (CS1), often leading to high dropout rates, student demotivation, and lack of interest. The literature has indicated that the adequate use of gamification might improve learning in several domains, including CS1. However, the understanding of which (and how) factors influence gamification’s success, especially for CS1 education, is lacking. Thus, there is a clear need to shed light on pre-determinants of gamification’s impact. To tackle this gap, we investigate how user and contextual factors influence gamification’s effect on CS1 students through a quasi-experimental retrospective study ( \( N = 399 \) ), based on a between-subject design (conditions: gamified or non-gamified) in terms of final grade (academic achievement) and the number of programming assignments completed in an educational system (i.e., how much they practiced). Then, we evaluate whether and how user and contextual characteristics (e.g., age, gender, major, programming experience, working situation, internet access, and computer access/sharing) moderate that effect. Our findings indicate that gamification amplified to some extent the impact of practicing. Overall, students practicing in the gamified version presented higher academic achievement than those practicing the same amount in the non-gamified version. Intriguingly, those in the gamified version that practiced much more extensively than the average showed lower academic achievements than those who practiced comparable amounts in the non-gamified version. Furthermore, our results reveal gender as the only statistically significant moderator of gamification’s effect: in our data, it was positive for females but non-significant for males. These findings suggest which (and how) personal and contextual factors moderate gamification’s effects, indicate the need to further understand and examine context’s role, and show that gamification must be cautiously designed to prevent students from playing instead of learning.

As programming skills are increasingly required world-wide and across disciplines, many students use online platforms that provide automatic feedback through a Programming Online Judge (POJ) mechanism. POJs are very popular e-learning tools, boasting large collections of programming problems. Despite their many benefits, students often struggle when solving problems not compatible with their prior knowledge. One important cause of this is that usually statements of problems are not classified according to programming topics (paradigms, data structures, etc.) and, hence, students waste time and effort in trying to solve exercises that are not tailored to their level and needs. Thus, to support students, we propose a new, "front-heavy" pipeline method to predict topics of POJ problems, using Bidirectional Encoder Representations from Transformers (BERT) for contextual text augmentation over the problem statements and further allowing for (lighter-weight) classical machine learning for classification. Our model outperformed all current state-of-the art, with an F1-score of 86% using stratified 10 fold cross-validation in a classically challenging multi-classification problem with seven categories. As a proof of concept, we conducted an experiment to show how our predictive model can be used as a human-AI hybrid complement for POJ, where learners would use AI-based recommendations to find the most appropriate problems.

Gamification has been widely employed in educational contexts to improve students' engagement and enhance the learning process. In this sense, the present work investigates the impact of gamification on students' behavior and performance through learning paths, i.e., sequences of learning objects followed by students while interacting with a virtual learning environment. To achieve this, we captured, analyzed and visually represented learning paths from 139 students enrolled in an online course. As result, we observed that students using gamification interacted more with the course elements, had longer paths and better grades. The main contribution of this work is to present learning paths as a tool to investigate students' interaction in a virtual learning environment with gamified elements and to analyze these elements' impact on students' behavior and performance.

Many researchers have started extracting student behaviour by cleaning data collected from web environments and using it as features in machine learning (ML) models. Using log data collected from an online judge, we have compiled a set of successful features correlated with the student grade and applying them on a database representing 486 CS1 students. We used this set of features in ML pipelines which were optimised, featuring a combination of an automated approach with an evolutionary algorithm and hyperparameter-tuning with random search. As a result, we achieved an accuracy of 75.55%, using data from only the first two weeks to predict the student final grades. We show how our pipeline outperforms state-of-the-art work on similar scenarios.

Taking into account that education supported by technology is a basic need for the citizen of a world in which the demand for computation and the access to information grow exponentially, this paper highlights the creation of cognitive computing systems that help people to acquire knowledge and to learn effectively within digital education environments. With this goal in mind, this paper summarizes the outcomes of a Systematic Mapping of Literature where the main aim was to identify what pedagogical approaches, methods, techniques, tools, and education activities have been used to recommend learning objects in the context of cognitive computing systems. We analyzed 348 papers from Scopus and Engineering Village digital libraries from which we selected 19 papers for data extraction. In order to increase the confidence of the proposed systematic review we calculated the Kappa Coefficient obtaining that the agreement was substantial (79.47%). From the data extracted we did several analysis. Concerning to pedagogical theories, 47.37% of papers presents a humanist approach and 26.32% a cognitivist approach, which shows one coherence with proposal of cognitive computing based on emotional and language processing. Most papers use natural-language, image or audio processing to detect emotion, attention and interactions to generate a user profile and, thus, to perform educational resource recommendation. Despite almost half papers do not indicate exactly the educational resources recommended, around 36% are textual materials and 45% are related to personalized exercises or interactive/games activities. Furthermore, we identified the spread use of analytical learning and cognitive computing for pedagogical activities recommendation. This paper ends up proposing new approaches and methods to educational recommendation for improving the students performance from interaction data in educational environments.

Manaus, the capital of Brazilian Amazon state demands a rapid increase in the number of highly qualified professionals, mostly for its Industrial Pole. Hardware and software development for the consumer electronics and telecommunication sectors are the most challenging areas, due to the amount of open positions contrasting with the quality of local educational Institutions. To address this issue a local higher education organization (Institute of Computing of the Federal University of Amazonas) started, in partnership with Samsung da Amazonia Ltd. (the local subsidiary of the Korean conglomerate), a large scale program based on the so called multiple vortices, and comprising of set of involving activities, including training in classroom, and enrolling students in initiation or graduate projects. After three year of running program, the authors proposed a survey to verify the results achieved so far, more specifically on those two mentioned activities. The survey is designed to answer two questions: A) How mature is the participant in understanding his career; B) How is the participant's perception of the program efficiency in meeting his expectations. The survey is composed by 41 yes/no questions covering five different topics: 1) Perception about career and marked demands; 2) Information about the program purpose and objectives; 3) Changes in the career plan and expectations of the future; 4) Perceived relevance of the program; 5) Strength and weaknesses of the running program. Anti-redundant questions (i.e. two answers leads the same conclusion when one is yes and the other is no) were used to reduce bias and to assign a confidence level to the respondent, afterwards used to weight the student's contribution to the final statistics. The survey was applied to current and former program participants. The paper will detail the results, and the corrective actions suggested by the students' answers. Furthermore, the survey itself, the design and application methodology, which was based on a pilot group of students, will be part of the results presented in this paper.

The lack of professionals is considered a major risk to the Industrial Pole installed in Manaus, Brazil. Specially, STEM education is intricately difficult and if the student is not well involved, supported and motivated, the chances of success are very low, which can be verified by the 25% to 30% dropout rate per semester in our institution. Given this scenario, a qualification program denominated “multiple vortexes of know-how” was conceived to address three aspects: i) reduce student dropout; ii) enlarge the reachable community; iii) offer different levels of knowledge. Vortex is in essence an action, working interconnected and coordinated to other vortexes. Four different actions were prepared: classroom disciplines; a talent development program; holding short-term events, and enrolling in intensive training. This present work describes the vortex that enrolled the major number of students: the classroom disciplines. A catalog of optional undergraduate and graduate disciplines was offered to the students. The best students received a prize based on their performance. In summary, we had around 650 students qualified with 84 scholarship prizes granted. The paper will present as a contribution to the program, the catalog of disciplines and recommendations on how to proceed with the activities.

Usability and UX (User eXperience) are some of the most important factors for evaluating the quality of mobile applications.They focus on how easy to use an application is and the emotions that such use evokes.However, these aspects are often evaluated separately in industry through different evaluation techniques.Although it is possible to identify more usability and UX problems by employing different UX and usability evaluation methods, this distributed approach may not be cost effective and may not allow to thoroughly explore the identified issues.In order to support the identification of both UX and usability problems in a single evaluation, we have proposed Userbility, an UX and usability inspection technique that allows evaluating these aspects in mobile applications.This paper presents an empirical study over the second version of Userbility to verify its feasibility.In this study, we compared Userbility with the UX and Usability Guidelines Approach (UUGA) that helps the evaluation of usability and UX separately in mobile applications.According to the quantitative results, considering efficiency, UUGA was better than the Userbility technique.However, the qualitative results suggest that Userbility pointed more improvement suggestions, which could be useful for redesigning the evaluated application.

Brazil is experiencing economic growth, with a strong demand for highly qualified professionals. In the capital of the Brazilian state of Amazonas, Manaus, the situation is critical. The lack of professionals is considered a major risk, and to prepare students is a real challenge as there are few undergraduate courses, and the typical dropout rate is very high. STEM education is intricately difficult and if the student is not well involved, supported and motivated, the chances of success are very low. To address this scenario, a qualification program denominated "multiple vortexes of know-how" was conceived to address three aspects: i) reduce student dropout; ii) enlarge the reachable community; iii) offer different levels of knowledge. Vortex is in essence an action, working interconnected and coordinated to other vortexes. Four different actions were prepared: classroom disciplines; a talent development program; holding short-term events, and enrolling in intensive training. This paper describes the first edition of the program, which was accomplished last year at the Institute of Computing at the Federal University of Amazonas. Up until now, the results are quite positive: over 600 students were involved and motivated with these new possibilities. Moreover, the goal of this paper is also to present the lessons learned so far.

Offering upper-level courses for the population that lives in remote areas, like the Amazon, is a huge challenge due to the difficulties of access and the low development level. To address this challenge, the Center for Distance Education (CDE) in Federal University of Amazonas, implemented an educational project that provides distance courses in graduate and undergraduate degree in these places using a Learning Management System (LMS). However, evasion, failure and dropout rates in the disciplines are still worrying. In order to face this problem, MobiMonitor, a mobile application integrated with multi-agent system was developed. This app allows instrumenting course mediators with tools for monitoring students' behavior and performance in a LMS, especially monitoring student participation in specific academic activities (assignments and forums) and provides support in advance to identify students who need a direct pedagogical intervention in order to avoid possible course evasion, failure or dropout. MobiMonitor enables mediators to check these data on the fly, classifying students' participation into four levels (excellent, good, low and very low). The mediators access this report by monitoring through mobile application and send alert messages to students according to their attendance status. Experiments conducted in two poles show the feasibility of the proposal.

The development of systems which adapt to the needs of student learning is a complex task. Learning involves several factors, including effective communication in a specific context. In this paper, we propose a conceptual framework to allow the integration of adaptive hypermedia in learning management systems from a cognitive-semantic perspective.

The Amazon region is characterized by its low population density, with one large city, the capital Manaus, and the remainder of its population distributed in small and less economically developed towns. Most of these towns suffer from huge geographic isolation, as these are scattered in the forest and their access are only through rivers. With all these difficulties, taking education to this population consists of a real and daily challenge. To provide an opportunity for students of this region to enter into an upper-level course, one of the solutions devised by the Federal University of Amazonas, through its Center for Distance Education (CDE) was the creation of undergraduate courses in non-face mode. CDE project consisted of organizing headquarters, called poles, to receive courses in Administration, Public Administration, Fine Arts, Biology, Agricultural Sciences and Physical Education. This paper describes this educational experience and presents the structure of the pedagogical model supported by technology (PMT) which allows this scenario to become reality. The innovation of this model is to allow the 1,618 students, distributed in 17 different poles, assisted by CDE, to keep pace with their course through a structure of logistical and technological support adapted to their reality. Resources offered by a Course Management System (CMS), tutors and other specialized tools that support off-line activities make it possible for higher education to reach the most remote regions of Amazon.