Oral communication is a crucial skill in modern society. Nevertheless, it requires sustained practice and constructive feedback. Consequently, several studies have explored the development of oral communication trainers powered by Artificial Intelligence (AI). However, what characterizes expert speech remains unclear, especially given the need to adapt speech to contextual factors. In instructional environments, the speaker’s communication proficiency is a key determinant of audience learning outcomes. For this reason, we have analyzed 1250 speeches from five types of instructional discourses: in-person college classes ( Lectures ), online learning lessons ( Online Courses ), instructional animations ( Animated Lessons ), supplementary materials for school and high school ( Supplementary Lessons ), and public presentations ( Public Talks ). We extracted 16 speech metrics, including six additional multiple-participant metrics for Lectures . We obtained 250 videos of each discourse type, ensuring a minimum length of five minutes. Our analysis revealed expert values for each speech metric and showed how speech metrics vary across discourse types. We also developed an AI speech classifier that achieved an F1 score of 0.78. The model struggled to identify Online Courses , which is consistent with the Uniform Manifold Approximation and Projection analysis, showing that Online Courses are closely interjected with the speech of other instructional discourses. Furthermore, we identified distinct speech profiles in Lectures, Public Talks , and Online Courses , highlighting variations in speaking styles. This research provides valuable insights into expert speech in instructional discourses by offering reference values that can help speakers refine their delivery and support researchers in developing more effective speech training systems.

Block-based programming environments, such as Scratch and StarLogo Nova, are popular in STEM education for introducing algorithmic thinking to non-technical users. However, capturing and understanding students' interactions with these environments in real-time is challenging for educators, limiting their ability to provide effective feedback and support. In this research, we developed an integrated replay system within StarLogo Nova that records and replays student interactions, providing teachers with the ability to review and reflect on these interactions in detail. Additionally, the system also includes analytics features that help educators summarize user interactions and identify key moments, offering perspectives that go beyond what a standard screen capture can provide. We conducted a preliminary evaluation of the system to assess its accuracy in capturing user interactions, confirming its effectiveness for educators in analyzing student behavior. The results are promising, showing improved analytics and a 99.65% reduction in file size compared to traditional screen capture.

ABSTRACT In recent years, serious games (SGs) have emerged as a powerful tool in education by combining pedagogy and entertainment, facilitating the acquisition of knowledge and skills in engaging environments. SGs enable the collection of valuable interaction data from students, allowing for the analysis of student performance, with artificial intelligence (AI) playing a key role in processing this data to make informed inferences about their knowledge and skills. However, the lack of explainability in AI models represents a significant challenge. This research aims to develop an interpretable model for predicting students' performance in real‐time while playing an SG by: (1) calculating the performance of an interpretable prediction model of task completion in an SG and (2) demonstrating the application of the interpretable model for just‐in‐time (JIT) classroom interventions. Our results show that we are able to predict students' task completion in real‐time with a balanced accuracy result of 77.21% after a short playtime has elapsed. In addition, an explainable artificial intelligence (XAI) approach has been applied to ensure the interpretability of the developed models. This approach supports personalised learning experiences, unlocks AI benefits for non‐technical users, and maintains transparency in education.

Serious Games (SGs) have gained attention as powerful educational tools because of their potential to provide reliable assessments and evaluate hard-to-measure constructs and competencies that are difficult to capture using traditional forms of assessment. Specifically, this study presents a human-centered approach to model and detect persistence—a key component of successful learning outcomes—in the context of SGs. With this purpose in mind, we developed a comprehensive rubric to characterize persistence behaviors in SGs. To design the rubric, we identified a set of persistence profiles and characteristics from previous literature and elaborated a general rubric for identifying persistence behaviors at the level of individual attempts. These characteristics were then mapped onto measurable features within Shadowspect, the SG used for data collection. Following this rubric, two annotators manually labeled 1,374 level attempts from 64 students using two visualization methods: in-game and text replays. With a comprehensive dataset of 2,748 labeled attempts, we trained and evaluated Machine Learning (ML) models for each type of replay to classify persistence behaviors across four categories: Persistence , Non-persistence , Unproductive persistence , and No behavior . Our results indicate that while text-based replays enable efficient annotation with promising performance, in-game replays may provide finer detail for certain complex behaviors, highlighting the strengths and limitations of each visualization method. This work contributes the use of SGs for assessment, illustrating a transparent and adaptable AI-driven approach that enhances reliability and user-centered insights, highlighting the complementary role of human input in optimizing AI-based models to achieve meaningful, user-centered assessments in education.

Video games have assumed an important place in our daily lives. This has led to an increasing interest on the use of games for non-entertainment purposes, introducing the concept of Serious Games (SGs). In particular, SGs are being explored because of their potential to provide reliable assessments, but also because they can measure competences that would be difficult to measure using traditional forms of assessment. However, one of the key issues is that assessment machinery has to be designed specifically for each game, increasing the time and effort when designing and implementing Game-Based Assessments (GBAs). In this research, we introduce a novel approach to develop interoperable GBAs by: (1) designing and creating an ontology that can standardize the GBA area; (2) conducting a validation study on literature metrics to replicate them and designing novel metrics using data from different SGs; (3) conducting a case study that illustrates how our approach can be used in a real life scenario with real data. Our results confirm that the designed ontology can be used to effectively perform GBAs, along with the metrics replicated and designed in the system. We expect our work to solve the current limitations regarding GBA interoperability, thus allowing the deployment of Game-Based Assessments as a Service (GBAaaS).

Games are increasingly being recognized as valuable tools for learning. In addition, they are also being explored for their potential to provide valid and reliable assessments, as they allow to create authentic and engaging assessment contexts through interactive and immersive environments. However, there are challenges to enable Game-based Assessment (GBA) at scale, including the need for interoperability between assessment models and machinery, and the complexity of managing and processing large amounts of data generated by users' interaction with games. In this study, we propose a novel approach that combines the use of ontologies and Big Data technologies for developing interoperable GBAs. The architecture enables assessments to be performed using data from different games, and we also designed and implemented a service API that facilitates the Game-Based Assessment as a Service (GBAaaS) paradigm. GBAaaS simplifies the GBA development process and enables its adoption at scale, making it a promising approach for future developments in this field.

Abstract During the last few years, there has been increasing attention paid to serious games (SGs), which are games used for non‐entertainment purposes. SGs offer the potential for more valid and reliable assessments compared to traditional methods such as paper‐and‐pencil tests. However, the incorporation of assessment features into SGs is still in its early stages, requiring specific design efforts for each game and adding significant time to the design of Game‐based Assessments (GBAs). In this research, we present a completely novel framework that aims to perform interoperable GBAs by: (a) integrating a common GBA ontology model to process RDF data; (b) developing in‐game metrics to infer useful information and assess learners; (c) integrating a service API to provide an easy way to interact with the framework. We then validate our approach through performance evaluation and two use cases, demonstrating its effectiveness in real‐world scenarios with large‐scale datasets. Our results show that the developed framework achieves excellent performance, replicating metrics from previous literature. We anticipate that our work will help alleviate current limitations in the field and facilitate the deployment of GBAs as a Service.

The recent pandemic has changed the way we see education. During recent years, Massive Open Online Course (MOOC) providers, such as Coursera or edX, are reporting millions of new users signing up on their platforms. Though online review systems are standard among many verticals, no standardized or fully decentralized review systems exist in the MOOC ecosystem. In this vein, we believe that there is an opportunity to leverage available open MOOC reviews in order to build simpler and more transparent reviewing systems, allowing users to really identify the best courses out there. Specifically, in our research we analyze 2.4 million reviews (which is the largest MOOC reviews dataset used until now) from five different platforms in order to determine the following: (1) if the numeric ratings provide discriminant information to learners, (2) if NLP-driven sentiment analysis on textual reviews could provide valuable information to learners, (3) if we can leverage NLP-driven topic finding techniques to infer themes that could be important for learners, and (4) if we can use these models to effectively characterize MOOCs based on the open reviews. Results show that numeric ratings are clearly biased (63% of them are 5-star ratings), and the topic modeling reveals some interesting topics related with course advertisements, the real applicability, or the difficulty of the different courses.