Generative AI (GenAI) has posed challenges for traditional assessment frameworks, necessitating innovative approaches to effectively quantify assessment difficulty and distinguish authentic student learning from AI-assisted outcomes. This paper proposes a novel integration of the reinterpreted Revised Bloom's Taxonomy (RBT) and the Structure of Observed Learning Outcomes (SOLO) taxonomy. Moving beyond RBT's traditional hierarchical structure, we develop a radar chart visualization that recognizes how assessment tasks can simultaneously engage multiple cognitive processes with varying complexity levels. The framework offers a nuanced approach to defining and evaluating assessment difficulty, measuring both the breadth of cognitive skills and the depth of thinking structure, along with knowledge domains. Implementation examples demonstrate how this mapping can be systematically applied across various contexts and multiple scales-from individual tasks to comprehensive learning programs, with the potential for automating competency assessment processes. By establishing a quantitative framework for assessing cognitive complexity, this paper offers educators a systematic and standardized method to understand and measure student learning in GenAI-enhanced educational environments.

This research addresses the need for transformation in Computer Science (CS) education as generative AI (GenAI) tools reshape assessment frameworks. GenAI offers new learning opportunities but complicates the distinction between authentic student learning and AI-assisted outcomes, widening the gap between academia and industry needs. We propose a competency framework integrating 12 CS question types with a novel mapping of Revised Bloom's Taxonomy (RBT) and Structure of Observed Learning Outcomes (SOLO) taxonomy. Our radar model moves beyond RBT's hierarchy, recognizing that assessment tasks engage multiple cognitive processes with varying complexity levels. The research consists of four studies: (1) developing a competency-based model, (2) investigating CS literacy requirements for GenAI tool use, (3) validating the model through classroom studies, and (4) creating a scalable framework. This work establishes a quantitative approach for assessing competencies, offering a systematic method for evaluating authentic student learning across diverse educational settings in the AI era.

Knowledge acquiring and forgetting are integral components of the student learning process, both of which are critical for Deep Knowledge Tracing (DKT) tasks that leverage students’ historical learning data to forecast future performance using deep learning methods. However, forgetting mechanisms proposed in many DKT models are not grounded in adequate theories and lack clear interpretation. Building on psychological theories, this study focuses on a crucial aspect of modelling forgetting - memory decay - and performs ablation study and comparative analysis of various decay functions in DKT models to investigate the robustness of their implementation of forgetting. Rather than proposing a new model, our goal is to understand how forgetting is manifested in the popular DKT task, identify shortcomings, and highlight potential opportunities to inform the design of new models grounded in forgetting-related theories. Our results show that the wide-adopted Ebbinghaus’s forgetting curve underperforms when replaced by sigmoid and inverse decay functions in some scenarios. We also present a discussion on the factors that influence the models’ performance, including the effects of completely removing forgetting components from DKT models and their computational implications.

Exploring Educational Escape Room as an Assessment Tool for Computer Science Courses

Social media searches are frequently employed by users to keep them up to date about ongoing events and learn broadly about public opinion on topics that are unfamiliar to them. Nevertheless, there are rising concerns about the results returned that can reinforce users' existing biases - the inclination to one opinion over another. This paper introduces a tool, called BiasChecker, that contributes to the check for bias in search results on a social media platform. BiasChecker follows a distributed and extendable architecture that allows us to simulate users following and unfollowing accounts, search for different polarised topics in a concurrent manner and measure bias. It may be applied to multiple social media platforms. The proposed tool takes into account several factors that can interfere with the detection of bias, e.g., the cross-over effect, geolocation, IP address, and language.

Simplification is a common practice to allow students to understand complex concepts. However, such practice may lead to oversimplification and knowledge compartmentalization problems. This paper proposes a method to minimize the effects of oversimplification and knowledge compartmentalization through a five-step processing chain, following instructional design principles, to foster advanced knowledge acquisition. The proposed method was applied to online courses and a discussion for an example of a Data Science course is provided to show its applicability.

This article describes experiments with a tool, called CourseObservatory, that applies sentiment analysis to comments made by students during course surveys. The main objective is to provide course coordinators and teachers with relevant information about the courses students take based on their qualitative feedbacks. The experiments use a dataset that contains comments in Portuguese found in questionnaires filled out by students from mid-2005 to the first semester of 2018, with a total of nearly 170,000 comments, after a cleaning process that removes blank comments. The experiments show that comments made by students are influenced by the final status achieved (approved or failed), among other facts.

Learning basic concepts before complex ones is a natural form of learning. This paper addresses the specific problem of identifying concept prerequisites to inform about the basic knowledge required to understand a particular concept. Briefly, given a target concept c, the goal is to (a) find candidate concepts in a Knowledge Graph (KG) that serve as possible prerequisite for c; and, (b) evaluate the prerequisite relation between the target and candidates concepts via a supervised learning model. Our approach explores the DBpedia Knowledge Graph and its semantic relations to find candidate concepts as well as a pruning step to reduce the candidate concept set. Finally, we employ supervised learning algorithms to evaluate and generate a list of prerequisites for the target concept. A ground truth created based on expert knowledge is used to validate our approach, exhibiting promising results with a precision varying between 83% and 92.9%.

Exploratory search is a specialized form of information retrieval and discovery. The main aim of the exploratory search is to investigate given information domain to acquire knowledge about it - that is, to learn something about that domain. The level of difficulty of information can affect the learning process and its success. Information with low complexity and relative ease may be uninteresting to learners, who might find it tedious and unrewarding. On the other hand, information with high complexity and difficulty can demoralize learners who perceive it as being impossible, triggering feelings of anxiety. Whether too complex or not complex enough, the seek for information can result in little or no learning. This paper aims to (i) identify features that indicate the complexity of the information represented by a concept; (ii) analyze the features in order to provide a coherent sequence of concepts that balances complexity and learners interest; and, (iii) discuss the challenges, difficulties, and validity of sequencing concepts based on features that indicate complexity. The results show that diversity and variety of concepts as well as detailed semantic descriptions and relations act as indicators for the ideal order of presentation of the information.

Online Q&A communities play a key role in informal learning. This paper reports on an investigation in Q&A communities aiming at bringing a new perspective to the problem of detecting trustworthy users through feature learning methods. These users write useful posts and contribute to the community growth, thus to knowledge dissemination. We propose two feature learning methods and demonstrate that they outperform the state of the as well as are competitive with works that addressed the same problem via hand-engineering process.

Web search engines have gained importance as tools capable of connecting informal and self-learning with formal learning by aiding individuals in retrieving relevant information through the formulation and modification of their queries. Understand the differences between query states and their transitions becomes increasingly important, as doing so makes the optimization of search engines' results according to educational uses and needs possible. This paper introduces the ESKiP Taxonomy of Query States, a classification framework validated in an experiment involving two different query log datasets. It enables the comparison between the behaviors of users in search for knowledge (learners) and users performing transactional or factual searches in Web search engines.

Identifying and monitoring students who are likely to dropout is a vital issue for universities. Early detection allows institutions to intervene, addressing problems and retaining students. Prior research into the early detection of at-risk students has opted for the use of predictive models, but a comprehensive assessment of the suitability of different algorithms and approaches is complicated by the large number of variable features that constitute a student's educational experience. Predictive models vary in terms of their amplitude, temporality and the learning algorithms employed. While amplitude refers to the ability of the model to operate on multiple degrees, temporality is often considered due to the natural temporal aspect of the data. In the absence of a comparative framework of learning algorithms, the aim of this paper has been to provide such an analysis, based on a proposed classification of strategies for predicting dropouts in Higher Education Institutions. Three different student representations are implemented (namely Global Feature-Based, Local Feature-Based, and Time Series) in conjunction with the appropriate learning algorithms for each of them. A description of each approach, as well as its implementation process, are presented in this paper as technical contributions. An experiment based on a dataset of student information from two degrees, namely Business Administration and Architecture, acquired through an automated management system from a university in Brazil is used. Our findings can be summarized as: (i) of the three proposed student representations, the Local Feature-Based was the most suitable approach for predicting dropout. In addition to providing high quality results, the Local Feature-Based representations are simple to build, and the construction of the model is less expensive when compared to more complex ones; (ii) as a conclusion of the results obtained via Local Feature-Based, dropout can be said to be accurately predicted using grades of a few core courses, so there is no need for a complex features extraction process; (iii) considering temporal aspects of the data does not seem to contribute to the prediction performance although it increases computational costs as the model complexity increases.

This paper presents an exploratory search model capable of assisting the visualization of search patterns and clarifying best practices associated to users' decision-making process, with implications in areas related to information retrieval, humancomputer interaction, Web searching and educational technology. The Exploratory Search Knowledge-Intensive Process model considers tasks and search activities as part of a chain of actions that help clarify the reasons why a subject is searched. It also supports the visualization on how the information retrieved is used to define decision criteria about which data is worth extracting, to draw inferences, and to create a shortcut to understanding.

This paper reports on an investigation into outstanding and ordinary users of two Question & Answer (Q&A) communities. Considering some learning-related perspectives such as participation, linguistic traits, social ties, influence, and focus, we found that outstanding users are (i) more likely to engage in discussions; (ii) they tend to use more sophisticated linguistic traits; (iii) their inclusion into a discussion results in longer debates; (iv) they value the diversity of their connections; (v) they participate in several topics, rather than one specialist niche. These findings allow us to use behavioral patterns to predict whether a given user is outstanding and also predict which answer gives a definitive solution for a question.

Searching as Learning and Information Seeking require exploratory search to be modeled for supporting learning. The present paper introduces a model of exploratory search that was applied on web searching in language teacher education, which promoted its evolution and validation, and enabled a visualization of search pattern and learning process. This model was able to help clarify best practices associated to users' decision-making process regarding suitable and not suitable information and to capture the relevance of context variables, personal skills and expertise that users utilize as filters for the search.

The identification of prerequisite relationships among concepts is a fundamental step toward the organization of knowledge for educational purposes. In the context of a learning process, simplest concepts that are requirements to understand and address more complex concepts should be presented first. Therefore, the identification of prerequisite relationships is a fundamental step for effective course design and automatic learning path generation systems. Although there have been recent advances in machine learning methods for the automatic identification of prerequisite relationships between concepts, little research has been done on whether these automatic strategies can be extended to establish precedence relationships among learning resources. The precedence relation between two learning resources establishes which of the resources must be presented first. In this paper, we approach this problem and propose a strategy to identify the precedence relation. Given two learning resources our strategy analyzes prerequisites among the concepts addressed by the learning resources to estimate the precedence relation. A set of 1588 pairs of learning resources extracted from MOOCs refined by human experts is used to evaluate the strategy. The experimental results show that it is possible to identify the precedence relation between learning resources through the automatic identification of prerequisite relationships between concepts.

Higher Education Institutions store a sizable amount of data, including student records and the structure of a degree curriculum. This paper focuses on the problem of identifying how closely students follow the recommended order of the courses in a degree curriculum, and to what extent their performance is affected by the order they actually adopt. It addresses this problem by applying techniques to mine frequent itemsets to student records. The paper illustrates the application of the techniques for a case study involving over 60,000 student records in two undergraduate degrees at a Brazilian University.

This paper investigates expert behaviour in Q&A communities in order to understand their influence in online discussions. Our evaluation shows that experts are more likely to provide help than non-experts, and when they participate in a discussion, the quality and length of the discussions tend to increase. In addition, we propose the usage of two models (Artificial Neural Network and Stochastic Gradient Boosting) to predict worthy discussions in the community. The results show that some adjustments in the models' parameters and in the input data can significantly improve the quality of the predictions.

This paper1 first argues that ontology design may benefit from treating ontologies as theories and from the definition of a set of operations that map ontologies into ontologies, especially their constraints. The paper then defines the class of ontologies used and proposes four operations to manipulate them. It proceeds to discuss how the operations may help design new ontologies. The core of the paper describes an implementation of the operations as a Protégé plug-in, called OntologyManagerTab, and includes use case examples to validate the discussion.

ISCOOL is an interactive educational game for text analysis and interpretation. It draws from several reference datasets, providing users with information about people, organisations and locations, as well as word definitions and historical facts that serve as the basis for reading comprehension and provide a wider context for information to be accessed, interpreted and understood. In game-play, users choose images to best illustrate a read text. The interactivity of the game encourages users to test their knowledge, and critically analyse what was read. ISCOOL was assessed by students with low literacy levels in English. The results show high levels of acceptance and applicability to genuine learning activities.

Understanding students' strengths and weaknesses can help in the design of teaching materials to successfully bridge identified gaps. Formal exams are useful to gauge the extent of learners' memorised information, but have been critiqued for not reflecting the true extent of learners knowledge. Diagnostic assessment assists teachers in setting task-specific plans and goals, for both individual students, and the learner-group as a whole. In this paper, we describe Treasure Explorers, a Game With a Purpose (GWAP) with a multilayered structure that facilitates the learning process, promotes user retention, and is designed to reward contributing players. A comprehensive evaluation based on game log and TAM model was conducted. The log of over 5,500 records reveals that aspects of engagement and learning had occurred, and qualitative evaluations show the applicability and usability of games as diagnostic assessment tools. Treasure Explorers assists teachers in identifying student difficulties, and has significant potential to help educators plan for lesson content.

Question and Answer (Q&A) communities (such as Stackoverflow) have become important places for information exchange and knowledge creation. Their success relies predominantly on two aspects of the feedback generated by their members: quality and speed. Of these, the former reflects on the reputation of the community, whilst the latter is indicative of the efficiency of the Q&A system to correctly answer a given question. In this paper, we present a three phase study for identifying and recommending topical experts in Q&A communities. The first phase investigates the most relevant criteria for identifying reputable members of the community (often experts in a given field), the second phase introduces an approach based on semantic annotations to ascertain their area of specialism, and the last phase presents a method to recommend experts to answer questions in their areas of expertise. Our evaluation (carried out using real-world data from the Biology Stack Exchange Q&A community) shows that the numbers of answers provided by each member can be used as reliable indicators of expertise, and semantic annotations can be successfully used to identify the topics in which they specialize. Furthermore, on average, 74% of the recommendations suggested by our method were successful.

Abstract: Over the last ten years, members of the WEBIST community have dedicated their time and efforts to face a number of research challenges, making significant advances in Information Systems and pointing to new directions for innovation and learning. After ten successful WEBIST conferences and several scientific pub-lications, an extensive analysis of the WEBIST conferences has been carried out (involving authors, publi-cations, conference impact, topics coverage, community analysis and other aspects) to possibly assist us to further advance Information Systems. Thus, in this paper, we present an in-depth analysis of the last ten WEBIST conferences based on social network analysis, bibliometrics and statistical measures and describe a Web-based application built on top of triplified datasets to interactively explore the findings and possibly assist the Information Systems community to reveal new directions. 1

The volume of information on the Web is constantly growing. Consequently, finding specific pieces of information becomes a harder task. Wikipedia, the largest online reference Website is beginning to witness this phenomenon. Learners often turn to Wikipedia in order to learn facts regarding different subjects. However, as time passes, Wikipedia articles get larger and specific information gets more difficult to be located. In this work, we propose an automatic annotation method that is able to precisely assign categories to any textual resource. Our approach relies on semantic enhanced annotations and Wikipedia's categorization schema. The results of a user study shows that our proposed method provides solid results for classifying text and provides a useful support for locating information. As implication, our research will help future learners to easily identify desired learning topics of interest in large textual resources.

Forums play a key role in the process of knowledge creation, providing means for users to exchange ideas and to collaborate. However, educational forums, along several others online educational environments, often suffer from topic disruption. Since the contents are mainly produced by participants (in our case learners), one or a few individuals might change the course of the discussions. Thus, realigning the discussed topics of a forum thread is a task often conducted by a tutor or moderator. In order to support learners and tutors to harmonically align forum discussions that are pertinent to a given lecture or course, in this paper, we present a method that combines semantic technologies and a statistical method to find and expose relevant topics to be discussed in online discussion forums.

The volume of information on the Web is constantly growing. Consequently, finding specific pieces of information becomes a harder task. Wikipedia, the largest online reference Website is beginning to witness this phenomenon. Learners often turn to Wikipedia in order to learn facts regarding different subjects. However, as time passes, Wikipedia articles get larger and specific information gets more difficult to be located. In this work, we propose an automatic annotation method that is able to precisely assign categories to any textual resource. Our approach relies on semantic enhanced annotations and the categorization schema of Wikipedia. The results of a user study show that our proposed method provides solid results for classifying text and provides a useful support for locating information. As implication, our research will help future learners to easily identify desired learning topics of interest in large textual resources.

Abstract: Learning material usually has a logical structure, with a beginning and an end, and lectures or sections that build upon one another. However, in informal Web-based learning this may not be the case. In this paper, we present a method for automatically calculating a tentative order in which objects should be learned based on the estimated complexity of their contents. Thus, the proposed method is based on a process that enriches textual objects with links to Wikipedia articles, which are used to calculate a complexity score for each object. We evaluated our method with two different datasets: Wikipedia articles and online learning courses. For Wikipedia data we achieved correlations between the ground truth and the predicted order of up to 0.57 while for subtopics inside the online learning courses we achieved correlations of 0.793. 1

A good movie is like a good book. As a good book can serve entertaining and learning purposes, so does a movie. In addition to that, movies are in general more engaging and reach a wider audience. In this work, we present and evaluate a method that overcomes the challenge of generating recommendations among heterogeneous resources. In our case, we recommend movies in the context of a learning object. We evaluate our method with 60 participants that judged the relevance of the recommendations. Results show that, in over 74% of the cases the recommendations are in fact related to the given learning object, outperforming a text-based recommendation approach. The implications of our work can take learning outside the classroom and invoke it during the joy of watching a movie.

A competence is the effective performance in a domain at different levels of proficiency. Educational institutions apply competences to understand whether a person has a particular level of ability or skill. Educational resource enriched with competence information allows learners identifying, on a fine-grained level, which resources to study with the aim to reach a specific competence target. However, the process of annotating learning objects with competence levels is a very time consuming task, ideally, this task should be performed by experts on the subjects of the educational resources. Due to this, most educational resources available online do not enclose competence information. In this paper, we present a method to tackle the problem of automatically assigning an educational resource with competence levels. To solve these problems, we exploit information extracted from external repositories available on the Web, which lead us to a domain independent approach. We demonstrate the quality of the proposed methods through an evaluation on real world data with an additional user study. Results show that the automatic competence level assignment achieves 84% precision on ground truth data. The key implications of our approach are: first, it effectively facilitates experts in the arduous task of competence assignment and second, it directly supports learners to retrieve proper leveled material.

This paper presents a text simplification method that transforms complex sentences into simplified forms. Our method uses NLP-techniques to simplify the text based on the target audience context, improving its overall understandability. We evaluate our approach in two aspects: grammatical structure and understandability. In both aspects, our approach achieved good results, showing its applicability to the learning process.

Connectivity and relatedness of Web resources are two concepts that define to what extent different parts are connected or related to one another. Measuring connectivity and relatedness between Web resources is a growing field of research, often the starting point of recommender systems. Although relatedness is liable to subjective interpretations, connectivity is not. Given the Semantic Web's ability of linking Web resources, connectivity can be measured by exploiting the links between entities. Further, these connections can be exploited to uncover relationships between Web resources. In this paper, we apply and expand a relationship assessment methodology from social network theory to measure the connectivity between documents. The connectivity measures are used to identify connected and related Web resources. Our approach is able to expose relations that traditional text-based approaches fail to identify. We validate and assess our proposed approaches through an evaluation on a real world dataset, where results show that the proposed techniques outperform state of the art approaches.

Every day, millions of new images, videos and audios are uploaded to the web. However, unlike text-based content, audio and video objects cannot be indexed by search engines. Thus, much valuable multimedia content stay unreachable for a great majority of online users. To overcome this problem we introduce a technique that automatically generates semantically enhanced descriptions of audio and video objects. The goal is to facilitate indexing and retrieval of the objects with the help of traditional search engines. Basically, the technique automatically generates static Web pages that describe the content of the digital audio and video objects, organized in such a way as to facilitate locating segments of the audio or video that correspond to the descriptions. The technique is a mashup ofWeb services that also provides translation of the descriptions and semantic enhancement. We thoroughly analyzed the click-data comparing accesses to the digital content before and after the automatic generation of the descriptions. The outcomes suggest that the technique significantly improve the retrieval of items, not only in terms of visibility, but also brings down language barriers, by supporting multilingual access.