Automated short answer grading using LLMs requires per-question calibration with human-graded examples to align outputs with instructor standards. Existing approaches treat all responses as equally informative for calibration. We propose uncertainty-guided sampling that combines stratification to ensure score range coverage with uncertainty-based selection to target responses where human judgment is most informative. Using Gaussian kernel regression, we map LLM scores to instructor preferences from these selected samples. Evaluated on Mohler benchmark and PT\_ASAG dataset, our method achieves 30% RMSE improvement over state-of-the-art (0.501 vs 0.717, m=15) in single-cohort deployment. Uncertainty sampling with m=10 (0.582 RMSE) matches rank-based at m=15 (0.596), requiring 33% fewer samples. However, cross-validation reveals this advantage diminishes on unseen answers, and per-class analysis shows density bias limits performance on minority grades. Our findings suggest that LLM uncertainty enables sample-efficient, instructor-centered calibration, though density-aware strategies are needed for imbalanced distributions.

We propose an automated short answer grading approach that combines the reasoning capabilities of modern LLMs (Claude 3.5 Sonnet, GPT-4o, Gemini 2.0 Flash) with question-specific Gaussian regression models that learn instructor grading preferences from a small calibration set, selected via an informative sampling strategy. The approaches matches fine-tuned transformer accuracy while requiring four to five times fewer labeled examples. We evaluate our approach on two well known datasets that differ in language, domain, student age, class size, and score distribution, namely the English-language Mohler benchmark and the Portuguese-language PT_ASAG dataset. The proposed approach achieves an RMSE of 0.397 vs. 0.717 SOTA on Mohler, and an RMSE of 0.589 vs. 0.67 SOTA on PT_ASAG. This performance substantially outperforms non-adaptive baselines (raw LLM scores: 1.12 RMSE on Mohler) and offers a computationally advantageous and data efficient means compared to fine-tuning methods. Our findings suggest that lightweight per-question user modeling is a viable paradigm for educational AI, balancing automation with instructor control through rapid adaptation to instructor-specific grading preferences.

Context: One of the critical phases in the software development life cycle is software testing. Testing helps with identifying potential bugs and reducing maintenance costs. The goal of automated test generation tools is to ease the development of tests by suggesting efficient bug-revealing tests. Recently, researchers have leveraged Large Language Models (LLMs) of code to generate unit tests. While the code coverage of generated tests was usually assessed, the literature has acknowledged that the coverage is weakly correlated with the efficiency of tests in bug detection. Objective: To improve over this limitation, in this paper, we introduce MuTAP ( Mu tation T est case generation using A ugmented P rompt) for improving the effectiveness of test cases generated by LLMs in terms of revealing bugs by leveraging mutation testing. Methods: Our goal is achieved by augmenting prompts with surviving mutants, as those mutants highlight the limitations of test cases in detecting bugs. MuTAP is capable of generating effective test cases in the absence of natural language descriptions of the Program Under Test (PUTs). We employ different LLMs within MuTAP and evaluate their performance on different benchmarks. Results: Our results show that our proposed method is able to detect up to 28% more faulty human-written code snippets. Among these, 17% remained undetected by both the current state-of-the-art fully-automated test generation tool (i.e., Pynguin) and zero-shot/few-shot learning approaches on LLMs. Furthermore, MuTAP achieves a Mutation Score (MS) of 93.57% on synthetic buggy code, outperforming all other approaches in our evaluation. Conclusion: Our findings suggest that although LLMs can serve as a useful tool to generate test cases, they require specific post-processing steps to enhance the effectiveness of the generated test cases which may suffer from syntactic or functional errors and may be ineffective in detecting certain types of bugs and testing corner cases in PUT s.

Learning management systems provide a wide breadth of data waiting to be analyzed and utilized to enhance student and faculty experience in higher education. As universities struggle to support students’ engagement, success and retention, learning analytics is being used to build predictive models and develop dashboards to support learners and help them stay engaged, to help teachers identify students needing support, and to predict and prevent dropout. Learning with Big Data has its challenges, however: managing great quantities of data requires time and expertise. To predict students at risk, many institutions use machine learning algorithms with LMS data for a given course or type of course, but only a few are trying to make predictions for a large subset of courses. This begs the question: “How can student dropout be predicted on a very large set of courses in an institution Moodle LMS?” In this paper, we use automation to improve student dropout prediction for a very large subset of courses, by clustering them based on course design and similarity, then by automatically training, testing, and selecting machine learning algorithms for each cluster. We developed a promising methodology that outlines a basic framework that can be adjusted and optimized in many ways and that further studies can easily build on and improve.

Accurate assessment of developer expertise is crucial for the assignment of an individual to perform a task or, more generally, to be involved in a project that requires an adequate level of knowledge. Potential programmers can come from a large pool. Therefore, automatic means to provide such assessment of expertise from written programs would be highly valuable in such context.

Estimating the number of latent topics is a prerequisite to using topic modeling algorithms such as LDA. A number of methods for this purpose are compared, including the de facto approach based on perplexity. Results from synthetic data show that a combination of methods yields better estimates than any single method, and in particular it provides an indicator of the reliability of the estimated number of topics, something no single method can do. In this paper, we extend the findings based on synthetic data over real data and introduce a technique, “Cosine inter-intra topics”, to assess the validity of the ground truth of real data. Moreover, we measure the reliability of estimated number of topics through the variance of methods. Results corroborate the ones from synthetic data, suggesting the approach generalizes to real text corpus.

Documents are often organized according to an embedded structure, where a set of documents covers a topic and gets extended to a more specialized topic. We refer to this structure as embedded topics and address the issue of inferring the number and order of topics in a given corpus. While this problem is akin to finding clusters of documents and has been addressed in numerous studies in areas such as topic modeling, information extraction and knowledge discovery, we show that existing approaches are not effective in the specific context of embedded topic structures, and propose a novel technique for that purpose. We also propose an approach to uncover the order of such embedded topics. To determine the number of topics, the proposed method relies on the analysis of eigenvalues of a conditional probability matrix derived from the document-term matrix. We use Kmeans to determine the actual topic clusters, and conditional probability computation to determine the order. We compare the performance of our method to alternative methods for determining clusters and dimensionality. Results show that the proposed approach can effectively derive the right number of topics and embedding structure order.

In Intelligent Tutoring System (ITS), tracing the student's knowledge state during learning has been studied for several decades in order to provide more supportive learning instructions. In this paper, we propose a novel model for knowledge tracing that i) captures students' learning ability and dynamically assigns students into distinct groups with similar ability at regular time intervals, and ii) combines this information with a Recurrent Neural Network architecture known as Deep Knowledge Tracing. Experimental results confirm that the proposed model is significantly better at predicting student performance than well known state-of-the-art techniques for student modelling.

Neural networks and word embeddings are powerful tools to capture latent factors. These tools can provide effective measures of similarities between users or items in the context of sparse data. We propose a novel approach that relies on neural networks and word embeddings to the problem of matching a learner looking for mentoring, and a tutor that is willing to provide this mentoring. Tutors and learners can issue multiple offers/requests on different topics. The approach matches over the whole array of topics specified by learners and tutors. Its performance for tutor-learner matching is compared with the state of the art. It yields similar results in terms of precision, but improves the recall.

General Bayesian network classifier (GBNC) contains only features necessary for classification, so an ideal structure learning solution is to learn GBNC without having to learn the whole Bayesian network (BN). A local search based algorithm called LAS-GBNC is proposed. Given faithfulness assumption, LAS-GBNC relies on the information about each variable's appearance in the so-called d-separator(cut set) to sort candidate CI tests dynamically, performing `effective' ones with priority. Experimental studies indicate that (1) LAS-GBNC achieves the same quality of networks as PC and IPC-BNC, (2)It is much more efficient than PC due to its local search design, and (3) It is obviously faster than IPC-BNC because of its adaptive search strategy.

Selective Sampling design refers to the situation where a study has a fixed number of observations but can decide to allocate them differently among the variables during the data gathering phase, such that some variables will have a greater ratio of missing values than others. In particular, we can decide to allocate more, or less missing values to uncertain variables: those for which the relative frequency is closer to 50% (higher uncertainty), or further from 50% (lower certainty). The main objective of the study is to investigate how a Selective Sampling process helps improve the performance of classification methods. This study specifically asks: "Can Selective Sampling affect the performance of the classification methods?" We focus on the three different classification models of Naïve Bayes, Logistic Regression and Tree Augmented Naive Bayes (TAN) for binary datasets. Three different schemes of sampling are defined: 1-Uniform (random samples) as a baseline, 2-Most Uncertain (higher sampling rate of uncertain items) and 3-Least Uncertain (lower sampling rate of uncertain items). We investigate the impacts of these different schemes on the performance of the three models on 11 different datasets. The results from 100 fold cross-validation show that Selective Sampling in all of the datasets improves the prediction performance of the TAN model and, in more than half of the datasets (54.6%), brings a higher prediction performance to Naïve Bayes and Logistic Regression classifiers.

In this paper, we study the role of query recommendation on mobile search engine. We start with the discussion of the role of query recommendation in modern search engines. Secondly, a mobile search engine, Roboo® (http://wap.roboo.com), is introduced and we discuss the need for query recommendation over mobile search engine. Thirdly, the query recommendation solution working on Roboo® is introduced in detail, including the models, how they are constructed and how they operate online. Finally, we demonstrate the benefits of query recommendation brought on Roboo® based on the analysis of real search log data collected since its release online in August, 2008. Considering the scarcity of scientific publications about the practical values of query recommendation on commercial mobile search engine, this paper should represent an interesting and useful reference for both academic and industrial colleagues.

Flight simulators are expensive devices that airlines use to train their pilots. Currently, the instructor interact with the simulator through touch screens. We analyzed how a voice driven interface can improve the trainer's interaction time efficiency and fluency with the simulator. Real training scenarios were analyzed and 12 representative tasks were chosen for this study. Time comparisons between the voice driven interface and two touch screen interfaces are reported. Twenty voice commands have been derived from the 12 tasks. The analysis of task completion time for touch screen is based on a model-based approach that relieves us from having users performing tasks with the interfaces, the KLM-GOMS model. Results show an average execution time gain of 33.8% using voice commands compared to touch screen commands. However, even though the majority of commands have faster input time for the voice activated interface, some are faster to enter through the touch screen, which suggests that an interface that allows both types of interaction mode might be best.

One current limitation of wearable computers preventing their use as our everyday companions is directly linked to the interaction mechanisms employed for information selection: interactions with wearable computers are simply not integrated enough into the flow of real world user actions. This paper presents a new interaction paradigm where a user can interact with competing software processes embodied in his/her environment through tangible and meaningful real-world actions. After reminding the need for seamless human-wearable computer interactions, we describe this novel concept, the benefits of using natural actions in a mobile setting and the hardware and software architecture of our current prototype. A preliminary user study shows that this paradigm is well received and even preferred over more conventional, and slower, interaction mechanisms.

This paper studies a novel approach advocating the virtual alteration of real-world interfaces through a form of augmented reality. Following an introduction reminding the need for easy to use and more consistent interfaces across our many day to day devices, this paper makes the case for using wearable computers to enhance the interactions between humans and conventional appliances. We present the rationale behind our research and summarize our current prototype's functionalities, architecture and implementation. Preliminary results suggest that virtually altering the interface of real world devices improves execution times for simple tasks using these devices.

Navigation and interaction patterns of Web users can be relatively complex, especially for sites with interactive applications that support user sessions and profiles. We describe such a case for an interactive virtual garment dressing room. The application is distributed over many Web sites, supports personalization and user profiles, and the notion of a multi-site user session. It has its own data logging system that generates approximately 5GB of complex data per month. The analysis of those logs requires more sophisticated processing than is typically done using a relational language. Even the use of procedural languages and DBMS can prove tedious and inefficient. We show an approach to the analysis of complex log data based on a parallel stream processing architecture and the use of specialized languages, namely a grammatical parser and a logic programming module that offers an efficient, flexible, and powerful solution.

We are concerned with the problem of measuring the uncertainty in a broad class of belief networks, as encountered in evidential reasoning applications. In our discussion, we give an explicit account of the networks concerned, and call them the Dempster-Shafer (D-S) belief networks. We examine the essence and the requirement of such an uncertainty measure based on well-defined discrete event dynamical systems concepts. Furthermore, we extend the notion of entropy for the D-S belief networks in order to obtain an improved optimal dynamical observer. The significance and generality of the proposed dynamical observer of measuring uncertainty for the D-S belief networks lie in that it can serve as a performance estimator as well as a feedback for improving both the efficiency and the quality of the D-S belief network-based evidential inferencing. We demonstrate, with Monte Carlo simulation, the implementation and the effectiveness of the proposed dynamical observer in solving the problem of evidential inferencing with optimal evidence node selection.

In this paper, we describe a Web-based system that defines training needs for object-oriented developers by identifying the strong and the weak areas of their knowledge and skills. The system is based on the use of two tools, GAA [8] and UKAT [3], developed at the Computer Research Institute of Montreal (CRIM). UKAT (User Knowledge Assessment Tool) uses a state-of-the-art knowledge assessment method to create a user profile of the proficiency in a subject domain. GAA (Intelligent Guide) is a Web-based training system that uses the UKAT to personalize a training course and facilitate self-learning.

The paper describes an algorithmic means for inducing implication networks from empirical data samples. The induced network enables efficient inferences about the values of network nodes if certain observations are made. This implication induction method is approximate in nature as probabilistic network requirements are relaxed in the construction of dependence relationships based on statistical testing. In order to examine the effectiveness and validity of the induction method, several Monte Carlo simulations were conducted, where theoretical Bayesian networks were used to generate empirical data samples-some of which were used to induce implication relations, whereas others were used to verify the results of evidential reasoning with the induced networks. The values in the implication networks were predicted by applying a modified version of the Dempster-Shafer belief updating scheme. The results of predictions were, furthermore, compared to the ones generated by Pearl's (1986) stochastic simulation method, a probabilistic reasoning method that operates directly on the theoretical Bayesian networks. The comparisons consistently show that the results of predictions based on the induced networks would be comparable to those generated by Pearl's method, when reasoning in a variety of uncertain knowledge domains-those that were simulated using the presumed theoretical probabilistic networks of different topologies.

This paper describes an approach for constructing consistent observers for dynamic systems based on a formalism of nondeterministic event modeling (NEM). The first part of this paper establishes the definitions of NEM with a main focus on the notion of entropy that is extended for measuring the amount of information from nondeterministic events. The paper demonstrates the importance of the proposed entropy measure in contrast to the classical uncertainty measure when used in dynamic systems. The importance of the information measure stems from a potential failure of the classical uncertainty when observing nondeterministic events. Similar problems, which we describe as unobservable states, are often seen in dynamic systems where an uncertainty measure is used in conjunction with some arbitrary decision process. Based on well-defined dynamic systems concepts, the second part of the paper formulates the notion of consistent observer for nondeterministic event modeling. The consistent observer utilizes the monotonically decreasing entropy function of nondeterministic events. An example will be given in this paper which numerically illustrates the notions of NEM and consistent observer.