International audience

International audience

Large Language Models (LLMs) have witnessed a rapid growth in emerging challenges and capabilities of language understanding, generation, and reasoning. Despite their remarkable performance in natural language processing-based applications, LLMs are susceptible to undesirable and erratic behaviors, including hallucinations, unreliable reasoning, and the generation of harmful content. These flawed behaviors under-mine trust in LLMs and pose significant hurdles to their adoption in real-world applications, such as legal assistance and medical diagnosis, where precision, reliability, and ethical considerations are paramount. These could also lead to user dissatisfaction, which is currently inadequately assessed and captured. Therefore, to effectively and transparently assess users' satisfaction and trust in their interactions with LLMs, we design and develop LLMChain, a decentralized blockchain-based reputation system that combines automatic evaluation with human feedback to assign contextual reputation scores that accurately reflect LLM's behavior. LLMChain helps users and entities identify the most trustworthy LLM for their specific needs and provides LLM developers with valuable information to refine and improve their models. To our knowledge, this is the first time that a blockchain-based distributed framework for sharing and evaluating LLMs has been introduced. Implemented using emerging tools, LLMChain is evaluated across two benchmark datasets, showcasing its effectiveness and scalability in assessing seven different LLMs.

International audience

International audience

As the number of tools and platforms that have been developed to support learning programming demonstrates, learning programming is becoming more and more ubiquitous in all curricula. In this paper, we present an exploratory study that aims to identify students' programming behaviors. The analysis is based on unsupervised classification algorithms, and features have been selected from prior works on educational data mining. Six students' behaviors were identified using the k-means algorithm. ANCOVA, an extension of analysis of variance (ANOVA), was used to test the main and interaction effects of students' behaviors on their final course scores.

Nowadays, Learning Management Systems (LMS) play an intrinsic role in education. They gather traces about the learner (course view, wiki view, quiz attempt, etc.) in event logs. These logs offer the opportunity to provide dashboards and analysis on learners. There are several techniques that analyze event logs for different purposes (adaptation, recommendation, performance detection, etc.). Within this framework, our central focus is upon Educational Process Mining technique which generates process models for improving learning resource recommendation. We set forward an architecture leading to discover process models and recommend to the learner not only learning resource but also process models, each of which is relative to a specific learning resource. These models exert a certain influence on the result of learning resource recommendation. One of the reason that endows our work with an original aspect is that it automatically analyses event logs based on multi-features extracted from the learner’s profiles. However, the state of the art works require a manual analysis step based on learning results uniquely. We evaluated the discovered process models grounded on the event logs of Moodle LMS. These event logs contain 42,438 traces of 100 students who learned a course over one semester. Results corroborate the good performance of our work.

Research in Computational Narratives has evidenced the need to provide formal models of narratives integrating action representation together with temporal and causal constraints. Adopting an adequate formalization for narrative actions is critical to the development of generative or interactive systems capable of telling stories whilst ensuring narrative coherence, or dynamic adaptation to user interaction. It may also allow to verify properties of narratives at design time. In this paper, we discuss the issues of interactive story design, verification, and piloting for a specific genre of industrial application, in the field of interactive entertainment: in the games we consider, teams of participants in a Virtual Reality application are guided in real time through a narrative experience by a human storyteller. Like in an escape game, the interactive experience is timed: it should be long enough to provide satisfaction to the players, but come to a conclusion before the game session is over in order to provide closure and a sense of achievement to them. We describe how we integrate narrative time in the story design and use it to the verification of temporal properties of scenarios, building on previous work using Linear Logic and Petri Nets.

As the adoption of digital technologies expands, it becomes vital to build trust and confidence in the integrity of such technology. The SPIRIT project investigates the proof of concept of employing novel secure and privacy-ensuring techniques in services set-up in the Internet of Things (IoT) environment, aiming to increase the trust of users in IoTbased systems. The proposed system integrates three highly novel technology concepts developed by the consortium partners. Specifically, a technology, ermed ICMetrics, for deriving encryption keys directly from the operating characteristics of digital devices; secondly, a technology based on a contentbased signature of user data in order to ensure the integrity of sentdata upon arrival; a third technology, termed semantic firewall, which is able to allow or deny the transmission of data derived from an IoT device according to the information contained within the data and the information gathered about the requester.

The most frequently used method of validating automated graders scores has been to compare them with scores awarded by instructors. While this is theoretically possible, research suggests that it is difficult to obtain a constant assessment as there are common problems in human scoring such as inattentiveness, halo effects, sequence effects, etc. The purpose of this study is to analyze the effectiveness of an automated scoring tool called Algo+ by comparing it with human scoring. Specifically, a correlational research design was used to examine the correlations between Algo+ and human raters' performance. We found that automated scores awarded by Algo+ exhibited different positive correlations with scores awarded by instructors that came from two different countries. Furthermore, better correlation was noticed with teachers' overall average scores. In most cases Algo+' behavior was similar to human instructors in awarding scores and it was indistinguishable from teachers. The Ward's hierarchical clustering methods were employed to classify types of teachers' behavior while they scored students' responses. Three types of teachers were classified - lenient, severe, and middle teachers. Algo+ was classified middle in the two exercises.

The design of the digital resource is an expensive process that consumes time and money. By optimizing an adaptive process would reduce time for the designers and users. The increase in the production of digital data in these last few years has raised several issues regarding the management of heterogeneous and multiple source data in the user's environment. In this paper, we focus on adaptive digital resource modelling to assist users in the consolidated management of digital resources in an interactive and adaptive system. In this paper we propose an approach of digital resources modelling which will make the use of digital resources more adaptive. We introduce an adaptive resource model which includes LOM (Learning Object Metadata) metadata, manipulation rules and operators. This model allows an automatic adaptation to user's profiles and user's contexts. An experimental implementation called PRISE (Personal Interactive research Smart Environment) is carried out in our laboratory. PRISE is an interactive and adaptive system to assist researchers in using and managing their digital resources efficiently. PRISE architecture is based on three essential parts of the system : the user model, the process model and the resource model. Our aim is to maintain a consistency in terms of interaction between the users and the digital resources.

Whilst narrative representations have played a prominent role in AI research, there has been a renewed interest in the topic with the development of interactive narratives. A typical approach aims at generating narratives from baseline action representations, most often using planning techniques. However, this research has developed empirically, often as an application of planning. In this paper, we explore a more rigorous formalisation of narrative concepts, both at the action level and at the plot level. Our aim is to investigate how to bridge the gap between action descriptions and narrative concepts, by considering the latter from the perspective of resource consumption and causality. We propose to use Linear Logic, often introduced as a logic of resources, for it provides, through linear implication, a better description of causality than in Classical and Intuitionistic Logic. Besides advances in the fundamental principles of narrative formalisation, this approach can support the formal validation of scenario description as a preliminary step to their implementation via other computational formalisms.