We explore the integration of extended reality technologies for advancing human-robot collaboration in manufacturing environments. Utilizing a commercial XR headset, the Meta Quest 3, we capture real-time hand and virtual object movements to intuitively control a Universal Robotics e-Series cobot. Our primary objective is to assess whether this interaction paradigm is intuitive for operators while effectively leveraging the cobot's capabilities. Data is streamed from the headset to a broker using MQTT. A local computing module subscribes to the data, translates coordinates into the robot's workspace, and generates control instructions for the cobot. Preliminary findings suggest that XR-based interaction can provide an immersive and efficient framework for human-robot collaboration. These results set the stage for further exploration of multi-user and multi-robot scenarios.

The current global pandemic has resulted in increased social isolation for many. To combat worsening mental states including increased stress and boredom resulting from loneliness, we present a virtual environment designed to simulate social interaction and improve mood. The virtual environment takes the form of a restaurant in which users hold a conversation with a virtual patron. Built in the Unity3D engine and experienced in an Oculus Quest virtual reality headset, our program communicates with an off-the-shelf electroencephalogram (EEG) headset to gather user affective states. Affective states trigger changes in lighting, sound, and conversation topics in real-time based on the user's emotions. The changes made to the environment reflect relevant psychology research to potentially improve the user's mood.

With large class sizes and instructors who may not be equipped to assist struggling students, many students abandon the field, deeming it to be too difficult and not for them. Consistent, constructive, supportive feedback through a Tutoring Companion can scaffold the learning process for students. This poster describes a reasoning model, using neural networks techniques, for a tutoring companion embedded into the Eclipse IDE. The companion provides support for students in a first-year university Java programming course. The companion collects data from students' events and programming assignments, analyzes it for relevant trends, and estimates each student's situation. The input data for the neural network comes from areas with which beginning computer science students often struggle, such as the presence of important keywords and the amount of time spent in a state with errors. Then, it determines the feedback to be provided for students to overcome a detected challenging situation, providing both hints on how to fix the problem with the code, as well as encouragement to help keep students motivated and learning. The effectiveness of the approach is examined among first-year computer science students through the completion of recursion and control flow programming assignments. The students complete surveys regarding their learning experience to assist in evaluating the companion's pedagogical effectiveness, which is discussed with an emphasis on the value of feedback provided.

Virtual worlds enable users' interactions through avatars. Avatars embody individual characteristics from their owners and exhibit those characteristics outward to the community. Motivated by the role of avatars in interpersonal communication, we integrated a generic real-time multimodal affect recognition hub as an input within an online virtual world to make an avatar mirror its owner's affect. Affect vectors (determined by pleasure, arousal, and dominance coordinates) in a continuous affective space are applied to characterize the user's affective state in real time.

The ability of a learning system to infer a student's affects has become highly relevant to be able to adjust its pedagogical strategies. Several methods have been used to infer affects. One of the most recognized for its reliability is face-based affect recognition. Another emerging one involves the use of brain-computer interfaces. In this paper we compare those strategies and explore if, to a great extent, it is possible to infer the values of one source from the other source.

One expected characteristic in modern systems is self-adaptation, the capability of monitoring and reacting to changes into the environment. A particular case of self-adaptation is affective-driven self-adaptation. Affective-driven self-adaptation is about having consciousness of user's affects (emotions) and drive self-adaptation reacting to changes in those affects. Most of the previous work around self-adaptive systems deals with performance, resources, and error recovery as variables that trigger a system reaction. Moreover, most effort around affect recognition has been put towards offline analysis of affect, and to date only few applications exist that are able to infer user's affect in real-time and trigger self-adaptation mechanisms. In response to this deficit, this work proposes a software product line approach to jump-start the development of affect-driven self-adaptive systems by offering the definition of a domain-specific architecture, a set of components (organized as a framework), and guidelines to tailor those components. Study cases with systems for learning and gaming will confirm the capability of the software product line to provide desired functionalities and qualities.

The human-element is crucial for designing and implementing interactive intelligent systems, and therefore on instructional design. This tutorial provides a description and hands-on demonstration for detection of affective states and a description of devices, methodologies and tools necessary for automatic detection of affective states. Automatic detection of affective states requires that the computer sense information that is complex and diverse, it can range from brain-waves signals, and biofeedback readings to face-based and gesture emotion recognition to posture and pressure sensing. Obtaining, processing and understanding that information, to create systems that improve learning, requires the use of several sensing devices (and their perceiving algorithms) and the application of software tools.

Meta-tutoring applies the basic policies of interactive tutoring to get students to adopt effective meta-cognitive strategies. Unfortunately, when the meta-tutor is removed, students often revert to using ineffective strategies. This paper is an early report on the progress of the Affective Meta-Tutoring (AMT) project, which will use an affective learning companion to motivate students to more permanently adopt effective meta-cognitive strategies.

The level up procedure is a method for evaluating the learning gains of educational software, and tutoring systems in particular, that includes some form of embedded assessment. The instruction is arranged in levels that take only a few minutes to master, and students level up when the software indicates they have achieved mastery. This paper reports some methodological lessons learned from applying this procedure in studies of a tutoring system that taught high school students how to model dynamic systems.

Intelligent Tutoring Systems are software applications capable of complementing and enhancing the learning process by providing direct customized instruction and feedback to students in various disciplines. Although Intelligent Tutoring Systems could differ widely in their attached knowledge bases and user interfaces (including interaction mechanisms), their behaviors are quite similar. Therefore, it must be possible to establish a common software model for them. A common software model is a step forward to move these systems from proof-of-concepts and academic research tools to widely available tools in schools and homes. The work reported here addresses: (1) the use of Design Patterns to create an object-oriented software model for Intelligent Tutoring Systems; (2) our experience using this model in a three-year development project and its impact on facets such as creating a common language among stakeholders, supporting an incremental development, and adjustment to a highly shifting development team; and (3) the qualities achieved and trade-offs made.

The computer's ability to recognize human emotional states given physiological signals is gaining in popularity to create empathetic systems such as learning environments, health care systems and videogames. Despite that, there are few frameworks, libraries, architectures, or software tools, which allow systems developers to easily integrate emotion recognition into their software projects. The work reported here offers a first step to fill this gap in the lack of frameworks and models, addressing: (a) the modeling of an agent-driven component-based architecture for multimodal emotion recognition, called ABE, and (b) the use of ABE to implement a multimodal emotion recognition framework to support third-party systems becoming empathetic systems.