We developed a real-time, multimodal Student Engagement Analytics Technology so that teachers can provide just-in-time personalized support to students who risk disengagement. To investigate the impact of the technology, we ran an exploratory semester-long study with a teacher in two classrooms. We used a multi-method approach consisting of a quasi-experimental design to evaluate the impact of the technology and a case study design to understand the environmental and social factors surrounding the classroom setting. The results show that the technology had a significant impact on the teacher's classroom practices (i.e., increased scaffolding to the students) and student engagement (i.e., less boredom). These results suggest that the technology has the potential to support teachers' role of being a coach in technology-mediated learning environments.

“Serious games” are becoming extremely relevant to individuals who have specific needs, such as children with an autism spectrum condition (ASC). Often, individuals with an ASC have difficulties in interpreting verbal and nonverbal communication cues during social interactions. The ASC-Inclusion EU-FP7 funded project aims to provide children who have an ASC with a platform to learn emotion expression and recognition, through play in the virtual world. In particular, the ASC-Inclusion platform focuses on the expression of emotion via facial, vocal, and bodily gestures. The platform combines multiple analysis tools, using onboard microphone and webcam capabilities. The platform utilizes these capabilities via training games, text-based communication, animations, video, and audio clips. This paper introduces current findings and evaluations of the ASC-Inclusion platform and provides detailed description for the different modalities.

Automated driving has the potential to reduce the amount of fatal crashes, lighten the burden of commutes, and democratize mobility access to wider populations. But delegation of control to automation is not without issues. One of the foreseen drawbacks is that users might experience negative emotional reactions to unanticipated or unexplainable automated maneuvers. In this paper we present a novel method to induce targeted emotional reactions, frustration and startle, in simulated automated driving environments. We describe the data collection process for 17 driver - passenger dyads and discuss the data labelling method for generating reliable novel emotion datasets. This contribution is a foundational methodology towards expanding emotional understanding in automated vehicles, a critical skill for building long-term trusted experiences.

There are some implementations towards understanding students' emotional states through automated systems with machine learning models. However, generic AI models of emotions lack enough accuracy to autonomously and meaningfully trigger any interventions. Collecting self-labels from students as they assess their internal states can be a way to collect labeled subject specific data necessary to obtain personalized emotional engagement models. In this paper, we outline preliminary analysis on emotional self-labels collected from students while using a learning platform.

Affective states play a crucial role in learning. Existing Intelligent Tutoring Systems (ITSs) fail to track affective states of learners accurately. Without an accurate detection of such states, ITSs are limited in providing truly personalized learning experience. In our longitudinal research, we have been working towards developing an empathic autonomous 'tutor' closely monitoring students in real-time using multiple sources of data to understand their affective states corresponding to emotional engagement. We focus on detecting learning related states (i.e., 'Satisfied', 'Bored', and 'Confused'). We have collected 210 hours of data through authentic classroom pilots of 17 sessions. We collected information from two modalities: (1) appearance, which is collected from the camera, and (2) context-performance, that is derived from the content platform. The learning content of the content platform consists of two section types: (1) instructional where students watch instructional videos and (2) assessment where students solve exercise questions. Since there are individual differences in expressing affective states, the detection of emotional engagement needs to be customized for each individual. In this paper, we propose a hierarchical semi-supervised model adaptation method to achieve highly accurate emotional engagement detectors. In the initial calibration phase, a personalized context-performance classifier is obtained. In the online usage phase, the appearance classifier is automatically personalized using the labels generated by the context-performance model. The experimental results show that personalization enables performance improvement of our generic emotional engagement detectors. The proposed semi-supervised hierarchical personalization method result in 89.23% and 75.20% F1 measures for the instructional and assessment sections respectively.

Face is a highly utilized biometric, and 3D modality is preferred due to better handling of variations such as pose and illumination. However, occlusions covering the face alter the 3D surface and degrade the recognition performance. To improve recognition rates, the occluded parts should be detected prior to any surface comparison. In this paper, we consider two different occlusion detection approaches: The first one is based on statistical facial surface modeling, where pixel-wise Gaussian Mixture Models are trained. The second algorithm considers occlusion detection as a binary image segmentation problem: The regional cues of depth values are incorporated with neighborhood cues, and the acquired surface is modeled as a graph. The surface pixels are labeled as either face or occlusion via the graph cut technique. Experiments on the Bosphorus and the UMB-DB databases, including realistic occlusion variations, show that both methods improve occlusion detection and face recognition rates as compared to the baseline technique.

With advances in sensor technology, the three-dimensional (3-D) face has become an emerging biometric modality, preferred especially in high security applications. However, dealing with occlusions covering the facial surface is a great challenge, which should be handled to enable applicability to fully automatic security systems. In this paper, we propose a fully automatic 3-D face recognition system which is robust to occlusions. We basically consider two problems: 1) occlusion handling for surface registration, and 2) missing data handling for classification based on subspace analysis techniques. For the alignment problem, we employ an adaptively-selected-model-based registration scheme, where a face model is selected for an occluded face such that only the valid nonoccluded patches are utilized. After registering to the model, occlusions are detected and removed. In the classification stage, a masking strategy, which we call masked projection, is proposed to enable the use of subspace analysis techniques with incomplete data. Furthermore, a regional scheme suitable for occlusion handling is incorporated in classification to improve the overall results. Experimental results on two databases with realistic facial occlusions, namely, the Bosphorus and the UMB-DB, are reported. Experimental results confirm that registration based on the adaptively selected model together with the masked subspace analysis classification offer an occlusion robust face recognition system.

Biometric identification from three-dimensional (3-D) facial surface characteristics has become popular, especially in high security applications. In this paper, we propose a fully automatic expression insensitive 3-D face recognition system. Surface deformations due to facial expressions are a major problem in 3-D face recognition. The proposed approach deals with such challenging conditions in several aspects. First, we employ a fast and accurate region-based registration scheme that uses common region models. These common models make it possible to establish correspondence to all the gallery samples in a single registration pass. Second, we utilize curvature-based 3-D shape descriptors. Last, we apply statistical feature extraction methods. Since all the 3-D facial features are regionally registered to the same generic facial component, subspace construction techniques may be employed. We show that linear discriminant analysis significantly boosts the identification accuracy. We demonstrate the recognition ability of our system using the multiexpression Bosphorus and the most commonly used 3-D face database, Face Recognition Grand Challenge (FRGCv2). Our experimental results show that in both databases we obtain comparable performance to the best rank-1 correct classification rates reported in the literature so far: 98.19% for the Bosphorus and 97.51% for the FRGCv2 database. We have also carried out the standard receiver operating characteristics (ROC III) experiment for the FRGCv2 database. At an FAR of 0.1%, the verification performance was 86.09%. This shows that model-based registration is beneficial in identification scenarios where speed-up is important, whereas for verification one-to-one registration can be more beneficial.

Deformations caused by facial expression variations complicate the task of 3D face registration which is vital for successful 3D face recognition systems. In this work, we propose to use a hierarchical component-based face registration technique capable of handling the difficulties caused by non-rigid nature of faces. Local components independently registered by the iterative closest point (ICP) algorithm provides a fast registration with the use of a generic face model and does not suffer from non-rigidity of human facial surface. Invariance of the proposed approach is further increased by utilizing curvature-based 3D surface descriptors. Identification experiments conducted on the multi-expression Bosphorus database reveal that the accuracy of the classical ICP-based approach can be significantly increased under extreme expression variations.