Emotion recognition is a critical component of affective computing technologies, enabling machines to understand and respond to human emotions more effectively. While traditional models rely on physiological signals, the inclusion of contextual factors, such as personality traits (PT) and emotional intelligence (EI), enhances the precision of these systems. In this paper, we propose a context-aware emotion recognition model using EEG signals, where PT and EI are integrated as additional inputs. Features were extracted using autoencoders, and models were tested with 64, 128, 256, and 512 feature sizes. XGBoost classifiers were employed for classification, and experiments were conducted in two phases: baseline models (using only EEG as input), and context-aware models (incorporating personality and EI scores along with EEG as input). Results show that context-aware models significantly outperformed baseline models, with the highest accuracy of 88.24% and F1-score of 0.8319 achieved when both contexts were included. Statistical tests confirmed a significant improvement in model performance with context, validating our hypothesis that context enhances emotion recognition accuracy. These findings highlight the importance of context awareness in advancing emotion recognition models for more accurate and reliable affective computing systems, with potential applications in mental health monitoring, personalized learning, and human-computer interaction.

Social robots, particularly in assisting children with autism, have exhibited positive impacts on mental health. While prior studies concentrated on social robots in the Global North, there’s limited exploration in the Global South. It’s essential to comprehend special educators’ perspectives for effective integration in resource-constrained settings. Our mixed-methods approach, involving interviews, workshops, and a panel discussion with 25 educators in India, uncovers challenges and opportunities in integrating social robots into autism interventions. The findings highlight the urgent need to democratise the benefits of social robotics. Special educators express concerns about their functional capacity and fear potential redundancy due to the replacement of human efforts by social robots. Despite initial scepticism, professionals suggest various ways to incorporate social robots, emphasising the importance of technological innovation in reshaping and enhancing their roles in autism therapy. We discuss the implications of these findings for developing context-aware solutions and policy-level initiatives necessary in resource-constrained settings.

Amid the evolving landscape of education, evaluating the impact of educational video content on students remains a challenge. Existing methods for assessment often rely on heuristics and self-reporting, leaving room for subjectivity and limited insight. This study addresses this issue by leveraging physiological sensor data to predict student-perceived content effectiveness. Within the realm of educational content evaluation, prior studies focused on conventional approaches, leaving a gap in understanding the nuanced responses of students to educational materials. To bridge this gap, our research introduces a novel perspective, building upon previous work in multimodal physiological data analysis. Our primary contributions encompass two key elements. First, we present the ’Effecti-Net’ architecture, a sophisticated deep learning model that integrates data from multiple sensor modalities, including Electroencephalogram (EEG), Eye Tracker, Galvanic Skin Response (GSR), and Photoplethysmography (PPG). Second, we introduce the ’DECEP’ dataset, a repository comprising 597 minutes of multimodal sensor data. To assess the effectiveness of our approach, we benchmark it against conventional methods. Remarkably, our model achieves a lowest MSE of 0.1651 and MAE of 0.3544 on the DECEP dataset. It offers educators and content creators a comprehensive framework that promotes the development of more engaging educational content.

Human opacity is the intrinsic quality of unknowability of human beings with respect to machines. The descriptive relationship between humans and machines, which captures how much information one can gather about the other, can be explicated using an opacity-transparency relationship. This relationship allows us to describe and normatively evaluate a spectrum of opacity where humans and machines may be either opaque or transparent. In this paper, we argue that the advent of Affective Computing (AC) has begun to shift the ideal position of humans on this spectrum towards greater transparency, while much of this technology is shifting towards opacity. We explore the implications of this shift with regard to the affective information of humans and how the threat to human opacity by AC systems has various adverse repercussions, such as infringement of one's autonomy, deception, manipulation, and increased anxiety. There are also distributive consequences that expose vulnerable groups to unjustified burdens and reduce them to mere profiles. We further provide an assessment of current AC technology, which follows the descriptive relationship between humans and machines from the lens of opacity and transparency. Finally, we foresee and address three possible objections to our claims. These are the beneficence of AC systems, their relation to privacy, and their restrictive capacity to capture human affects. Through these arguments, the paper aims to bring attention to the ontological relationship between humans and machines from the perspective of opacity and transparency while emphasizing on the gravity of the ethical concerns raised by their threat to human opacity.

Automatic personality assessment (APA) has immense potential to improve decision-making and human-machine interaction. Numerous techniques for APA have been proposed in existing literature, with prior psychological studies demonstrating convergent validity between personality and trait emotional intelligence (EI). However, to the best of our knowledge, none in APA literature has leveraged this relationship. Further, the primary language for most APA studies is English. To this end, we proposeVyaktitvaNirdharan, a multi-modal-multitask learning (MM-MTL) system for predicting personality and EI traits using dyadic conversations in a non-intrusive and near real-time fashion. While most of the previous research has focused on APA for the English language, VyaktitvaNirdharan is also amongst the first to use peer-to-peer conversations in Hindi, a low-resource language. We also show the convergent validity between the predictions of the developed system, confirming that the personality and EI traits estimated by our system preserve the properties of the original traits. Our system can identify the Big-Five and EI traits with average F1-scores of 0.88, and 0.82, respectively. The validity of our findings is further supported by a user study involving 7 stakeholders, including recruiters and clinical psychologists, conducted through a prototype. The proposed system demonstrates promising results and has the potential to aid in improving decision-making in recruitment, mental health wellness and enhance human-machine interaction.

Recent research has focused on the effectiveness of Virtual Reality (VR) in games as a more immersive method of interaction. However, there is a lack of robust analysis of the physiological effects between VR and flatscreen (FS) gaming. This paper introduces the first systematic comparison and analysis of emotional and physiological responses to commercially available games in VR and FS environments. To elicit these responses, we first selected four games through a pilot study of 6 participants to cover all four quadrants of the valence-arousal space. Using these games, we recorded the physiological activity, including Blood Volume Pulse and Electrodermal Activity, and self-reported emotions of 33 participants in a user study. Our data analysis revealed that VR gaming elicited more pronounced emotions, higher arousal, increased cognitive load and stress, and lower dominance than FS gaming. The Virtual Reality and Flat Screen (VRFS) dataset, containing over 15 hours of multimodal data comparing FS and VR gaming across different games, is also made publicly available for research purposes. Our analysis provides valuable insights for further investigations into the physiological and emotional effects of VR and FS gaming.

Realistic simulation for miniature control interactions, typically identified by precise and confined motions, commonly found in precision hand-held tools, like calipers, powered engravers, retractable knives, etc., are beneficial for skill training associated with these kinds of tools in virtual reality (VR) environments. However, existing approaches aiming to simulate hand-held tools' miniature control manipulation experiences in VR entail prototyping complexity and require expertise, posing challenges for novice users and individuals with limited resources. Addressing this challenge, we introduce MobiTangibles-proxies for precision hand-held tools' miniature control interactions utilizing smartphone-based magnetic field sensing. MobiTangibles passively replicate fundamental miniature control experiences associated with hand-held tools, such as single-axis translation and rotation, enabling quick and easy use for diverse VR scenarios without requiring extensive technical knowledge. We conducted a comprehensive technical evaluation to validate the functionality of MobiTangibles across diverse settings, including evaluations for electromagnetic interference within indoor environments. In a user-centric evaluation involving 15 participants across bare hands, VR controllers, and MobiTangibles conditions, we further assessed the quality of miniaturized manipulation experiences in VR. Our findings indicate that MobiTangibles outperformed conventional methods in realism and fatigue, receiving positive feedback.

Student attention is an indispensable input for uncovering their goals, intentions, and interests, which prove to be invaluable for a multitude of research areas, ranging from psychology to interactive systems. However, most existing methods to classify attention fail to model its complex nature. To bridge this gap, we propose AttentioNet, a novel Convolutional Neural Network-based approach that utilizes Electroencephalography (EEG) data to classify attention into five states: Selective, Sustained, Divided, Alternating, and relaxed state. We collected a dataset of 20 subjects through standard neuropsychological tasks to elicit different attentional states. The average across-student accuracy of our proposed model at this configuration is 92.3% (SD=3.04), which is well-suited for end-user applications. Our transfer learning-based approach for personalizing the model to individual subjects effectively addresses the issue of individual variability in EEG signals, resulting in improved performance and adaptability of the model for real-world applications. This represents a significant advancement in the field of EEG-based classification. Experimental results demonstrate that AttentioNet outperforms a popular EEGnet baseline (p-value < 0.05) in both subject-independent and subject-dependent settings, confirming the effectiveness of our proposed approach despite the limitations of our dataset. These results highlight the promising potential of AttentioNet for attention classification using EEG data.

Language and communication impairments are considered one of the core features of autism spectrum disorder (ASD). Quantifying the language atypicalities in autism is a challenging task. Prior works have explored acoustic, and text-based features to assess children’s language and communicative behaviours and have shown their relevance in the diagnosis of autism. In this work, we explore the semantic and pragmatic language features in children with autism (CwA) to understand their significance in the diagnosis of autism. We use natural language processing (NLP) and machine learning (ML) techniques to automatically extract relevant features and detect the existence of speech behaviours such as echolalia, semantic coherence, repetitive language, etc. We further analyse their correlation with the clinical diagnosis of autism. We conducted validation experiments on the transcripts of 76 children (35 ASD and 41 TD) extracted from the CHILDES databank. Our analysis shows that the semantic and pragmatic language features are representative candidates for autism diagnosis and are found to complement the syntactic and lexical features in the classification of CwA with an accuracy of 94%. Further, these features being more coherent and relatable to the standard diagnostic tools improves the interpretability of the diagnostic predictions made using speech signals.

Developing human-like conversational agents is a prime area in HCI research and subsumes many tasks. Predicting listener backchannels is one such actively-researched task. While many studies have used different approaches for backchannel prediction, they all have depended on manual annotations for a large dataset. This is a bottleneck impacting the scalability of development. To this end, we propose using semi-supervised techniques to automate the process of identifying backchannels, thereby easing the annotation process. To analyze our identification module’s feasibility, we compared the backchannel prediction models trained on (a) manually-annotated and (b) semi-supervised labels. Quantitative analysis revealed that the proposed semi-supervised approach could attain 95% of the former’s performance. Our user-study findings revealed that almost 60% of the participants found the backchannel responses predicted by the proposed model more natural. Finally, we also analyzed the impact of personality on the type of backchannel signals and validated our findings in the user-study.

Emotion recognition via vision has been deeply associated with facial expressions, and the inference of emotions has, more often than not, been based on the same. However, context, both environmental and social, plays an imperative role in emotion recognition but has not been incorporated widely so far. The meaning of emotion might entirely switch when shifted from one setting to another if only facial expressions are taken into account. Moreover, there exists no study in the Indian context about the same. To cater to this issue, we generate and introduce the Indian Contextual Emotion Recognition (ICER) dataset based on the multi-ethnic Indian context. This paper summarises the Contextual Emotion Learning Challenge (CELC 2021) organized in conjunction with the 16th IEEE Conference on Automatic Face and Gesture Recognition (FG) 2021. We outline the tasks posed in the challenge, the novel dataset, along with its challenges and the evaluation method. Lastly, we conclude by discussing the possible future directions.

Today’s world is undeniably technology-driven and children are the ones who adopt technology with ease. This fact could be leveraged to design assistive technologies for children using social robots as they are observed to be efficient pedagogical agents. Robotic technology is evolving rapidly and robots are designed to play social roles in education, health care and home assistance. However, there has been limited research focusing on the use of robotic technologies for designing interactions with children in the global south, owing to which the response behaviour towards robot-assisted interventions are unknown. To address this gap, we conducted a study to understand the response behaviour of Indian children of the age 3-6 years towards robot-assisted interventions during directive tasks. Our analysis shows that the children could follow to robot’s instructions during the tasks and complete the tasks successfully. The exploratory outcomes also highlight the acceptance and benefits of using robotic assistants as a facilitator in education, cognitive therapies and healthcare.

Chatbots are one of the emerging intelligent systems which interact with customers to solve different queries in a wide range of domain areas. During social interaction, politeness plays a vital role in achieving effective communication. Consequently, it becomes essential to understand how a chatbot’s politeness affects user experience during the interaction. To understand it, we conducted a between-subject user study with two chatbots where one of the chatbots employs polite triggers, and the other one replies intending to answer the queries. To introduce politeness in normal chatbot responses, we used the state-of-the-art tag and generate approach. We first analyzed how different personality traits influence the response of individual persons to polite triggers. In addition, we also investigated the effects of polite triggers among different genders and age groups using a cross-sectional analysis.

Electrodermal activity (EDA) is indicative of psychological processes related to human cognition and emotions. Previous research has studied many methods for extracting EDA features; however, their appropriateness for emotion recognition has been tested using a small number of distinct feature sets and on different, usually small, data sets. In the current research, we reviewed 25 studies and implemented 40 different EDA features across time, frequency and time-frequency domains on the publicly available AMIGOS dataset. We performed a systematic comparison of these EDA features using three feature selection methods, Joint Mutual Information (JMI), Conditional Mutual Information Maximization (CMIM) and Double Input Symmetrical Relevance (DISR) and machine learning techniques. We found that approximately the same numbers of features are required to obtain the optimal accuracy for the arousal recognition and the valence recognition. Also, the subject-dependent classification results were significantly higher than the subject-independent classification for both arousal and valence recognition. Statistical features related to the Mel-Frequency Cepstral Coefficients (MFCC) were explored for the first time for the emotion recognition from EDA signals and they outperformed all other feature groups, including the most commonly used Skin Conductance Response (SCR) related features.

Robot assisted therapy for children with Autism Spectrum Disorder (ASD) should take into account the stakeholders expectations about their potential benefits. Any disparity between the stakeholders expectations and the gained benefits may negatively impact the acceptance and adoption of the robot assisted therapy. In this research, we conducted an observational study with eleven parents and five clinical professionals related with the children with ASD who were preselected to undergo robot assisted therapeutic sessions. The aim was to investigate and identify the potential impact regarding the interventions delivered by the social robots during the interventions, roles of the social robots and benefit offered by them. Specifically, the social robot Cozmo was used for this study. Their opinions were collected using questionnaires and were analyzed quantitatively and qualitatively.The results of the study confirm a positive attitude towards the adoption of these technologies, both among the caregivers and the professionals.

For children with Autism Spectrum Disease (ASD), pre-requisite learning (PRL) skills are particularly important because they form the basis for acquiring other advanced cognitive skills. Globally, researchers have shown that robot-assisted therapy (RAT) can have several positive effects on children with ASD. However, previous researches have failed in clearly mapping the PRL skill training tasks and strategies to robot affordances. In this research, we foster a better understanding of the objectives of the PRL skills required for children with ASD and provide a mapping with robot affordances to execute PRL training activities. In-depth interviews and focus group discussions (N=25) with paediatricians, ASD therapists, and educators from three nonprofit organisations were conducted to understand the clinical practices for teaching PRL skills among children with ASD. Naturalistic observations were used to understand the exercise and training protocols implemented for improving PRL skills among children with ASD. Finally, clinical literature on robotic therapy and technical documents provided by the manufacturers were analysed for identifying commercially available robots and evaluating their features and affordances. Our analysis revealed that affordances offered by several commercially available robots could be effectively leveraged to develop Robot-Assisted Therapies (RATs) to improve PRL skills in children with ASD. Strategies and implications for developing RATs to improve PRL skills among children with ASD are also discussed.

Execution of cognitive stimulation interventions for cognitive training of individuals in need represents significant burden on caregivers in time and labor costs. Recent advancements in Socially Assistive Robotics (SAR) research can be exploited to reduce caregivers burden by work sharing with robots and supplementing/complementing human resources in execution of interventions. Current research evaluates the effectiveness of the SAR empowered cognitive training activity of Bingo Musical among thirty individuals with ID in multi-center trials. A multidimensional evaluation of caregivers workload was conducted; including subjective workload, time spent on users personalized interventions, and qualitative interviews with caregivers. The results of the research confirm a significant reduction in caregivers burden and raise a concern about the need of a specific training of the caregivers to take maximum advantage of SAR in health care.