This large-scale study assesses the impact of human oversight on countering discrimination in AI-aided decision-making for sensitive tasks. It follows a mixed method approach, including a quantitative experiment with Human Resources (HR) and banking professionals in Italy and Germany (N=1411), and qualitative analyses through interviews and workshops with participants and fair AI experts. The results show that human overseers were equally likely to follow advice from a fair AI as from a generic, discriminatory AI. Human oversight does not prevent discrimination by the generic AI. Fair AI reduces gender bias but not nationality bias. Participants’ choices are neither more nor less responsive to their preferences when using an AI or when left on their own. Interviews and workshops with participants highlight individual, organizational and societal biases. In case of conflict, participants prioritize their company’s interests over their own view of fairness. Participants also ask for better guidance on when to override AI recommendations. Fair AI experts stress the need for a comprehensive approach when designing oversight systems. Both technological and social aspects should be taken into consideration to ensure fairness.

Face recognition (FR) has reached a high technical maturity. However, its use needs to be carefully assessed from an ethical perspective, especially in sensitive scenarios. This is precisely the focus of this paper: the use of FR for the identification of specific subjects in moderately to densely crowded spaces (e.g. public spaces, sports stadiums, train stations) and law enforcement scenarios. In particular, there is a need to consider the trade-off between the need to protect privacy and fundamental rights of citizens as well as their safety. Recent Artificial Intelligence (AI) policies, notably the European AI Act, propose that such FR interventions should be proportionate and deployed only when strictly necessary. Nevertheless, concrete guidelines on how to address the concept of proportional FR intervention are lacking to date. This paper proposes a framework to contribute to assessing whether an FR intervention is proportionate or not for a given context of use in the above mentioned scenarios. It also identifies the main quantitative and qualitative variables relevant to the FR intervention decision (e.g. number of people in the scene, level of harm that the person(s) in search could perpetrate, consequences to individual rights and freedoms) and propose a 2D graphical model making it possible to balance these variables in terms of ethical cost vs security gain. Finally, different FR scenarios inspired by real-world deployments validate the proposed model. The framework is conceived as a simple support tool for decision makers when confronted with the deployment of an FR system.

This work presents a longitudinal study of diversity among the Affective Computing research community members. We explore several dimensions of diversity, including gender, geography, institutional types of affiliations and selected combinations of dimensions. We cover the last 10 years of the IEEE Transactions on Affective Computing (TAFFC) journal and the International Conference on Affective Computing and Intelligent Interaction (ACII), the primary sources of publications in Affective Computing. We also present an analysis of diversity among the members of the Association for the Advancement of Affective Computing (AAAC). Our findings reveal a “leaky pipeline” in the field, with a low –albeit slowly increasing over the years– representation of women. They also show that academic institutions clearly dominate publications, ahead of industry and governmental centres. In terms of geography, most publications come from the USA, contributions from Latin America or Africa being almost non-existent. Lastly, we find that diversity in the characteristics of researchers (gender and geographic location) influences diversity in the topics. To conclude, we analyse initiatives that have been undertaken in other AI-related research communities to foster diversity, and recommend a set of initiatives that could be applied to the Affective Computing field to increase diversity in its different facets. The diversity data collected in this work are publicly available, ensuring strict personal data protection and governance rules.

The study of the ethical impact of AI and the design of trustworthy systems needs the analysis of the scenarios where AI systems are used, which is related to the software engineering concept of “use case” and the “intended purpose” legal term. However, there is no standard methodology for use case documentation covering the context of use, scope, functional requirements and risks of an AI system. In this work, we propose a novel documentation methodology for AI use cases, with a special focus on the affective computing domain. Our approach builds upon an assessment of use case information needs documented in the research literature and the recently proposed European regulatory framework for AI. From this assessment, we adopt and adapt the Unified Modeling Language (UML), which has been used in the last two decades mostly by software engineers. Each use case is then represented by an UML diagram and a structured table, and we provide a set of examples illustrating its application to several affective computing scenarios.

ACII is the premier international forum for presenting the latest research on affective computing. In this work, we monitor, quantify and reflect on the diversity in ACII conference across time by computing a set of indexes. We measure diversity in terms of gender, geographic location and academia vs research centres vs industry, and consider three different actors: authors, keynote speakers and organizers. Results raise awareness on the limited diversity in the field, in all studied facets, and compared to other AI conferences. While gender diversity is relatively high, equality is far from being reached. The community is dominated by European, Asian and North American researchers, leading the rest of continents under-represented. There is also a strong absence of companies and research centres focusing on applied research and products. This study fosters discussion in the community on the need for diversity and related challenges in terms of minimizing potential biases of the developed systems to the represented groups. We intend our paper to contribute with a first analysis to consider as a monitoring tool when implementing diversity initiatives. The data collected for this study are publicly released through the European divinAI initiative.

Super-resolution (SR) has achieved great success due to the development of deep convolutional neural networks (CNNs). However, as the depth and width of the networks increase, CNN-based SR methods have been faced with the challenge of computational complexity in practice. More-over, most SR methods train a dedicated model for each target resolution, losing generality and increasing memory requirements. To address these limitations we introduce OverNet, a deep but lightweight convolutional network to solve SISR at arbitrary scale factors with a single model. We make the following contributions: first, we introduce a lightweight feature extractor that enforces efficient reuse of information through a novel recursive structure of skip and dense connections. Second, to maximize the performance of the feature extractor, we propose a model agnostic reconstruction module that generates accurate high-resolution images from overscaled feature maps obtained from any SR architecture. Third, we introduce a multi-scale loss function to achieve generalization across scales. Experiments show that our proposal outperforms previous state-of-the-art approaches in standard benchmarks, while maintaining relatively low computation and memory requirements.

Interpersonal human-human interaction is a dynamical exchange and coordination of social signals, feelings and emotions usually performed through and across multiple modalities such as facial expressions, gestures, and language. Developing machines able to engage humans in rich and natural interpersonal interactions requires capturing such dynamics. This paper addresses primitive emotional contagion during dyadic interactions in which roles are prefixed. Primitive emotional contagion was defined as the tendency people have to automatically mimic and synchronize their multimodal behavior during interactions and, consequently, to emotionally converge. To capture emotional contagion, a cross-recurrence based methodology that explicitly integrates short and long-term temporal dynamics through the analysis of both facial expressions and sentiment was developed. This approach is employed to assess emotional contagion at unimodal, multimodal and cross-modal levels and is evaluated on the Solid SAL-SEMAINE corpus. Interestingly, the approach is able to show the importance of the adoption of cross-modal strategies for addressing emotional contagion.

Although deep face recognition has achieved impressive results in recent years, controversy has arisen regarding racial and gender bias of the models, questioning their deployment into sensitive scenarios. This work quantifies for the first time the demographic imbalance of popular public face datasets in terms of identity, gender and ethnicity. We also publicly release DemogPairs, a new validation set with 10.8K facial images and 58.3M identity verification pairs, distributed in demographically-balanced folds of Asian, Black and White females and males. A benchmark of experiments is carried out using DemogPairs over state-of-the-art deep face recognition models (SphereFace, FaceNet and ResNet50), in order to analyze their cross-demographic behavior. Experimental results demonstrate that studied models suffer from a very structured and damaging demographic bias. Our experiments shine a light on novel testing protocols to appropriately validate the generalization capabilities of face recognition models.

This paper provides an overview of border control processes and how the inclusion of different biometric technologies contributes to its improvement. In particular, facial recognition is one of the latest biometric technology to have been added to this list of technologies. The Face Matching Tool (FMT), a system defined to assist border guards in the process of validating the identity of a travel document holder during the crossing border process, is presented in this paper. The system is built using advanced and high-performance deep learning models. Existing solutions for border control, such as the Automated Border Check gates (ABC gates), are possible thanks to the use of facial recognition as the main option for identity validation. These solutions imply a decrease in the queue time that a traveler expends crossing the border at airports around the globe. The FMT module, together with the rest of the iBorderCtrl system, reduces this waiting time while providing unconstrained and high facial recognition performances at the land borders.

Understanding emotional states is a challenging task which frequently leads to misinterpretation even in human observers. While the perception of emotions has been studied extensively in human psychology, little is known about what factors influence the human perception of emotions in robots and virtual characters. In this paper, we build on the Brunswik lens model to investigate the influence of (a) the agent's embodiment using a 2D virtual character, a 3D blended embodiment, a recording of the 3D platform and a recording of a human, as well as (b) the level of human-likeness on people's ability to interpret emotional facial expressions in an agent. In addition, we measure social traits of the human observers and analyze how they correlate to the success in recognizing emotional expressions. We find that interpersonal differences play a minor role in the perception of emotional states. However, both embodiment and human-likeness as well as related perceptual dimensions such as perceived social presence and uncanniness have an effect on the attribution of emotional states.

Mimicry plays an important role in social interaction. In human communication, it is used to establish rapport and bonding both with other humans, as well as robots and virtual characters. However, little is known about the underlying factors that elicit mimicry in humans when interacting with a robot. In this work, we study the influence of embodiment on participants' ability to mimic a social character. Participants were asked to intentionally mimic the laughing behavior of the Furhat mixed embodied robotic head and a 2D virtual version of the same character. To explore the effect of embodiment, we present two novel approaches to automatically assess people's ability to mimic based solely on videos of their facial expressions. In contrast to participants' self-assessment, the analysis of video recordings suggests a better ability to mimic when people interact with the 2D embodiment.

The “social learning and multimodal interaction for designing artificial agents” workshop aims at presenting scientific and philosophical advances related to social learning and multimodal interaction for enhancing the design of artificial agents. Papers presented in the workshop include studies on human behavior modeling, on social robotics and on virtual agents. Our two invited speakers, Prof. Catherine Pelachaud and Prof. Louis-Philippe Morency will enrich and open the door to further discussion by bringing their widely acknowledged expertise in the field.

In the last years, Affective Computing investigations have focused the efforts in the automatic extraction of human emotions and in increasing the success rates in the emotion recognition task. However, there is a lack of automatic tools that intuitively visualize the users' emotional information. In this work, we propose Emotracker, a novel tool based on the combination of eye tracking and facial emotional recognition technologies that allows the visualization of contents, user emotions and gaze at a glance, and offers a wide range of visualization options, including emotional saccade maps and emotional heat maps. The simultaneous consideration of gaze and emotion opens the door to the evaluation of user engagement, which is of great importance for content creators, but has also other promising fields of application in psychology research and, more specifically, in autism.

Affective Computing researches use to be focused in the automatic extraction of human emotions and in increasing the success rates in the emotion recognition task. However, there is a lack of automatic tools that intuitively visualize the users' emotional information. In this paper, the development of a novel tool that allows the visualization of contents, user emotions and gaze at a glance is presented. The tool, called Emotracker, is based on the combination of eye tracking and facial emotional recognition technologies, and offers a wide range of visualization options, including emotional saccade maps and emotional heat maps. The simultaneous consideration of gaze and emotion opens the door to the evaluation of user engagement in a wide range of applications.

Interactive Digital TV has arisen new forms of interaction not traditionally associated with this medium. One of its applications is the so-called “t-learning” or TV-based interactive learning. To date, most existing t-learning applications confine themselves to edutainment more than to more complex and pedagogical forms of learning, due to the technological constraints imposed by set-top boxes. This paper overcomes those technological barriers to propose the first t-learning affective aware tutoring tool. The tool allows to capture by means of a camera the facial expressions of the student while performing evaluations (tests) through a broadcasted interactive t-learning application at home. It integrates a novel Artificial Intelligence method for facial affect recognition, where facial expressions are evaluated with a psychological 2-dimensional continuous affective approach. Thanks to it, the t-learning tool is able to automatically extract emotional information from the learner, which is further presented in a simple and efficient way to the distance tutor so that he can be aware of student's encountered difficulties and emotional progression during the learning process.

The interpretation of user facial expressions is a very useful method for emotional sensing and it constitutes an indispensable part of affective Human Computer Interface designs. Facial expressions are often classified into one of several basic emotion categories. This categorical approach seems poor to treat faces with blended emotions, as well as to measure the intensity of a given emotion. This paper presents an effective system for facial emotional classification, where facial expressions are evaluated with a psychological 2-dimensional continuous affective approach. At its output, an expressional face is represented as a point in a 2D space characterized by evaluation and activation factors. The proposed system first starts with a classification method in discrete categories based on a novel combination of classifiers, that is subsequently mapped in a 2D space in order to be able to consider intermediate emotional states. The system has been tested with an extensive universal database and human assessment has been taken into consideration in the evaluation of results.