We present the acquisition process and the data of DGS-Fabeln-1, a parallel corpus of German text and videos containing German fairy tales interpreted into the German Sign Language (DGS) by a native DGS signer. The corpus contains 573 segments of videos with a total duration of 1 hour and 32 minutes, corresponding with 1428 written sentences. It is the first corpus of semi-naturally expressed DGS that has been filmed from 7 angles, and one of the few sign language (SL) corpora globally which have been filmed from more than 3 angles and where the listener has been simultaneously filmed. The corpus aims at aiding research at SL linguistics, SL machine translation and affective computing, and is freely available for research purposes at the following address: https://doi.org/10.5281/zenodo.10822097.

We present an approach for the prediction of valence, arousal, and dominance of people communicating via text/audio/video streams for a translation from and to sign languages.The approach consists of the fusion of the output of three CNN-based models dedicated to the analysis of text, audio, and facial expressions.Our experiments show that any combination of two or three modalities increases prediction performance for valence and arousal.

This study evaluates a socially interactive agent to create an embodied cobot. It tests a real-time continuous emotional modeling method and an aligned transparent behavioral model, BASSF (boredom, anxiety, self-efficacy, self-compassion, flow). The BASSF model anticipates and counteracts counterproductive emotional experiences of operators working under stress with cobots on tedious tasks. The flow experience is represented in the three-dimensional pleasure, arousal, and dominance (PAD) space. The embodied covatar (cobot and avatar) is introduced to support flow experiences through emotion regulation guidance. The study tests the model's main theoretical assumptions about flow, dominance, self-efficacy, and boredom. Twenty participants worked on a task for an hour, assembling pieces in collaboration with the covatar. After the task, participants completed questionnaires on flow, their affective experience, and self-efficacy, and they were interviewed to understand their emotions and regulation during the task. The results suggest that the dominance dimension plays a vital role in task-related settings as it predicts the participants' self-efficacy and flow. However, the relationship between flow, pleasure, and arousal requires further investigation. Qualitative interview analysis revealed that participants regulated negative emotions, like boredom, also without support, but some strategies could negatively impact well-being and productivity, which aligns with theory.

Self-awareness is a critical factor in social interaction. Teachers being aware of their own emotions and thoughts during class may enable reflection and behavioral change. While inducing self-awareness through mirrors or video is common in face-to-face training, it has been scarcely examined in digital training with virtual avatars. This paper examines the relationship between avatar visual similarity and inducing self-awareness in digital training environments. We developed a theory-based methodology to reliably manipulate perceptually relevant facial features of digital avatars based on human-human identification and emotional predisposition. Manipulating these features allows to create personalized versions of digital avatars with varying degrees of visual similarity.

This article presents a contest between the rating and the paired comparison voting in judging the perceived dominance of virtual characters, the aim being to select the voting mode that is the most convenient for voters while staying reliable. The comparison consists of an experiment where human subjects vote on a set of virtual characters generated by randomly altering a set of physical attributes. The minimum number of participants has been determined via numerical simulation. The outcome is a sequence of stereotypes ordered along their conveyed amount of submissiveness or dominance. Results show that the two voting modes result in equivalently expressive models of dominance. Further analysis of the voting procedure shows that, despite an initial slower learning phase, after about 30 votes the two modes exhibit the same judging speed. Finally, a subjective questionnaire reports a higher (63.8 percent) preference for the paired comparison mode.

In this paper, we describe the current main approaches to sign language translation which use deep neural networks with videos as input and text as output. We highlight that, under our point of view, their main weakness is the lack of generalization in daily life contexts. Our goal is to build a state-of-the-art system for the automatic interpretation of sign language in unpredictable video framing conditions. Our main contribution is the shift from image features to landmark positions in order to diminish the size of the input data and facilitate the combination of data augmentation techniques for landmarks. We describe the set of hypotheses to build such a system and the list of experiments that will lead us to their verification.

This paper introduces the control architecture of a platform aimed at promoting good mental health for workers interacting with collaborative robots (cobots). The platform aim is to render industrial production cells capable of automatically adapting their behavior in order to improve the operator’s quality of experience and level of engagement and to minimize his/her psychological strain. In order to achieve such a goal, an extremely rich and complex framework is required. Starting from the identification of the parameters that could influence the collaboration experience, the envisioned human- driven control structure is presented together with a detailed description of the components required to implement such an automated system. Future works will include proper tuning of control parameters with dedicated experimental sessions, together with the definition of organizational and technical guidelines for the design of a mental-health-friendly cobot-based manufacturing workplace.

Abstract YALLAH is a framework for the creation of real‐time interactive virtual humans. Its production pipeline supports the continuous, parallel development of both the character and the software, and allows users for the deployment of a new character in a few hours of work. YALLAH is based on freely available software, mostly open‐source, and its modular software architecture provides a framework for the seamless integration of new features. Finally, thanks to transpilation, the whole framework is conceived to accommodate multiple game engines.

Articulated hand pose and shape estimation is an important problem for vision-based applications such as augmented reality and animation.In contrast to the existing methods which optimize only for joint positions, we propose a fully supervised deep network which learns to jointly estimate a full 3D hand mesh representation and pose from a single depth image.To this end, a CNN architecture is employed to estimate parametric representations i.e. hand pose, bone scales and complex shape parameters. Then, a novel hand pose and shape layer, embedded inside our deep framework, produces 3D joint positions and hand mesh. Lack of sufficient training data with varying hand shapes limits the generalized performance of learning based methods. Also, manually annotating real data is suboptimal. Therefore, we present SynHand5M: a million-scale synthetic benchmark with accurate joint annotations, segmentation masks and mesh files of depth maps. Among model based learning (hybrid) methods, we show improved results on two of the public benchmarks i.e. NYU and ICVL. Also, by employing a joint training strategy with real and synthetic data, we recover 3D hand mesh and pose from real images in 30ms.

Abstract Gaze is an important aspect of social communication. Previous research has concentrated mainly on the role of speaker gaze and listener gaze in isolation, neglecting the effect of the listener's gaze behavior on the speaker's behavior. This paper presents an exploratory eye‐tracking study involving an interactive human‐like agent following participants' gaze. This study demonstrates that a rather simple gaze‐following mechanism convincingly simulates active listening behavior engaging the speaker. The study also highlights how speakers rely on their interlocutors' gaze when establishing common references.

Cultural heritage traditionally draws a lot of research attention when it comes to exploring the potential benefits from application of novel technology in realistic settings. The domain is rich of physical as well as virtual sites objects and infinite information about them. Hence, it is only natural that whenever new technology appears, it is experimented in cultural heritage -- from early dialog systems to state of the art Humanoid robots, eye trackers, virtual/augmented reality devices, and the Internet of Things (IoT). The AVI-CH workshop nicely demonstrate this with the diversity of topics presented by the papers accepted to the workshop.

Only time will tell if motion-controlled systems are the future of gaming and other industries and if mid-air gestural input will eventually offer a more intuitive way to play games and interact with computers. Whatever the eventual outcome, it is necessary to assess the ergonomics of mid-air input metaphors and propose design guidelines which will guarantee their safe use in the long run. This paper presents an ergonomic study showing how to mitigate the muscular strain induced by prolonged mid-air gesture interaction by encouraging postural shifts during the interaction. A quantitative and qualitative user study involving 30 subjects validates the setup. The simulated musculo-skeletal load values support our hypothesis and show a statistically significant 19% decrease in average muscle loads on the shoulder, neck, and back area in the modified condition compared to the baseline.

In this paper, we describe a virtual interpreter of the Italian sign language (Italian Sign Language, LIS). developed as part of the on--going ATLAS project, on the automatic translation from Italian to Italian Sign Language. The translation system communicates with the user through a virtual signer: the system takes as input a formal representation of a sign language sentence and produces the corresponding animation of the avatar. The architecture of the virtual signer consists of a resource planner, an executor of the planned sign animations, and an animation system.

Tabula-ex-cambio is an interactive installation that delivers visual and audio content. It works as a sort of perpetual billiard game, powered by the data originating from the trend of a stock exchange index. The visual content is a real-time animated 3D computer graphics, that represents a billiard game where each ball is associated with a stock in the index. The audio content is an electroacoustic music composition featuring as many layers of sounds as stocks in the index. All the balls in the billiard are musical sources that play an iterated sequence of sounds while moving on the billiard table; each sequence is altered in frequency by the trend of the related company index. The final delivered visual and audio contents depend on the interaction with the user, who can select a view/listening point on the billiard game. The physical installation consists of a deformed billiard structure connected to a stele through cables; the stele is a vertical display where the billiard game takes place. The visual display relies on the graphic engine Ogre, while the aural display is implemented in SuperCollider. The installation was exposed in Shanghai, Beijing, Birmingham, and Terni (Italy).

The rapid evolution of technology and the processing aspects of some contemporary art forms make maintenance and re-fruition of a number of past masterpieces a difficult task. This is especially true of multimedia installations, with multiple audio and video sources coordinated through some control device, which also accounts for user interaction issues. So, it is not inappropriate to speak of "archeology of multimedia" in the case of the remise-en-oeuvre of an installation. This paper describes a complete example of archeology of multimedia. In particular, the art exhibit is the reprise of the first truly multimedia show of the electronic era: Le Corbusier's Poême électronique, displayed at the World Exhibition in Brussels in 1958. The show, conceived for the Philips Pavilion and consisting of a black&white video, color light ambiances, music moving over sound routes, visual special effects, has never been reprised after the end of the exhibition. The Poême électronique has been entirely reconstructed after a philological investigation through image archives, project sketches and technical documentation from Philips and then delivered in two virtual reality settings. Moreover, two aspects of the project are worth being abstracted from the design and realization of this specific reconstruction and contribute to the design, pre-visualization and fruition of novel contemporary artworks: a language for the description of the control score and an architecture design for the integration of several media sources. The language and the architecture, with a sketch of a friendly interface, are illustrated at the end of the paper.

The paper describes a framework supporting the creation of 3D user interfaces to visualize awareness information about the cooperation context of distributed actors. The paper discusses the motivations behind the framework and illustrates ThreeDmap, an editor allowing the creation and customization of 3D interfaces supporting the perception of awareness information.