We report on the development of two video games with educational intent and entertainment ambitions. The first game – “Cash or Card?” – is a cashier simulation to raise awareness of the issues surrounding cash vs electronic payments, a hotly debated topic shrouded in false information and conspiracy theories. The second game – “Untitled Journalist Game” – is an investigative reporting simulation focusing on the analysis and evaluation of information sources, and on how information is presented in the media. We present the design and development process of the two games, and some preliminary usage data of “Cash or Card?” We found that the synergy of game designers, educators, and game researchers is critical in developing and evaluating games to educate and entertain so that all the actors can contribute confidently according to their expertise, and produce games that communicate important content while challenging and entertaining the players.

Learning to play a musical instrument such as a piano requires many hours of exercises, generally taken from a “method” book. These books are collections of progressive exercises intended to teach specific techniques and address the commonest mistakes and difficulties that players face while learning. One downside of these books is that the exercises are not personalized to the students and thus cannot address specific difficulties and characteristics of each learner. Given the many recent advances in the field of music generation, we propose that it should be possible to generate exercises automatically to form a personalized method for each student. The teacher would describe the characteristics of the student and their strengths and weaknesses to a software system, as well as the teaching goals that should be covered in the generated exercises, and the system would create exercises that are specific to the needs of the student and the concerns of the teacher, allowing for a more effective and engaging learning experience. In this paper, we describe a project trying to design such a system, stating research questions, describing the tentative methodology, and outlining its potential impact for both research in music generation and in computer-supported education.

Digital games aimed at improving cognitive and/or motor-sensory skills need to be carefully designed to take into account the characteristics and needs of particular categories of users. Several novel mini-games explicitly aimed at children with visual impairment (VI) were co-designed by a multidisciplinary team which involved computer engineers and a therapy team from the Robert Hollman Foundation (Padova, Italy). These games are played by children moving within a large-scale interactive environment - i.e., a floor portion placed under a motion capture system capable of tracking one or more people - with the game linking the players movements to the audio and visual output to produce meaningful interactions. We report on a pilot study of the usability of the system involving 11 children with VI. The results allowed us to improve the system and to define a set of guidelines useful for designers and developers of similar systems. Supplementary Information: The online version contains supplementary material available at 10.1007/s11042-022-13665-7.

Abstract Making music with others is both an artistic act and a social activity. Music therapists can leverage the social aspects of music to increase the well-being of their patients by interacting with them musically, improvising rhythms and melodies together on shared musical instruments. This activity requires highly trained professionals and is therefore expensive for the clients. We propose a serious game that can help people without musical training interact by collaboratively creating a rhythm using MIDI drum pads. The gaming system analyzes the rhythm in real-time and adds musical feedback that is synchronized to what the users play, enhancing the aesthetical experience that is crucial to the musical interaction and its therapeutic effects. We assessed our system through quantitative metrics showing its capability of following a user-established tempo. Test players also completed a questionnaire, which showed they found the experience pleasant and engaging, and that the musical augmentation was helpful to their interaction.

This paper investigated how different emotional states of a walker can be rendered and recognized by means of footstep sounds synthesis algorithms. In a first experiment, participants were asked to render, according to imagined walking scenarios, five emotions (aggressive, happy, neutral, sad, and tender) by manipulating the parameters of synthetic footstep sounds simulating various combinations of surface materials and shoes types. Results allowed to identify, for the involved emotions and sound conditions, the mean values and ranges of variation of two parameters, sound level and temporal distance between consecutive steps. Results were in accordance with those reported in previous studies on real walking, suggesting that expression of emotions in walking is independent from the real or imagined motor activity. In a second experiment participants were asked to identify the emotions portrayed by walking sounds synthesized by setting the synthesis engine parameters to the mean values found in the first experiment. Results showed that the involved algorithms were successful in conveying the emotional information at a level comparable with previous studies. Both experiments involved musicians and non-musicians. In both experiments, a similar general trend was found between the two groups.

This paper investigates the production and identification of emotional states of a walker using plantar vibro-tactile simulations. In a first experiment, participants were asked to render, according to imagined walking scenarios, five emotions (aggressive, happy, neutral, sad, and tender) by manipulating the parameters of synthetic footstep vibrations simulating various combinations of surface materials and shoes. Results allowed to identify, for the involved emotions and vibration conditions, the mean values and ranges of variation of two parameters, vibration amplitude and temporal distance between consecutive steps. Results were in accordance with those reported in previous studies on real walking, suggesting that the plantar vibro-tactile expression of emotions in walking is independent of the real or imagined motor activity. In a second experiment, participants were asked to identify the emotions portrayed by walking vibrations synthesized by setting the synthesis engine parameters to the mean values found in the first experiment. Results showed that the involved algorithms were successful in conveying the emotional information at a level comparable with previous studies. Results of both experiments revealed strong similarities with those of an analogous study on footstep sounds suggesting that emotionally expressive walking styles are consistently produced and recognized at auditory and plantar vibro-tactile level.

The important role of the valence and arousal dimensions in representing and recognizing affective qualities in music is well established. There is less evidence for the contribution of secondary dimensions such as potency, tension and energy. In particular, previous studies failed to find significant relations between computable musical features and affective dimensions other than valence and arousal. Here we present two experiments aiming at assessing how musical features, directly computable from complex audio excerpts, are related to secondary emotion dimensions. To this aim, we imposed some constraints on the musical features, namely modality and tempo, of the stimuli.The results show that although arousal and valence dominate for many musical features, it is possible to identify features, in particular Roughness, Loudness, and SpectralFlux, that are significantly related to the potency dimension. As far as we know, this is the first study that gained more insight into the affective potency in the music domain by using real music recordings and a computational approach.

Computational systems for generating expressive musical performances have been studied for several decades now. These models are generally evaluated by comparing their predictions with actual performances, both from a performance parameter and a subjective point of view, often focusing on very specific aspects of the model. However, little is known about how listeners evaluate the generated performances and what factors influence their judgement and appreciation. In this article, we present two studies, conducted during two dedicated workshops, to start understanding how the audience judges entire performances employing different approaches to generating musical expression. In the preliminary study, 40 participants completed a questionnaire in response to five different computer-generated and computer-assisted performances, rating preference and describing the expressiveness of the performances. In the second, “GATM” ( Gruppo di Analisi e Teoria Musicale ) study, 23 participants also completed the Music Cognitive Style questionnaire. Results indicated that music systemizers tend to describe musical expression in terms of the formal aspects of the music, and music empathizers tend to report expressiveness in terms of emotions and characters. However, high systemizers did not differ from high empathizers in their mean preference score across the five pieces. We also concluded that listeners tend not to focus on the basic technical aspects of playing when judging computer-assisted and computer-generated performances. Implications for the significance of individual differences in judging musical expression are discussed.

Video-based surveillance systems may benefit from the integration with microphone arrays for the localization of sound events. Applying the sound localization techniques to the surveillance of large areas requires addressing some open issues, such as the non uniform resolution of the microphones-based localization systems. This paper presents a new method for tracking moving sound events based on an Hidden Markov Model (HMM), which exploits a priori information derived from medium and longterm observations of the monitored area. The results obtained with simulated trajectories show that the HMM-based tracker is able to significantly reduce the localization error. Applications can be found in surveillance systems for large areas, such as square, streets, or parking lots, where it is of interest the monitoring of moving vehicles and people.

We present a novel approach to locate multiple acoustic sources in far-field environments, in order to solve an interesting problem in different application domain, such as: audio surveillance systems and soundscape analysis frameworks. This approach aims at finding a solution to the ambiguities in Direction Of Arrivals (DOAs) combination caused by simultaneous multiple sources. The algorithm is based on two steps: the separation of the sources by means of beamforming techniques and the comparison of the Incident Signal Power (ISP) spectrum by means of a spectral distance measure. We implemented a prototype, composed by two linear arrays, that has been successfully tested in a real noisy environment.

In this paper, we propose a novel recommendation policy for driving scenarios. While driving a car, listening to an audio track may enrich the atmosphere, conveying emotions that let the driver sense a more arousing experience. Here, we are introducing a recommendation policy that, given a video sequence taken by a camera mounted onboard a car, chooses the most suitable audio piece from a predetermined set of melodies. The mixing mechanism takes inspiration from a set of generic qualitative aesthetical rules for cross-modal linking, realized by associating audio and video features. The contribution of this paper is to translate such qualitative rules into quantitative terms, learning from an extensive training dataset cross-modal statistical correlations, and validating them in a thoroughly way. In this way, we are able to define what are the audio and video features that correlate at best (i.e., promoting or rejecting some aesthetical rules), and what are their correlation intensities. This knowledge is then employed for the realization of the recommendation policy. A set of user studies illustrate and validate the policy, thus encouraging further developments toward a real implementation in an automotive application.

Expression is an important aspect of music performance. It is the added value of a performance and is part of the reason that music is interesting to listen to and sounds alive. Understanding and modeling expressive content communication is important for many engineering applications in information technology. For example, in multimedia products, textual information is enriched by means of graphical and audio objects. In this paper, we present an original approach to modify the expressive content of a performance in a gradual way, both at the symbolic and signal levels. To this purpose, we discuss a model that applies a smooth morphing among performances with different expressive content, adapting the audio expressive character to the user's desires. Morphing can be realized with a wide range of graduality (from abrupt to very smooth), allowing adaptation of the system to different situations. The sound rendering is obtained by interfacing the expressiveness model with a dedicated postprocessing environment, which allows for the transformation of the event cues. The processing is based on the organized control of basic audio effects. Among the basic effects used, an original method for the spectral processing of audio is introduced.