In this pilot study, we used a balanced A/B testing method to evaluate how an environment—a Virtual Reality (VR) head-mounted display (HMD) versus a traditional desktop—affects the performance of 20 participants on a musical pitch-colour association test. We aimed to discern the influence of testing environments on musical pitch-colour associations. Our findings revealed no significant difference in performance between the VR HMD and the traditional desktop conditions. Previous studies highlight VR’s potential, but the results of our investigation—focusing on limited immersive and presence qualities—suggest that working in VR as opposed to a standard desktop environment has no significant influence on musical pitch-colour association test results. This outcome prompts a further investigation into other inherent VR characteristics, such as spatial audio, natural environment simulations, and interactive object manipulation, which could potentially enhance the effectiveness of auditory-visual association tests. This study is the first in a series of studies to explore how VR technologies can be exploited to augment multi-modal testing, with a fully immersive musical pitch-colour association test envisioned.

This demo paper presents a Virtual Reality (VR) application designed to explore innovative methods of learning basic music theory through sound-colour cross-modal associations.Leveraging the immersive capabilities of the latest head-mounted displays, the application provides an interactive and engaging platform with significant potential for music education.The demo marks an initial step in investigating how VR's immersive features can enhance the understanding of sound-colour associations and their role in learning musical concepts, specifically intervals and melodies within the C Major scale.Two groups will participate in the study: a control group will learn using a predefined sound-colour mapping, while the experimental group will create their own personalised mappings through a pre-test in VR, which will be applied in subsequent lessons.The forthcoming study aims to provide valuable insights into the effectiveness of VR and cross-modality in facilitating music theory education.

Transactive Memory System (TMS) is a mental representation of the distribution of knowledge between the members of a team. Can an Embodied Conversational Agent perform as well as a Human when intervening as a leader to support the development of the team’s TMS? And, if yes, are there differences in the way the team perceives their respective interventions? In this paper, a perceptive online study is conducted on how Human leader interventions affect the perception of a team’s TMS. The results are compared to the ones from a previous study evaluating an Embodied Conversational agent leader rather than a human one. Both the agent and the human adopt nonverbal behaviors characterizing 2 leadership styles: Transformational (TFL) and Transactional (TAL). TFL is expected to stimulate team members curiosity and creativity in problem-solving; instead, TAL emphasizes the role of the leader in supervising the team, providing it with feedback when needed. The results show that the intervention from both the agent and the human are perceived to potentially improve the perceived TMS of a team. Another interesting insight is that the TFL style works better when performed by the Human, where both the TAL and TFL style perform well when realized by the agent.

Can we sharpen our auditory and visual senses and better understand the relationship between these modalities to benefit our interactions in human-computer interfaces? This research paper proposes a framework to understand auditory-visual associations and explore the impact of emotion, personality, age and gender in understanding information from both modalities. Studies into the areas of emotion and personality as well as their association with the auditory and visual senses have increased within the fields of psychology, neuroscience, affective computing and human-computer interaction (HCI). From a HCI perspective, advances in technologies and machine learning techniques provide a new way to understand people and to develop systems where computers work along side people to help develop efficient interactions and clearer perceptions of our environment. The proposed framework will be developed along side a personalised auditory-visual interface that can be used to provide intelligent interactions with users that can help them learn from efficient associations between their senses. This research can be used to create personalised auditory-visual-emotion-personality profiles that can be use in adaptive musical teaching platforms, as well as mental health and wellness applications for more personalised care programs.

Can cross-modal associations between notes and colours and between notes and emotions be strengthened by using an ad-hoc designed computer program? In this paper we present an experiment that we conducted to see (H1) if participants' associations between the auditory and visual senses could be strengthened through immersion in a multi-sensory program and (H2) if there was any correlation between auditory-visual and the auditory-emotion associations. In our experiment, participants were asked to use the SoundStrokes program, an application that translates notes played on an analog music instrument (i.e., a guitar) into coloured graphical shapes (i.e., rectangles). Between several training sessions with the program, participants were asked to associate single music notes to colours and emotions. Applications of our work may vary from entertainment, to enrich in a systematic manner the user emotional experience in, for example, videogames or immersive experiences, to digital health, in which relatively simple tools can be developed, for example, to reduce anxiety through music.