Abstract The research communities studying visualization and sonification for data display and analysis share exceptionally similar goals, essentially making data of any kind interpretable to humans. One community does so by using visual representations of data, and the other community employs auditory (non‐speech) representations of data. While the two communities have a lot in common, they developed mostly in parallel over the course of the last few decades. With this STAR, we discuss a collection of work that bridges the borders of the two communities, hence a collection of work that aims to integrate the two techniques into one form of audiovisual display, which we argue to be “more than the sum of the two.” We introduce and motivate a classification system applicable to such audiovisual displays and categorize a corpus of 57 academic publications that appeared between 2011 and 2023 in categories such as reading level, dataset type, or evaluation system, to mention a few. The corpus also enables a meta‐analysis of the field, including regularly occurring design patterns such as type of visualization and sonification techniques, or the use of visual and auditory channels, showing an overall diverse field with different designs. An analysis of a co‐author network of the field shows individual teams without many interconnections. The body of work covered in this STAR also relates to three adjacent topics: audiovisual monitoring, accessibility, and audiovisual data art. These three topics are discussed individually in addition to the systematically conducted part of this research. The findings of this report may be used by researchers from both fields to understand the potentials and challenges of such integrated designs while hopefully inspiring them to collaborate with experts from the respective other field.

Abstract One of the commonly used visualization techniques for multivariate data is the parallel coordinates plot. It provides users with a visual overview of multivariate data and the possibility to interactively explore it. While pattern recognition is a strength of the human visual system, it is also a strength of the auditory system. Inspired by the integration of the visual and auditory perception in everyday life, we introduce an audio-visual analytics design named Parallel Chords combining both visual and auditory displays. Parallel Chords lets users explore multivariate data using both visualization and sonification through the interaction with the axes of a parallel coordinates plot. To illustrate the potential of the design, we present (1) prototypical data patterns where the sonification helps with the identification of correlations, clusters, and outliers, (2) a usage scenario showing the sonification of data from non-adjacent axes, and (3) a controlled experiment on the sensitivity thresholds of participants when distinguishing the strength of correlations. During this controlled experiment, 35 participants used three different display types, the visualization, the sonification, and the combination of these, to identify the strongest out of three correlations. The results show that all three display types enabled the participants to identify the strongest correlation — with visualization resulting in the best sensitivity. The sonification resulted in sensitivities that were independent from the type of displayed correlation, and the combination resulted in increased enjoyability during usage.

Abstract Both sonification and visualization convey information about data by effectively using our human perceptual system, but their ways to transform the data differ. Over the past 30 years, the sonification community has demanded a holistic perspective on data representation, including audio-visual analysis, several times. A design theory of audio-visual analysis would be a relevant step in this direction. An indispensable foundation for this endeavor is a terminology describing the combined design space. To build a bridge between the domains, we adopt three of the established theoretical constructs from visualization theory for the field of sonification. The three constructs are the spatial substrate, the visual mark, and the visual channel. In our model, we choose time to be the temporal substrate of sonification. Auditory marks are then positioned in time, such as visual marks are positioned in space. Auditory channels are encoded into auditory marks to convey information. The proposed definitions allow discussing visualization and sonification designs as well as multi-modal designs based on a common terminology. While the identified terminology can support audio-visual analytics research, it also provides a new perspective on sonification theory itself.

Abstract In this paper, we describe a proof-of-concept for the implementation of a wearable auditory biofeedback system based on a sensor-instrumented insole. Such a system aims to assist everyday users with static and dynamic exercises for gait rehabilitation interventions by providing auditory feedback based on plantar pressure distribution and automated classification of functional gait disorders. As ground reaction force (GRF) data are frequently used in clinical practice to quantitatively describe human motion and have been successfully used for the classification of gait patterns into clinically relevant classes, a feed-forward neural network was implemented on the firmware of the insoles to estimate the GRFs using pressure and acceleration data. The estimated GRFs approximated well the GRF measurements obtained from force plates. To distinguish between physiological gait and gait disorders, we trained and evaluated a support vector machine with labeled data from a publicly accessible dataset. The automated gait classification was then sonified for auditory feedback. The potential of the implemented auditory feedback for preventive and supportive applications in physical therapy was finally assessed with both expert and non-expert participants. A focus group revealed experts’ expectations for the proposed system, while a usability study assessed the clarity of the auditory feedback to everyday users. The evaluation shows promising results regarding the usefulness of our system in this application area.

In their daily lives, people use more than one sense to perceive and interpret their environment. Likewise, audio-visual interfaces can support human data analysts better than interfaces relying on just one sense. While the research communities of sonification and visualization have both carried out extensive research on the auditory and visual representation of data, comparatively little is known about their systematic and complementary combination for data analysis. After two workshops at Audio Mostly 2021 and IEEE VIS, this 3rd workshop on audio-visual analytics continues building a community of researchers interested in combining visualization and sonification.

Abstract We describe two proof-of-concept approaches on the sonification of estimated operation states and conditions focusing on two scenarios: a laboratory setup of a manipulated 3D printer and an industrial setup focusing on the operations of a punching machine. The results of these studies form the basis for the development of an “intelligent” noise protection headphone as part of Cyber Physical Production Systems which provides auditorily augmented information to machine operators and enables radio communication between them. Further application areas are implementations in control rooms (equipped with multi-channel loudspeaker systems) and utilization for training purposes. As a first proof-of-concept, the data stream of error probability estimations regarding partly manipulated 3D printing processes were mapped to three sonification models, providing evidence about momentary operation states. The neural network applied indicates a high accuracy (> 93%) of the error estimation distinguishing between normal and manipulated operation states. None of the manipulated states could be identified by listening. An auditory augmentation, or sonification of these error estimations, provides a considerable benefit to process monitoring. For a second proof-of-concept, setup operations of a punching machine were recorded. Since all operations were apparently flawlessly executed, and there were no errors to be reported, we focused on the identification of operation phases. Each phase of a punching process could be algorithmically distinguished at an estimated probability rate of > 94%. In the auditory display, these phases were represented by different instrumentations of a musical piece in order to allow users to differentiate between operations auditorily.

Air traffic control officers (ATCOs) are specialized workers responsible to monitor and guide airplanes in their assigned airspace. Such a task is highly visual and mainly supported by 2D visualizations. In this paper, we designed and assessed an application for visualizing air traffic in both orthographic (2D) and perspective (3D) views. A user study was then performed to compare these two types of representations in terms of situation awareness, workload, performance, and user acceptance. Results show that the 3D view yielded both higher situation awareness and less workload than the 2D view condition. However, such a performance does not match the opinion of the ATCOs about the 3D representation.