Abstract The research communities studying visualization and sonification for data display and analysis share exceptionally similar goals, essentially making data of any kind interpretable to humans. One community does so by using visual representations of data, and the other community employs auditory (non‐speech) representations of data. While the two communities have a lot in common, they developed mostly in parallel over the course of the last few decades. With this STAR, we discuss a collection of work that bridges the borders of the two communities, hence a collection of work that aims to integrate the two techniques into one form of audiovisual display, which we argue to be “more than the sum of the two.” We introduce and motivate a classification system applicable to such audiovisual displays and categorize a corpus of 57 academic publications that appeared between 2011 and 2023 in categories such as reading level, dataset type, or evaluation system, to mention a few. The corpus also enables a meta‐analysis of the field, including regularly occurring design patterns such as type of visualization and sonification techniques, or the use of visual and auditory channels, showing an overall diverse field with different designs. An analysis of a co‐author network of the field shows individual teams without many interconnections. The body of work covered in this STAR also relates to three adjacent topics: audiovisual monitoring, accessibility, and audiovisual data art. These three topics are discussed individually in addition to the systematically conducted part of this research. The findings of this report may be used by researchers from both fields to understand the potentials and challenges of such integrated designs while hopefully inspiring them to collaborate with experts from the respective other field.

Abstract One of the commonly used visualization techniques for multivariate data is the parallel coordinates plot. It provides users with a visual overview of multivariate data and the possibility to interactively explore it. While pattern recognition is a strength of the human visual system, it is also a strength of the auditory system. Inspired by the integration of the visual and auditory perception in everyday life, we introduce an audio-visual analytics design named Parallel Chords combining both visual and auditory displays. Parallel Chords lets users explore multivariate data using both visualization and sonification through the interaction with the axes of a parallel coordinates plot. To illustrate the potential of the design, we present (1) prototypical data patterns where the sonification helps with the identification of correlations, clusters, and outliers, (2) a usage scenario showing the sonification of data from non-adjacent axes, and (3) a controlled experiment on the sensitivity thresholds of participants when distinguishing the strength of correlations. During this controlled experiment, 35 participants used three different display types, the visualization, the sonification, and the combination of these, to identify the strongest out of three correlations. The results show that all three display types enabled the participants to identify the strongest correlation — with visualization resulting in the best sensitivity. The sonification resulted in sensitivities that were independent from the type of displayed correlation, and the combination resulted in increased enjoyability during usage.

Abstract Both sonification and visualization convey information about data by effectively using our human perceptual system, but their ways to transform the data differ. Over the past 30 years, the sonification community has demanded a holistic perspective on data representation, including audio-visual analysis, several times. A design theory of audio-visual analysis would be a relevant step in this direction. An indispensable foundation for this endeavor is a terminology describing the combined design space. To build a bridge between the domains, we adopt three of the established theoretical constructs from visualization theory for the field of sonification. The three constructs are the spatial substrate, the visual mark, and the visual channel. In our model, we choose time to be the temporal substrate of sonification. Auditory marks are then positioned in time, such as visual marks are positioned in space. Auditory channels are encoded into auditory marks to convey information. The proposed definitions allow discussing visualization and sonification designs as well as multi-modal designs based on a common terminology. While the identified terminology can support audio-visual analytics research, it also provides a new perspective on sonification theory itself.

In their daily lives, people use more than one sense to perceive and interpret their environment. Likewise, audio-visual interfaces can support human data analysts better than interfaces relying on just one sense. While the research communities of sonification and visualization have both carried out extensive research on the auditory and visual representation of data, comparatively little is known about their systematic and complementary combination for data analysis. After two workshops at Audio Mostly 2021 and IEEE VIS, this 3rd workshop on audio-visual analytics continues building a community of researchers interested in combining visualization and sonification.

Digital active noise control (ANC) for headphones usually has to predict the noise because of the latency of common audio converters. In adaptive feedback ANC, the prediction is based on the noise that entered the headphone. This noise is low‐pass filtered because of the physical barrier of the ear cups. In this study, this low‐pass characteristic is exploited to define a prediction filter which does not require real‐time updates. For broadband noises, the prediction filter performs better than adaptive prediction methods like the least mean squares algorithm or iterated one‐step predictions in the relevant frequency band. This is shown in simulations as well as in measurements. In addition, the authors show that their prediction filter is more robust against changes in the acoustics of the headphone.

This paper proposes an approach for cross-modal user interface design using interaction patterns. A meta-domain is introduced in which the user interface can be designed without the determination of its realisation in a particular interaction domain. When fully described in the meta-domain, the user interface can be realised in different domains (visual, auditory, tactile etc) exploiting the strengths of each different mode to its maximum. The paper proposes a set of interaction patterns along with visual and auditory realisations. To prove the concept, a prototype was designed for a real-world application, Microsoft Explorer. An auditory display prototype was developed making extensive use of virtual audio environments for non-visual visualisation of objects in the Explorer interface, taking advantage of a spatial layout. The prototype was evaluated with visually impaired, blind and sighted computer users.