Networked Music Performance (NMP) is one of the most challenging real-time applications in which musicians can play together using the internet without being physically together. Nowadays, the connection technology can have performance that guarantees an adequate Quality of Experience (QoE) for NMP. On the other hand, there are millions of musicians who live in remote and rural areas without access to high-speed network connections. The promising LEO satellite internet technology could fill this gap and democratize NMP in remote places. This paper aims to provide an analysis of Starlink’s capability to meet the stringent requirements of NMP. We present an analysis of network metrics that are relevant to NMP applications and assess the NMP software JackTrip in order to test a real NMP application with LEO satellite internet. The evaluation of the RTT, One-way delay, Packet loss, and jitter in different scenarios poses some challenges for the NMP scenario. We present the results of two live NMP jamming sessions with musicians, proposing an adapted QoE model that incorporates both objective and subjective metrics. With a tailored buffer strategy and the ability of a musician, we were able to obtain an adequate QoE, overcoming the challenges introduced by LEO satellite internet.

We propose a learnable content adaptive front end for audio signal processing. Before the modern advent of deep learning, we used fixed representation non-learnable front-ends like spectrogram or mel-spectrogram with/without neural architectures. With convolutional architectures supporting various applications such as ASR and acoustic scene understanding, a shift to a learnable front ends occurred in which both the type of basis functions and the weight were learned from scratch and optimized for the particular task of interest. With the shift to transformer-based architectures with no convolutional blocks present, a linear layer projects small waveform patches onto a small latent dimension before feeding them to a transformer architecture. In this work, we propose a way of computing a content-adaptive learnable time-frequency representation. We pass each audio signal through a bank of convolutional filters, each giving a fixed-dimensional vector. It is akin to learning a bank of finite impulse-response filterbanks and passing the input signal through the optimum filter bank depending on the content of the input signal. A content-adaptive learnable time-frequency representation may be more broadly applicable, beyond the experiments in this paper.

This paper proposes a novel way of doing audio synthesis at the waveform level using Transformer architectures. We propose a deep neural network for generating waveforms, similar to wavenet [1]. This is fully probabilistic, auto-regressive, and causal, i.e. each sample generated depends on only the previously observed samples. Our approach outperforms a widely used wavenet architecture by up to 9% on a similar dataset for predicting the next step. Using the attention mechanism, we enable the architecture to learn which audio samples are important for the prediction of the future sample. We show how causal transformer generative models can be used for raw waveform synthesis. We also show that this performance can be improved by another 2% by conditioning samples over a wider context. The flexibility of the current model to synthesize audio from latent representations suggests a large number of potential applications. The novel approach of using generative transformer architectures for raw audio synthesis is, however, still far away from generating any meaningful music similar to wavenet, without using latent codes/meta-data to aid the generation process.

Dancing and playing music require people to coordinate actions with auditory rhythms. In laboratory perception-action coordination tasks, people are asked to synchronize taps with a metronome. When synchronizing with a metronome, people tend to anticipate stimulus onsets, tapping slightly before the stimulus. The anticipation tendency increases with longer stimulus periods of up to 3500ms, but is less pronounced in trained individuals like musicians compared to non-musicians. Furthermore, external factors influence the timing of tapping. These factors include the presence of auditory feedback from one's own taps, the presence of a partner performing coordinated joint tapping, and transmission latencies (TLs) between coordinating partners. Phenomena like the anticipation tendency can be explained by delay-coupled systems, which may be inherent to the sensorimotor system during perception-action coordination. Here we tested whether a dynamical systems model based on this hypothesis reproduces observed patterns of human synchronization. We simulated behavior with a model consisting of an oscillator receiving its own delayed activity as input. Three simulation experiments were conducted using previously-published behavioral data from 1) simple tapping, 2) two-person alternating beat-tapping, and 3) two-person alternating rhythm-clapping in the presence of a range of constant auditory TLs. In Experiment 1, our model replicated the larger anticipation observed for longer stimulus intervals and adjusting the amplitude of the delayed feedback reproduced the difference between musicians and non-musicians. In Experiment 2, by connecting two models we replicated the smaller anticipation observed in human joint tapping with bi-directional auditory feedback compared to joint tapping without feedback. In Experiment 3, we varied TLs between two models alternately receiving signals from one another. Results showed reciprocal lags at points of alternation, consistent with behavioral patterns. Overall, our model explains various anticipatory behaviors, and has potential to inform theories of adaptive human synchronization.

Lack of trust can arise when people do not know what autonomous vehicles perceive in the environment. To convey this information without causing alarm or compelling people to act, we designed and evaluated a way to sonify an autonomous vehicle's perception of salient driving events using abstract auditory icons, or "earcons." These are localized in space using an in-car quadraphonic speaker array to correspond with the direction of events. We describe the interaction design for these awareness cues and a validation experiment (N=28) examining the effects of sonified events on drivers' sense of situation awareness, comfort, and trust. Overall, this work suggests that our designed earcons do improve people's awareness of in-simulation events. The effect of the increased situational awareness on trust and comfort is inconclusive. However, post-study design feedback suggests that sounds should have low levels of intensity and dissonance, and a sense of belonging to a common family.

In this paper, an installation is presented that connects Stanford and Stockholm through a one-of-a- kind combination of instrument and venue: the Skandia Wurlitzer theatre organ (Wurlitzer serial no. 1254) situated in the KTH R1 Experimental Performance Space, a disused nuclear reactor. A continuous stream of musical data, audio, and video between the two places explored the capabilities of the digital to play with the concepts of presence and embodiment, virtuality and the physical.

Distributed physical models of musical instruments have been used to acoustically “ping” Internet connections between two network hosts. Sound waves propagated through Internet acoustics behave just as in air, water or along a stretched string. In this case, a musical synthesis technique creates waves on the Internet path between two hosts. When waves recirculate between two endpoints, a musical tone is created if the round trip travel time lies within the range of our pitch sense (roughly 250µs to 50ms). The resulting tones provide a quick and intuitive evaluation of quality of service (QoS), displaying its significant aspects including latency, jitter and packet loss. Stable, clear tones with high pitch indicate good end-to-end capabilities that a path must support for immersive, real-time applications. A “network harp” recently demonstrated 320 synthesized strings oscillating across the Western half of the Internet2 Abilene Network.

A theory of bow and breath noise generation has been tested by analyzing recorded cello tones and by simulation using physical models of the cello and clarinet. For the synthesis to be successful, the listener's perceptions of noise and sound must fuse. Evidence is presented that the noise must be pulse modulated in a period synchronous way, as is shown for voiced fricatives. A method is described which is practical for enhancing naturalness of synthesis from physical models. No additional control parameters are required. Changes in the noise sound follow in a predictable way control changes in bowing, breath, and embouchure parameters in the simulations.>

Nearly periodic waveforms produced by musical instruments fluctuate from period to period both in the length of the period and in the shape of the waveform of each period. A measure of these fluctuations that is sensitive to correlations in waveshapes for medium time scales (up to 16 in the present implementation) is described. The method also produces two-dimensional color or gray-scale plots that allow visualization of the complexity of waveforms produced by acoustic or electronic instruments over hundreds of periods of the waveform.>

Experiments in automatic music recognition at CCRMA have been in progress for five years. Digitized sound recordings of instrumental music are analyzed and transcribed by computer. The current effort is directed at polyphonic examples with a variety of instruments and musical styles. The paper discusses acoustic analysis issues in accurately transcribing polyphonic input. The overall goal of the work is to provide a tool for the study of musical performance, for applications requiring tracking of live musicians, for manuscript work and for segmentation of digital audio recordings.