Current music emotion recognition (MER) systems rely on emotion data averaged across listeners and over time to infer the emotion expressed by a musical piece, often neglecting time- and listener-dependent factors. These limitations can restrict the efficacy of MER systems and cause misjudgements. We present two exploratory studies on music emotion perception. First, in a live music concert setting, fifteen audience members annotated perceived emotion in the valence-arousal space over time using a mobile application. Analyses of inter-rater reliability yielded widely varying levels of agreement in the perceived emotions. A follow-up lab-based study to uncover the reasons for such variability was conducted, where twenty-one participants annotated their perceived emotions whilst viewing and listening to a video recording of the original performance and offered open-ended explanations. Thematic analysis revealed salient features and interpretations that help describe the cognitive processes underlying music emotion perception. Some of the results confirm known findings of music perception and MER studies. Novel findings highlight the importance of less frequently discussed musical attributes, such as musical structure, performer expression, and stage setting, as perceived across audio and visual modalities. Musicians are found to attribute emotion change to musical harmony, structure, and performance technique more than non-musicians. We suggest that accounting for such listener-informed music features can benefit MER in helping to address variability in emotion perception by providing reasons for listener similarities and idiosyncrasies.

The mobile music laboratory is an installation for interactive music listening and research on embodied cognition, with focus on contemporary classical music (CCM). CCM is often less familiar to listeners than other types of music, but its success is essential for composers pioneering musical and cultural developments. This installation gives listeners of any background a way to actively engage with CCM. It enables users to interactively control the temporal expression of CCM through intuitive conducting-tapping gestures while traces of their cognitive and physiological processes are recorded for subsequent analysis. The mobile music laboratory system is comprised of a real-time controller with MIDI playback and a data acquisition system. As the user controls the music, Lab Streaming Layer synchronously records gesture timing and audio as well as neural and cardiac signals. The installation is designed to engender purposeful music engagement with CCM while simultaneously serving as a mobile laboratory for studying embodied CCM listening experiences, thus enabling ecologically valid music cognition research outside of auditory psychology labs.

Playing techniques contain distinctive information about musical expressivity and interpretation. Yet, current research in music signal analysis suffers from a scarcity of computational models for playing techniques, especially in the context of live performance. To address this problem, our paper develops a general framework for playing technique recognition. We propose the adaptive scattering transform, which refers to any scattering transform that includes a stage of data-driven dimensionality reduction over at least one of its wavelet variables, for representing playing techniques. Two adaptive scattering features are presented: frequency-adaptive scattering and direction-adaptive scattering. We analyse seven playing techniques: vibrato, tremolo, trill, flutter-tongue, acciaccatura, portamento, and glissando. To evaluate the proposed methodology, we create a new dataset containing full-length Chinese bamboo flute performances (CBFdataset) with expert playing technique annotations. Once trained on the proposed scattering representations, a support vector classifier achieves state-of-the-art results. We provide explanatory visualisations of scattering coefficients for each technique and verify the system over three additional datasets with various instrumental and vocal techniques: VPset, SOL, and VocalSet.

Playing techniques are important expressive elements in music signals. In this paper, we propose a recognition system based on the joint time-frequency scattering transform (jTFST) for pitch evolution-based playing techniques (PETs), a group of playing techniques with monotonic pitch changes over time. The jTFST represents spectro-temporal patterns in the time-frequency domain, capturing discriminative information of PETs. As a case study, we analyse three commonly used PETs of the Chinese bamboo flute: acciacatura, portamento, and glissando, and encode their characteristics using the jTFST. To verify the proposed approach, we create a new dataset, the CBF-petsDB, containing PETs played in isolation as well as in the context of whole pieces performed and annotated by professional players. Feeding the jTFST to a machine learning classifier, we obtain F-measures of 71% for acciacatura, 59% for portamento, and 83% for glissando detection, and provide explanatory visualisations of scattering coefficients for each technique.

Automatic music generation systems have gained in popularity and sophistication as advances in cloud computing have enabled large-scale complex computations such as deep models and optimization algorithms on personal devices. Yet, they still face an important challenge, that of long-term structure, which is key to conveying a sense of musical coherence. We present the MorpheuS music generation system designed to tackle this problem. MorpheuS' novel framework has the ability to generate polyphonic pieces with a given tension profile and long- and short-term repeated pattern structures. A mathematical model for tonal tension quantifies the tension profile and state-of-the-art pattern detection algorithms extract repeated patterns in a template piece. An efficient optimization metaheuristic, variable neighborhood search, generates music by assigning pitches that best fit the prescribed tension profile to the template rhythm while hard constraining long-term structure through the detected patterns. This ability to generate affective music with specific tension profile and long-term structure is particularly useful in a game or film music context. Music generated by the MorpheuS system has been performed live in concerts.

Transcribing the singing voice into music notes is challenging due to pitch fluctuations such as portamenti and vibratos. This paper presents a probabilistic transcription method for monophonic sung melodies that explicitly accounts for these local pitch fluctuations. In the hierarchical Hidden Markov Model (HMM), an upper-level ergodic HMM handles the transitions between notes, and a lower-level left-to-right HMM handles the intra- and inter-note pitch fluctuations. The lower-level HMM employs the pitch dynamic model, which explicitly expresses the pitch curve characteristics as the observation likelihood over f0and Δf0using a compact parametric distribution. A histogram-based tuning frequency estimation method, and some post-processing heuristics to separate merged notes and to allocate spuriously detected short notes, improve the note recognition performance. With model parameters that support intuitions about singing behavior, the proposed method obtained encouraging results when evaluated on a published monophonic sung melody dataset, and compared with state-of-the-art methods.

Keys provide musical context and key modulation (changes) forms a crucial feature of music. In the age of big music data collections, automatic key segmentation is an important step towards music indexing and structure analysis. When using template-based key-finding methods, the best segmentation must minimize intra-segment distance to keys while maximizing inter-segment distance for neighboring keys. We present a general dynamic programming (DP) solution to this segmentation problem that is applicable to all distance-based key-finding methods and that does not require the number of segments to be pre-defined. This meta-algorithm is applied to the Kostka-Payne and Beatles datasets with three widely used distance-based key-finding methods. The key-finding results are evaluated using a compound score, and precision and recall. Statistical analysis of the results show that a precision value of 0.9 can be achieved with both datasets, for excerpts in one key, an average compound score above 0.8 is reported.

Data mining tasks such as music indexing, information retrieval, and similarity search, require an understanding of how listeners process music internally. Many algorithms for automatically analyzing the structure of recorded music assume that a large change in one or another musical feature suggests a section boundary. However, this assumption has not been tested: while our understanding of how listeners segment melodies has advanced greatly in the past decades, little is known about how this process works with more complex, full-textured pieces of music, or how stable this process is across genres. Knowing how these factors affect how boundaries are perceived will help researchers to judge the viability of algorithmic approaches with different corpora of music. We present a statistical analysis of a large corpus of recordings whose formal structure was annotated by expert listeners. We find that the acoustic properties of boundaries in these recordings corroborate findings of previous perceptual experiments. Nearly all boundaries correspond to peaks in novelty functions, which measure the rate of change of a musical feature at a particular time scale. Moreover, most of these boundaries match peaks in novelty for several features at several time scales. We observe that the boundary-novelty relationship can vary with listener, time scale, genre, and musical feature. Finally, we show that a boundary profile derived from a collection of novelty functions correlates with the estimated salience of boundaries indicated by listeners.

To identify repeated patterns and contrasting sections in music, it is common to use self-similarity matrices (SSMs) to visualize and estimate structure. We introduce a novel application for SSMs derived from audio recordings: using them to learn about the potential reasoning behind a listener's annotation. We use SSMs generated by musically-motivated audio features at various timescales to represent contributions to a structural annotation. Since a listener's attention can shift among musical features (e.g., rhythm, timbre, and harmony) throughout a piece, we further break down the SSMs into section-wise components and use quadratic programming (QP) to minimize the distance between a linear sum of these components and the annotated description. We posit that the optimal section-wise weights on the feature components may indicate the features to which a listener attended when annotating a piece, and thus may help us to understand why two listeners disagreed about a piece's structure. We discuss some examples that substantiate the claim that feature relevance varies throughout a piece, using our method to investigate differences between listeners' interpretations, and lastly propose some variations on our method.

We present Mimi4x, an interactive audio-visual installation that engages members of the general public in high-level structural decision-making in musical improvisation. Through a MIDI controller, the user manages four instances of the Mimi 1.5 improvisation system. The visuals in each Mimi system animate the generation of new music, depicting the recombination of musical material, and the flow of generated music in time. A composer prepares original music material for each Mimi to use in its improvisation. Through Mimi4x, the user is directly involved in the improvisation/composition process, and takes on some of the role of the composer. The user chooses which oracles will perform and when, and can adjust the recombination rate for, and playback volume of, each oracle during performance. This simple method of interaction is accessible to all users regardless of their musical background, yet gives the user significant and satisfying influence over the music generated.

Robust data retrieval in the presence of uncertainty is a challenging problem in multimedia information retrieval. In query-by-humming (QBH) systems, uncertainty can arise in query formulation due to user-dependent variability, such as incorrectly hummed notes, and in query transcription due to machine-based errors, such as insertions and deletions. We propose a fingerprinting (FP) algorithm for representing salient melodic information so as to better compare potentially noisy voice queries with target melodies in a database. The FP technique is employed in the QBH system back end; a hidden Markov model (HMM) front end segments and transcribes the hummed audio input into a symbolic representation. The performance of the FP search algorithm is compared to the conventional edit distance (ED) technique. Our retrieval database is built on 1500 MIDI files and evaluated using 400 hummed samples from 80 people with different musical backgrounds. A melody retrieval accuracy of 88% is demonstrated for humming samples from musically trained subjects, and 70% for samples from untrained subjects, for the FP algorithm. In contrast, the widely used ED method achieves 86% and 62% accuracy rates, respectively, for the same samples, thus suggesting that the proposed FP technique is more robust under uncertainty, particularly for queries by musically untrained users.

An increasing number of novel applications produce a rich set of different data types that need to be managed efficiently and coherently. In this article we present our experience with designing and implementing a data management infrastructure for a distributed immersive performance (DIP) application. The DIP project investigates a versatile framework for the capture, recording, and replay of video, audio, and MIDI (Musical Instrument Digital Interface) streams in an interactive environment for collaborative music performance. We are focusing on two classes of data streams that are generated within this environment. The first category consists of high-resolution isochronous media streams, namely audio and video. The second class comprises MIDI data produced by electronic instruments. MIDI event sequences are alphanumeric in nature and fall into the category of the data streams that have been of interest to data management researchers in recent years. We present our data management architecture, which provides a repository for all DIP data. Streams of both categories need to be acquired, transmitted, stored, and replayed in real time. Data items are correlated across different streams with temporal indices. The audio and video streams are managed in our own High-performance Data Recording Architecture (HYDRA), which integrates multistream recording and retrieval in a consistent manner. This paper reports on the practical issues and challenges that we encountered during the design, implementation and experimental phases of our prototype. We also present some analysis results and discuss future extensions for the architecture.

In this article, we propose a solution to the problem of query by example for polyphonic music audio. We first present a generic mid-level representation for audio queries. Unlike previous efforts in the literature, the proposed representation is not dependent on the different spectral characteristics of different musical instruments and the accurate location of note onsets and offsets. This is achieved by first mapping the short term frequency spectrum of consecutive audio frames to the musical space (the spiral array) and defining a tonal identity with respect to center of effect that is generated by the spectral weights of the musical notes. We then use the resulting single dimensional text representations of the audio to create a-gram statistical sequence models to track the tonal characteristics and the behavior of the pieces. After performing appropriate smoothing, we build a collection of melodic n-gram models for testing. Using perplexity-based scoring, we test the likelihood of a sequence of lexical chords (an audio query) given each model in the database collection. Initial results show that, some variations of the input piece appears in the top 5 results 81% of the time for whole melody inputs within a 500 polyphonic melody database. We also tested the retrieval engine for small audio clips. Using 25s segments, variations of the input piece are among the top 5 results 75% of the time.

In this paper, we explore the effect of musical context on audio onset detection using machine learning techniques. We extract the signal intensity and frequency energy of audio as the attributes of input instances for the machine learning techniques. The audio is synthesized from MIDI files, providing exact information of onset events. We test three state-of-the-art machine learning algorithms, support vector machines (SVM), neural networks (NN), and Naive Bayes (NB) with Ada boosting, for learning and classifying audio onsets. We found that SVMs perform best in general, based on the average of training and 10-fold cross validation errors as the evaluation criterion. We then test the SVM and NN, the two best performing methods, on Bach's Prelude in C major BWV 943, transposed to different keys and time-stretched to various tempi. The error rates ranged from 23.91% (when training set key and tempo equals those of the test set) to 37.22% (when the key is off by four accidentals) and 37.91% (when tempo is 20 beats per minute faster). The results show that audio onset detection performs significantly better when the key and tempo attributes of the test and training sets concur, than when they are different, thus supporting the utility of tempo and key knowledge in designing onset detection systems, or in prescribing confidence statistics to onset detection outcomes

This paper presents the spiral array model and its associated algorithms for tonal induction, approximation, and segmentation. The spiral array is a geometric model for tonality that clusters perceptually similar tonal entities. The model summarizes music information as interior points inside an array of spirals. Distances in the spiral array space are used to quantify tonal similarity. The paper traces the evolution, and presents general forms, of the existing algorithms for key finding, pitch spelling, and segmentation, and proposes a new O(n) algorithm, Argus, for tonal segmentation. The proposed algorithm computes a value that quantifies the discrepancy between the local contexts in the future and past at each point in time. Discrepancy values exceeding control thresholds are shown to mark the segmentation boundaries of the test set that concur with expert analyses. A number of window sizes and threshold settings are investigated. The algorithm is demonstrated using Edward MacDowell’s To A Wild Rose and tested on Franz Schubert’s Allegretto from Moment Musical D780 No. 6 and Thema from Impromptu D935 No. 4. The algorithm accurately locates tonal boundaries in all three case studies.

Key finding is an integral step in content-based music indexing and retrieval. In this paper, we present an O(n) real-time algorithm for determining key from polyphonic audio. We use the standard Fast Fourier Transform with a local maximum detection scheme to extract pitches and pitch strengths from polyphonic audio. Next, we use Chew's Spiral Array Center of Effect Generator (CEG) algorithm to determine the key from pitch strength information. We test the proposed system using Mozart's Symphonies. The test data is audio generated from MIDI source. The algorithm achieves a maximum correct key recognition rate of 96% within the first fifteen seconds, and exceeds 90% within the first three seconds. Starting from the extracted pitch strength information, we compare the CEG algorithm's performance to the classic Krumhansl-Schmuckler (K-S) probe tone profile method and Temperley's modified version of the K-S method. Correct key recognition rates for the K-S and modified K-S methods remain under 50% in the first three seconds, with maximum values of 80% and 87% respectively within the first fifteen seconds for the same test set. The CEG method consistently scores higher throughout the fifteen-second selections.

This paper presents a pitch-based approach for creating musical fingerprints for similarity assessment. An effective measure for musical similarity impacts music indexing and classification in music retrieval systems. The proposed method creates key distributions from polyphonic music, and compares the key distributions of pairs of pieces, by calculating their correlation coefficient, to determine a degree of similarity between them. The proposed method assumes no knowledge of the time structure of the piece, nor does it require pieces to be the same length. We present results using this method to assess similarity among selected variations by Mozart. The results show that the correlation coefficients of pieces from the same set of variations are centered on 0.88 (with a standard deviation of 0.11), and that of pieces across different sets of variations are centered on 0.32 (with a standard deviation of 0.31).

We present a method for segmenting music with different grid levels in order to properly quantize note values in the transcription of music. This method can be used in automatic music transcription systems and music information retrieval systems to reduce a performance of a music piece to the printed or digital score. The system takes only the onset data of performed music from either MIDI or audio, and determine the best maximal grid level onto which to fit the note onsets. This maximal grid level, or tatum, is allowed to vary from section to section in a piece. We obtain the optimal segmentation of the piece using dynamic programming. We present results from an audio based performance of Milhaud's Botafogo, as well as several MIDI performances of the Rondo-Allegro from Beethoven's Pathetique. The results show a reduction of error compared to quantization based only on one global metric level, and promises to create rhythm transcriptions that are parsimonious and readable.

We present MuSA.RT, Opus 1, a multimodal interactive system for music analysis and visualization using the Spiral Array model. Real-time MIDI input from a live performance is processed, analyzed and mapped to the 3D model, revealing tonal structures such as pitches, chords and keys. A user can concurrently navigate through the Spiral Array space using a gamepad or set the camera control to automatic pilot. The interaction among and concurrent processing of the different data streams is made possible through the Modular Flow Scheduling Middleware.