We release synth1B1, a multi-modal audio corpus consisting of 1 billion 4-second synthesized sounds, paired with the synthesis parameters used to generate them. The dataset is 100x larger than any audio dataset in the literature. We also introduce torchsynth, an open source modular synthesizer that generates the synth 1B1 samples on-the-fly at 16200x faster than real-time (714MHz) on a single GPU. Finally, we release two new audio datasets: FM synth timbre and subtractive synth pitch. Using these datasets, we demonstrate new rank-based evaluation criteria for existing audio representations. Finally, we propose a novel approach to synthesizer hyperparameter optimization.

The two commonly accepted models of affect used in affective computing are categorical and two-dimensional. However, categorical models are limited to datasets that only contain music for which human annotators fully agree upon, while two-dimensional models use descriptors to which users may not relate to (e.g., Valence and Arousal). This paper explores the hypothesis that the music emotion problem is circular, and shows how circular models can be used for automatic music emotion recognition. This hypothesis is tested through experiments on the two commonly accepted models of affect, as well as on an original circular model proposed by the authors. First, an original dataset was assembled and annotated as a way to investigate agreement among annotators. Then, polygonal approximations of circular regression are proposed as a practical method to investigate whether the circularity of the annotations can be exploited. Experiments with different polygons demonstrate consistent improvements over the categorical model on a dataset containing musical extracts for which the human annotators did not fully agree upon. Finally, a proposed multi-tagging strategy based on the circular predictions is put forward as a pragmatic method to automatically annotate music based on the circular models.

We propose to pre-stack a U-Net as a way of improving the polyphonic music transcription performance of various baseline Convolutional Neural Networks (CNNS). The U-Net, a network architecture based on skip-connections between layers acts as a transformation network followed by a transcription network. Notably, we do not introduce any additional loss terms specific to the transformation network, but instead, jointly train the entire combined model with the original loss function that was designed for the back-end transcription network. We argue that this U-Net network transforms the input signal into a representation that is more effective for transcription, and thus enables the observed improvements in accuracy. We empirically confirm with several experiments using the MusicNet dataset, that the proposed configuration consistently improves the accuracy of transcription networks. This enhancement cannot be achieved by simply introducing more neurons or more layers to the baseline CNNs. Moreover, we show that using the proposed architecture we can go beyond general music transcription and perform transcription in an instrument-specific fashion. By doing so, the original general transcription performance is also increased.

We introduce a data-driven approach to automatic pitch correction of solo singing performances. The proposed approach predicts note-wise pitch shifts from the relationship between the respective spectrograms of the singing and accompaniment. This approach differs from commercial systems, where vocal track notes are usually shifted to be centered around pitches in a user-defined score, or mapped to the closest pitch among the twelve equal-tempered scale degrees. The proposed system treats pitch as a continuous value rather than relying on a set of discretized notes found in musical scores, thus allowing for improvisation and harmonization in the singing performance. We train our neural network model using a dataset of 4,702 amateur karaoke performances selected for good intonation. Our model is trained on both incorrect intonation, for which it learns a correction, and intentional pitch variation, which it learns to preserve. The proposed deep neural network with gated recurrent units on top of convolutional layers shows promising performance on the real-world score-free singing pitch correction task-autotuning.

We introduce the "Intonation" dataset of amateur vocal performances with a tendency for good intonation, collected from Smule, Inc. The dataset can be used for music information retrieval tasks such as autotuning, query by humming, and singing style analysis. It is available upon request on the Stanford CCRMA DAMP website.1We describe a semi-supervised approach to selecting the audio recordings from a larger collection of performances based on intonation patterns. The approach can be applied in other situations where a researcher needs to extract a subset of data samples from a large database. A comparison of the "Intonation" dataset and the remaining collection of performances shows that the two have different intonation behavior distributions.

We present a method for compilation of multi-dimensional stream processing programs from affine recurrence equations with unbounded domains into imperative code with statically allocated memory. The method involves a novel polyhedral schedule transformation called periodic tiling. It accommodates existing polyhedral optimizations to improve memory access patterns and expose parallelism. This enables efficient execution of programming languages with unbounded recurrence equations, as well as optimization of existing languages from which this form can be derived. The method is experimentally evaluated on 5 DSP algorithms with large problem sizes. Results show potential for improved throughput compared to hand-optimized C++ (speedups on a 6-core Intel Xeon CPU up to 10× with a geometric mean 3.3×). 1

Investigating singing style is a difficult problem as individual styles are intertwined with melodies from different songs. In this paper, a methodology to investigate singing style is proposed. The proposed approach utilizes convolutional neural networks in a siamese architecture. In addition, we investigate variants of the networks to improve the audio feature extraction process. The potential of the proposed method for analyzing singing style is demonstrated using experiments on pop music singing recordings. The results indicate that the use of the proposed method is indeed effective in learning audio features that are relevant for characterizing singing style.

In this paper, we propose a novel real-time sound event detection framework, which combines multi-label learning and edge computing, to classify and localize abnormal sound events for city surveillance. Multiple devices equipped with acoustic sensors are deployed to collect the audio information. A learning-based approach is introduced to address the difficulties of accurately classifying the temporally overlapping acoustic events in a noisy environment. Then, edge computing is adopted to handle the high processing complexity of the learned analytics. Computation-intensive tasks of classification and localization can be offloaded to the nearby edge server for low-latency sound detection. An ensemble-based cooperative decision-making algorithm is also presented to aggregate the information from distributed devices in order to obtain better classification results. Extensive evaluations show the effectiveness of edge computing which helps reduce the time latency as well as the superiority of cooperative post-processing on the edge server to obtain a high accuracy.

Semantic vectors, or language embeddings, are used in computational linguistics to represent language for a variety of machine related tasks including translation, speech to text, and natural language understanding. These semantic vectors have also been extensively studied in correlation with human brain data, showing evidence that the representation of language in the human brain can be modeled through these vectors with high correlation. Further, various attempts have been made to study how the human brain represents and understands music. For example, it has been shown that EEG data of subjects listening to music can be used for tempo detection and singer gender recognition. We propose studying the relationship between the EEG data of subjects listening to audio and the audio feature vectors modeled after the semantic vectors in computational linguistics. This could provide new insight into how the brain processes and understands music.

We propose an optimized framework for binary image processing, characterized by a highly bit-packed representation of pixels and their square neighbourhood. The Super-Packed (SPmat) representation for binary images enables the easy use of bit-wise computations for developing fast processing algorithms, such as: morphology, contours, run-length, and thinning, in a unified framework. With several experiments, we show that the aforementioned algorithms can be consistently sped-up, and outperform by a large margin available software implementations. The software package is freely available on github at the url https://github.com/fpeder/spmat to support reproducibility.

Automatic music transcription can be defined as the analysis of the acoustic signal to extract a symbolic representation of music. Existing transcription systems typically consider just the notes played at a given moment; however, other aspects such as expressiveness and playing technique can also be considered. This work is focused on how chords are played. Specifically, we consider a special type of chords, those played in arpeggio style, or simply arpeggios, in which the notes are played fast, sequentially from the lowest to the highest pitched note or vice versa and with a large overlap of the notes' sound. The main goal of this paper is to determine the pitch direction in which the arpeggiated chord was played. Two different classification methods are considered: a Fisher linear discriminant and an SVM linear classification scheme. Different features are presented for this task: one is based on the Mel-frequency cepstral coefficients (MFCCs) and two others, specifically designed for this task, rely on different analyses of the spectrogram. Evaluations have been done with a wide number of musical instruments. The results show that the pitch direction can be reliably detected using the proposed methods.

In this paper, we demonstrate how to use asymmetric data relabeling based on feature histograms as a pre-processing step for improving the overall classification performance of different classifiers in situations when only positive and unlabeled data is available. Additionally, this strategy can be used to identify with some level of confidence those data instances that should probably be labeled as positive. Moreover, this approach can be adapted to assess the quality of a given dataset, in terms of how many positive instances are not labeled. We examine our approach using synthetic data and demonstrate its applicability using real, publicly available data.

Analyzing and formalizing the intricate mechanisms of music is a very challenging goal for Artificial Intelligence. Dealing with real audio recordings requires the ability to handle both uncertainty and complex relational structure at multiple levels of representation. Until now, these two aspects have been generally treated separately, probability being the standard way to represent uncertainty in knowledge, while logical representation being the standard way to represent knowledge and complex relational information. Several approaches attempting a unification of logic and probability have recently been proposed. In particular, Markov logic networks (MLNs), which combine first-order logic and probabilistic graphical models, have attracted increasing attention in recent years in many domains. This paper introduces MLNs as a highly flexible and expressive formalism for the analysis of music that encompasses most of the commonly used probabilistic and logic-based models. We first review and discuss existing approaches for music analysis. We then introduce MLNs in the context of music signal processing by providing a deep understanding of how they specifically relate to traditional models, specifically hidden Markov models and conditional random fields. We then present a detailed application of MLNs for tonal harmony music analysis that illustrates the potential of this framework for music processing.

Algorithms for musical tempo estimation have become increasingly complicated in recent years. These algorithms typically utilize two fundamental properties of musical rhythm: some features of the audio signal are self-similar at periods related to the underlying rhythmic structure, and rhythmic events tend to be spaced regularly in time. We present a streamlined tempo estimation method ( stem) that distills ideas from previous work by reducing the number of steps, parameters, and modeling assumptions while retaining good accuracy. This method is designed for music with a constant or near-constant tempo. The proposed method either outperforms or has similar performance to many existing state-of-the-art algorithms. Self-similarity is captured through autocorrelation of the onset strength signal (OSS), and time regularity is captured through cross-correlation of the OSS with regularly spaced pulses. Our findings are supported by the most comprehensive evaluation of tempo estimation algorithms to date in terms of the number of datasets and tracks considered. During the process we have also corrected ground truth annotations for the datasets considered. All the data, the annotations, the evaluation code, and three different implementations (C++, Python, MATLAB) of the proposed algorithm are provided in order to support reproducibility.

The five articles in this special section focus on data mining techniques and applications in the music industry. Music has been an important application area for data mining and machine learning techniques for many years. Music data mining is an interdisciplinary area that studies computational methods for understanding and delivering music data and is a topic of growing importance with large commercial relevance and substantial potential. The research area of music data mining has gradually evolved during this time period in order to address the challenge of effectively accessing and interacting with these increasing large collections of music and associated data such as styles, artists, lyrics and music reviews. The algorithms and systems developed frequently employ sophisticated and advanced data mining and machine learning techniques in their attempt to better capture the frequently elusive relevant music information.

We propose an innovative approach for music description at several time-scales in a single unified formalism. More specifically, chord information at the analysis-frame level and global semantic structure are integrated in an elegant and flexible model. Using Markov Logic Networks (MLNs) low-level signal features are encoded with high-level information expressed by logical rules, without the need of a transcription step. Our results demonstrate the potential of MLNs for music analysis as they can express both structured relational knowledge through logic as well as uncertainty through probabilities.

Tempo estimation is a fundamental problem in music information retrieval. It also forms the basis of other types of rhythmic analysis such as beat tracking and pattern detection. There is a large body of work in tempo estimation using a variety of different approaches that differ in their accuracy as well as their complexity. Fundamentally they take advantage of two properties of musical rhythm: 1) the music signal tends to be self-similar at periodicities related to the underlying rhythmic structure, 2) rhythmic events tend to be spaced regularly in time. We propose an algorithm for tempo estimation that is based on these two properties. We have tried to reduce the number of steps, parameters and modeling assumptions while retaining good performance and causality. The proposed approach outperforms a large number of existing tempo estimation methods and has similar performance to the best-performing ones. We believe that we have conducted the most comprehensive evaluation to date of tempo induction algorithms in terms of number of datasets and tracks.

This work improves the realism of synthesis and performance of string quartet music by generating audio through physical modelling of the violins, viola, and cello. To perform music with the physical models, virtual musicians interpret the musical score and generate actions which control the physical models. The resulting audio and haptic signals are examined with support vector machines, which adjust the bowing parameters in order to establish and maintain a desirable timbre. This intelligent feedback control is trained with human input, but after the initial training is completed, the virtual musicians perform autonomously. The system can synthesize and control different instruments of the same type (e.g., multiple distinct violins) and has been tested on two distinct string quartets (total of 8 violins, 2 violas, 2 cellos). In addition to audio, the system creates a video animation of the instruments performing the sheet music.

Recent years have seen a significant increase of interest in rich multi-modal user interfaces going beyond conventional mouse/keyboard/screen interaction. The new interface technologies are broadly impacting music technology and culture. New musical interfaces use a variety of sensing (and actuating) modalities to receive and present information to users, and often require techniques from signal processing and machine learning in order to extract and fuse high level information from noisy, high dimensional signals over time. Hence they pose many interesting signal processing challenges while offering fascinating possibilities for new research. At the same time the richness of possibilities for new forms of musical interaction requires a new approach to the design of musical technologies and has implications for performance aesthetics and music pedagogy. This tutorial begins with a general and gentle introduction to the theory and practice of the design of new technologies for musical creation and performance. It continues with an overview of signal processing and machine learning methods which are needed for more advanced work in new musical interface design.

Spectrogram factorization methods such as Non-Negative Matrix Factorization (NMF) are frequently used as a way to separate individual sound sources from complex sound mixtures. More recently, they have also been used as a first stage for the automatic transcription of polyphonic music. The problem of sound source separation is different (but related) to the problem of automatic music transcription. The output of the first is the separated audio signals corresponding to each sound source, whereas the output of the second is a symbolic representation/music score that encodes the discrete pitches/notes that are played and when they are played. Many variations of factorization methods have been proposed. Two important design choices are the way spectra are represented and what distance measures are used to compare them in the optimization used for factorization. A common assumption has been that a variant that yields better signal separation will result in better automatic transcription. In this work, we investigate experimentally this question and show that this relationship is not necessarily true.

An important task in Music Information Retrieval is content-based similarity retrieval in which given a query music track, a set of tracks that are similar in terms of musical content are retrieved. A variety of audio features that attempt to model different aspects of the music have been proposed. In most cases the resulting audio feature vector used to represent each music track is high dimensional. It has been observed that high dimensional music similarity spaces exhibit some anomalies: hubs which are tracks that are similar to many other tracks, and orphans which are tracks that are not similar to most other tracks. These anomalies are an artifact of the high dimensional representation rather than actually based on the musical content. In this work we describe a distance normalization method that is shown to reduce the number of hubs and orphans. It is based on post-processing the similarity matrix that encodes the pair-wise track similarities and utilizes clustering to adapt the distance normalization to the local structure of the feature space.

The International ACM Workshop on Music Information Retrieval with User-Centered and Multimodal Strategies (MIRUM) at ACM Multimedia was proposed in order to gather experts from the Music and Multimedia Information Retrieval communities, as well as other neighboring fields, and to provide a high-profile platform for presenting current work on Music Information Retrieval with a strong focus on user-centered and multimodal approaches. Following a successful first edition at ACM Multimedia 2011, a second edition of MIRUM was held at ACM Multimedia 2012, which is the focus of this overview. After a description of the rationale and focus areas of the workshop, the accepted submissions and other program elements are summarized.

Music transcription refers to the process of detecting musical events (typically consisting of notes, starting times and durations) from an audio signal. Most existing work in automatic music transcription has focused on offline processing. In this work we describe our efforts in building a system for real time music transcription for the vibraphone. We describe experiments with three audio-based methods for music transcription that are representative of the state of the art. One method is based on multiple pitch estimation and the other two methods are based on factorization of the audio spectrogram. In addition we show how information from a video camera can be used to impose constraints on the symbol search space based on the gestures of the performer. Experimental results with various system configurations show that this multi-modal approach leads to a significant reduction of false positives and increases the overall accuracy. This improvement is observed for all three audio methods, and indicates that visual information is complimentary to the audio information in this context.

The N-Normalization is an efficient method for normalizing a given similarity computed among multimedia objects. It can be considered for clustering and kernel enhancement. However, most approaches to N-Normalization parametrize the method arbitrarily in an ad-hoc manner. In this paper, we show that the optimal parameterization is tightly related to the geometry of the problem at hand. For that purpose, we propose a method for estimating an optimal parameterization given only the associated pair-wise similarities computed from any specific dataset. This allows us to normalize the similarity in a meaningful manner. More specifically, the proposed method allows us to improve retrieval performance as well as minimize unwanted phenomena such as hubs and orphans.

The 1st International ACM Workshop on Music Information Retrieval with User-Centered and Multimodal Strategies (MIRUM) at ACM Multimedia was proposed in order to gather experts from the Music and Multimedia Information Retrieval communities, as well as other neighboring fields. The workshop aims to provide a high-profile platform for presenting current work on Music Information Retrieval, with strong focus on user-centered and multimodal approaches. These focus areas are not only relevant to the Music Information Retrieval field, but equally recognized as emerging and relevant in the Multimedia domain. This way, a cross-disciplinary dialogue on open challenges can be initiated, facilitating new bridging opportunities and increased exchanges of expertise between communities. In this summary, we provide an overview of the 1st MIRUM workshop. After a description of the rationale and focus areas of the workshop, the accepted submissions and other program elements are summarized.

The Orchive is a large audio archive of hydrophone recordings of Killer whale (Orcinus orca) vocalizations. Researchers and users from around the world can interact with the archive using a collaborative web-based annotation, visualization and retrieval interface. In addition a mobile client has been written in order to crowdsource Orca call annotation. In this paper we describe and compare different strategies for the retrieval of discrete Orca calls. In addition, the results of the automatic analysis are integrated in the user interface facilitating annotation as well as leveraging the existing annotations for supervised learning. The best strategy achieves a mean average precision of 0.77 with the first retrieved item being relevant 95% of the time in a dataset of 185 calls belonging to 4 types.

In a musical signals, the spectral and temporal contents of instruments often overlap. If the number of channels is at least the same as the number of instruments, it is possible to apply statistical tools to highlight the characteristics of each instrument, making their identification possible. However, in the underdetermined case, in which there are fewer channels than sources, the task becomes challenging. One possible way to solve this problem is to seek for regions in the time and/or frequency domains in which the content of a given instrument appears isolated. The strategy presented in this paper explores the spectral disjointness among instruments by identifying isolated partials, from which a number of features are extracted. The information contained in those features, in turn, is used to infer which instrument is more likely to have generated that partial. Hence, the only condition for the method to work is that at least one isolated partial exists for each instrument somewhere in the signal. If several isolated partials are available, the results are summarized into a single, more accurate classification. Experimental results using 25 instruments demonstrate the good discrimination capabilities of the method.

This paper discusses a new approach for clustering sequences of bar-long percussive and bass-line patterns in audio music collections and its application to genre classification. Many musical genres and styles are characterized by two kinds of distinct representative patterns, i.e., percussive patterns and bass-line patterns. So far, in most automatic genre classification systems, rhythmic and bass melody information has not been effectively used. In order to extract bar-long unit rhythmic patterns for a music collection, we propose a clustering method based on one-pass dynamic programming andk-means clustering. For clustering bass-line patterns, a method based onk-means clustering capable of handling pitch-shifting is proposed. After extracting these two fundamental kinds of patterns for each style/genre, feature vectors which are suitable for representing information about the patterns are proposed for supervised learning. Experimental results show that the automatically calculated rhythmic pattern information and bass pattern information can be used to effectively classify musical genre/style and improve upon current approaches based on timbral features.

Capturing the gestures of music performers is a common task in interactive electroacoustic music. The captured gestures can be mapped to sounds, synthesis algorithms, visuals, etc., or used for music transcription. Two of the most common approaches for acquiring musical gestures are: 1) “hyper-instruments” which are “traditional” musical instruments enhanced with sensors for directly detecting the gestures and 2) “indirect acquisition” in which the only sensor is a microphone capturing the audio signal. Hyper-instruments require invasive modification of existing instruments which is frequently undesirable. However, they provide relatively straightforward and reliable sensor measurements. On the other hand, indirect acquisition approaches typically require sophisticated signal processing and possibly machine learning algorithms in order to extract the relevant information from the audio signal. The idea of using direct sensor(s) to train a machine learning model for indirect acquisition is proposed in this paper. The resulting trained “surrogate” sensor can then be used in place of the original direct invasive sensor(s) that were used for training. That way, the instrument can be used unmodified in performance while still providing the gesture information that a hyper-instrument would provide. In addition, using this approach, large amounts of training data can be collected with minimum effort. Experimental results supporting this idea are provided in two detection contexts: 1) strike position on a drum surface and 2) strum direction on a sitar.

Music can significantly effect game play and help players understand underlying patterns in the game, or the effects of their actions on the characters. Conversely, inappropriate music can have a negative effect on players by creating additional difficulties. While game makers recognize the effects of music on game play, solutions that provide users with a choice in personal music are not forthcoming. We design, implement and evaluate an algorithm for automatically adapting an arbitrary music track from a personal library and synchronizing play back to the user, without requiring any access to the video game source code.

A new approach to instrument identification based on individual partials is presented. It makes identification possible even when the concurrently played instrument sounds have a high degree of spectral overlapping. A pairwise comparison scheme which emphasizes the specific differences between each pair of instruments is used for classification. Finally, the proposed method only requires a single note from each instrument to perform the classification. If more than one partial is available the resulting multiple classification decisions can be summarized to further improve instrument identification for the whole signal. Encouraging classification results have been obtained in the identification of four instruments (saxophone, piano, violin and guitar).

Many musical genres and styles are characterized by distinct representative rhythmic patterns. In most automatic genre classification systems global statistical features based on timbral dynamics such as mel-frequency cepstral coefficients (MFCC) are utilized but so far rhythmic information has not so effectively been used. In order to extract bar-long unit rhythmic patterns for a music collection we propose a clustering method based on one-pass dynamic programming and k-means clustering. After extracting the fundamental rhythmic patterns for each style/genre a pattern occurrence histogram is calculated and used as a feature vector for supervised learning. Experimental results show that the automatically calculated rhythmic pattern information can be used to effectively classify musical genre/style and improve upon current approaches based on timbral features.

Music listeners frequently use words to describe music. Personalized music recommendation systems such as Last.fm and Pandora rely on manual annotations (tags) as a mechanism for querying and navigating large music collections. A well-known issue in such recommendation systems is known as the cold-start problem: it is not possible to recommend new songs/tracks until those songs/tracks have been manually annotated. Automatic tag annotation based on content analysis is a potential solution to this problem and has recently been gaining attention. We describe how stacked generalization can be used to improve the performance of a state-of-the-art automatic tag annotation system for music based on audio content analysis and report results on two publicly available datasets.

The goal of this turorial is to provide a thorough theoretical overview of the state-of-the-art in Music Information Retrieval followed by a practical hands-on demonstration of several existing tools and resources that can be used for research in this area. Specific emphasis will be given on how MIR techniques relate to other fields of current multimedia research. MIR is an inherently interdisciplinary area touching on several research areas such as digital signal processing, machine learning, perception, visualization, human-computer interaction, content-based retrieval and digital libraries. Music has several unique characteristics that differentiate it from other areas of multimedia research. The different problems and techniques proposed to solve them that will described in the first part of the tutorial will be followed with concrete practical examples of applying these techniques using existing software tools and datasets.

Marsyas is an open source software framework for music analysis, retrieval and synthesis with specific emphasis on Music Information Retreival applications. It has been in development for 10 years and has been used for a variety of projects in both academia and industry in several countries. Based on a novel dataflow architecture named implicit patching it provides a variety of existing processing modules for digital signal processing, machine learning and audio input/output that can be combined at run-time to form complex dataflow networks expressing audio processing algorithms (black-box functionality). In addition it allows the easy addition of new processing modules that need to be compiled for performance purposes. Finally Marsyas is designed with inter-operability in mind and provides various mechanisms for communicating with other software including bindings to the run-time functionality in scripting languages (Python, Ruby), run-time data interchange with MATLAB, support for the Music Instrument Digital Interface (MIDI) protocol and Open Sound Control (OSC) for communicating with controller devices, and infrastructure for easy interfacing to the GUI components of the Qt toolkit.

The leading voice is an important feature of musical pieces and can often be considered as the dominant harmonic source. We propose in this paper a new scheme for the purpose of efficient dominant harmonic source separation. This is achieved by considering an harmonicity cue which is first compared with state-of-the-art cues using a generic evaluation methodology. The proposed separation scheme is then compared to a generic computational auditory scene analysis framework. Computational speed-up and performance comparison is done using source separation and music information retrieval tasks.

The design and implementation of multimedia signal processing systems is challenging especially when efficiency and real-time performance is desired. In many modern applications, software systems must be able to handle multiple flows of various types of multimedia data such as audio and video. Researchers frequently have to rely on a combination of different software tools for each modality to assemble proof-of-concept systems that are inefficient, brittle and hard to maintain. Marsyas is a software framework originally developed to address these issues in the domain of audio processing. In this paper we describe MarsyasX, a new open-source cross-modal analysis framework that aims at a broader score of applications. It follows a dataflow architecture where complex networks of processing objects can be assembled to form systems that can handle multiple and different types of multimedia flows with expressiveness and efficiency.

The predominant melodic source, frequently the singing voice, is an important component of musical signals. In this paper, we describe a method for extracting the predominant source and corresponding melody from ldquoreal-worldrdquo polyphonic music. The proposed method is inspired by ideas from computational auditory scene analysis. We formulate predominant melodic source tracking and formation as a graph partitioning problem and solve it using the normalized cut which is a global criterion for segmenting graphs that has been used in computer vision. Sinusoidal modeling is used as the underlying representation. A novel harmonicity cue which we term harmonically wrapped peak similarity is introduced. Experimental results supporting the use of this cue are presented. In addition, we show results for automatic melody extraction using the proposed approach.

This paper presents a technique to transform high-effort voices into breathy voices using adaptive pre-emphasis linear prediction (APLP). The primary benefit of this technique is that it estimates a spectral emphasis filter that can be used to manipulate the perceived vocal effort. The other benefit of APLP is that it estimates a formant filter that is more consistent across varying voice qualities. This paper describes how constant pre-emphasis linear prediction (LP) estimates a voice source with a constant spectral envelope even though the spectral envelope of the true voice source varies over time. A listening experiment demonstrates how differences in vocal effort and breathiness are audible in the formant filter estimated by constant pre-emphasis LP. APLP is presented as a technique to estimate a spectral emphasis filter that captures the combined influence of the glottal source and the vocal tract upon the spectral envelope of the voice. A final listening experiment demonstrates how APLP can be used to effectively transform high-effort voices into breathy voices. The techniques presented here are relevant to researchers in voice conversion, voice quality, singing, and emotion.

The goal of computational auditory scene analysis (CASA) is to create computer systems that can take as input a mixture of sounds and form packages of acoustic evidence such that each package most likely has arisen from a single sound source. We formulate sound source tracking and formation as a graph partitioning problem and solve it using the normalized cut which is a global criterion for segmenting graphs that has been used in computer vision. It measures both the total dissimilarity between the different groups as well as the total similarity within groups. We describe how this formulation can be used with sinusoidal modeling, a common technique for sound analysis, manipulation and synthesis. Several examples showing the potential of this approach are provided.

In this paper we describe a system for detecting the tempo of sitar performance using a multimodal signal processing approach. Real-time measurements are obtained from sensors on the instrument and by wearable sensors on the performer's body. Experiments comparing audio-based and sensor-based tempo tracking are described. The real-time tempo tracking method is based on extracting onsets and applying Kalman filtering. We show how late fusion of the audio and sensor tempo estimates can improve tracking. The obtained results are used to inform design parameters for a real-time system for human-robot musical performance.

Abstract Query‐by‐humming systems offer content‐based searching for melodies and require no special musical training or knowledge. Many such systems have been built, but there has not been much useful evaluation and comparison in the literature due to the lack of shared databases and queries. The MUSART project testbed allows various search algorithms to be compared using a shared framework that automatically runs experiments and summarizes results. Using this testbed, the authors compared algorithms based on string alignment, melodic contour matching, a hidden Markov model, n‐grams, and CubyHum. Retrieval performance is very sensitive to distance functions and the representation of pitch and rhythm, which raises questions about some previously published conclusions. Some algorithms are particularly sensitive to the quality of queries. Our queries, which are taken from human subjects in a realistic setting, are quite difficult, especially for n‐gram models. Finally, simulations on query‐by‐humming performance as a function of database size indicate that retrieval performance falls only slowly as the database size increases.

Portable digital music players are becoming pervasive and the size of personal digital music collections has been steadily increasing (5-10 thousand tracks are common today). The emerging area of music information retrieval (MIR) deals with all aspects of managing, analyzing and organizing music in digital formats. The majority of work in MIR follows a search/retrieval paradigm. More recently, the importance of browsing as an interaction paradigm has been realized and several novel interfaces have been proposed. In this paper, we describe a tangible interface for content-aware browsing of music collections. The radio drum is a gestural interface based on capacitance sensors that can detect the x,y,z positions of two drum sticks in a 3D volume. We describe two possible mappings that can be used for browsing music collections without relying on metadata. The first is an explicit mapping of tempo and beat strength, and the second is a music similarity space using audio feature extraction and a self organizing map (SOM)

Sensing instrumental gestures is a common task in interactive electroacoustic music performances. The sensed gestures can then be mapped to sounds, synthesis algorithms, visuals etc. Two of the most common approaches for acquiring these gestures are: 1) Hybrid instruments which are "traditional" musical instruments enhanced with sensors that directly detect gestures 2) Indirect acquisition in which the only measurement is the acoustic signal and signal processing techniques are used to acquire the gestures. Hybrid instruments require modification of existing instruments which is frequently undesirable. However they provide relatively straightforward and reliable measuring capability. On the other hand, indirect acquisition approaches typically require sophisticated signal processing and possibly machine learning algorithms in order to extract the relevant information from the audio signals. In this paper the idea of using direct sensors to train a machine learning model for indirect acquisition is explored. This approach has some nice advantages, mainly: 1) large amounts of training data can be collected with minimum effort 2) once the indirect acquisition system is trained no sensors or modifications to the playing instrument are required. Case studies described in paper include 1) strike position on a snare drum 2) strum direction on a sitar

We report on the tempo induction contest organized during the International Conference on Music Information Retrieval (ISMIR 2004) held at the University Pompeu Fabra in Barcelona, Spain, in October 2004. The goal of this contest was to evaluate some state-of-the-art algorithms in the task of inducing the basic tempo (as a scalar, in beats per minute) from musical audio signals. To our knowledge, this is the first published large scale cross-validation of audio tempo induction algorithms. Participants were invited to submit algorithms to the contest organizer, in one of several allowed formats. No training data was provided. A total of 12 entries (representing the work of seven research teams) were evaluated, 11 of which are reported in this document. Results on the test set of 3199 instances were returned to the participants before they were made public. Anssi Klapuri's algorithm won the contest. This evaluation shows that tempo induction algorithms can reach over 80% accuracy for music with a constant tempo, if we do not insist on finding a specific metrical level. After the competition, the algorithms and results were analyzed in order to discover general lessons for the future development of tempo induction systems. One conclusion is that robust tempo induction entails the processing of frame features rather than that of onset lists. Further, we propose a new "redundant" approach to tempo induction, inspired by knowledge of human perceptual mechanisms, which combines multiple simpler methods using a voting mechanism. Machine emulation of human tempo induction is still an open issue. Many avenues for future work in audio tempo tracking are highlighted, as for instance the definition of the best rhythmic features and the most appropriate periodicity detection method. In order to stimulate further research, the contest results, annotations, evaluation software and part of the data are available at http://ismir2004.ismir.net/ISMIR_Contest.html

Content-based analysis of music can help manage the increasing amounts of music information available digitally and is becoming an important part of multimedia research. The use of drums and percussive sounds is pervasive to popular and world music. In this paper we describe an automatic system for detecting and transcribing low and medium-high frequency drum events from audio signals. Two different subband front-ends are utilized. The first is based on bandpass filters and the second is based on wavelet analysis. Experimental results utilizing music, drum loops and Indian tabla thekas as signals are provided. The proposed system can be used as a preprocessing step for rhythm-based music classification and retrieval. In addition it can be used for pedagogical purposes

We present some experiments in the semi-automatic extraction of singing voice structure. The main characteristic of the proposed approach is that the segmentation is performed specifically for each individual song using a process we call bootstrapping. In bootstrapping, a small random sampling of the song is annotated by the user. This annotation is used to learn the song-specific voice characteristics and the trained classifier is subsequently used to classify and segment the whole song. We present experimental results on a collection of pieces with jazz singers that show the potential of this approach and compare it with the traditional approach of using multiple songs for training. It is our belief that the idea of song-specific bootstrapping is applicable to other types of music and audio computer-supported annotation.

Abstract The constantly increasing amount of audio available in digital form necessitates the development of software systems for analyzing and retrieving digital audio. In this work, we describe our efforts in developing such systems. More specifically, we describe the design philosophy behind our approach, the specific problems we try to solve, and how we evaluate the performance of our algorithms. Automatic music analysis and retrieval of non‐speech digital audio is a relatively new field, and the existing techniques are far from perfect. To improve the performance of the developed techniques, two main techniques are used: (1) integration of information from multiple analysis and retrieval algorithms and (2) the use of graphical user interfaces that enable the user to provide feedback to the design, development, and evaluation of the algorithms. All the developed algorithms and user interfaces are integrated under MARSYAS, a software framework for research in computer audition.

A large portion of data exchanged in today's peer-to-peer (P2P) networks consists of music stored as MP3 compressed audio. Existing P2P systems typically are not scalable and only support primitive methods for the searching of music files, e.g., by looking up exact filenames or using simple metadata information such as artist or album name. In this paper, we present the design and evaluation of a scalable P2P system that uses rendezvous points (RPs) for music file registration and query resolution, and supports content-based music information retrieval (MIR) of audio signals.

Musical genres are categorical labels created by humans to characterize pieces of music. A musical genre is characterized by the common characteristics shared by its members. These characteristics typically are related to the instrumentation, rhythmic structure, and harmonic content of the music. Genre hierarchies are commonly used to structure the large collections of music available on the Web. Currently musical genre annotation is performed manually. Automatic musical genre classification can assist or replace the human user in this process and would be a valuable addition to music information retrieval systems. In addition, automatic musical genre classification provides a framework for developing and evaluating features for any type of content-based analysis of musical signals. In this paper, the automatic classification of audio signals into an hierarchy of musical genres is explored. More specifically, three feature sets for representing timbral texture, rhythmic content and pitch content are proposed. The performance and relative importance of the proposed features is investigated by training statistical pattern recognition classifiers using real-world audio collections. Both whole file and real-time frame-based classification schemes are described. Using the proposed feature sets, classification of 61% for ten musical genres is achieved. This result is comparable to results reported for human musical genre classification.

There is a huge amount of audio data available that is compressed using the MPEG audio compression standard. Sound analysis is based on the computation of short time feature vectors that describe the instantaneous spectral content of the sound. An interesting possibility is the calculation of features directly from compressed data. Since the bulk of the feature calculation is performed during the encoding stage this process has a significant performance advantage if the available data is compressed. Combining decoding and analysis in one stage is also very important for audio streaming applications. In this paper, we describe the calculation of features directly from MPEG audio compressed data. Two of the basic processes of analyzing sound are: segmentation and classification. To illustrate the effectiveness of the calculated features we have implemented two case studies: a general audio segmentation algorithm and a music/speech classifier. Experimental data is provided to show that the results obtained are comparable with sound analysis algorithms working directly with audio samples.