In gaming and other multimedia applications, there is an increasing need to simulate and transform the sounds of physical interactions with real-world objects. When these sounds are based upon source recordings, this motivates the need for flexible, yet ecologically valid resynthesis. With modal analysis as a front end, banded waveguide resynthesis enables flexible modifications to dynamic spectral content and varied forms of interaction. Unfortunately, traditional banded waveguide models using biquad filters can have problems with undesired transient behavior. To this end, we introduce a new perfect reconstruction filterbank that allows for each filter to be individually tuned to a desired mode frequency and precisely cancel the other selected frequencies. Comparisons with traditional biquad models amply demonstrate the improved transient response and overall fidelity of our model. In addition, examples of resynthesis, transformation, and alternative interaction with real-world recordings are given.

Typically, speech enhancement algorithms minimize a suitable error criterion in the spectral or time domain. Although the error criterions have included perceptual properties such as masking thresholds, non-uniform frequency resolution and sensitivity of the auditory system, these are only done heuristically and the error criterion does not explicitly include an auditory model in their formulation. In this paper, we propose an auditory-domain based speech enhancement algorithm that minimizes the distortion between the auditory representation of the estimated and desired signal. Simulation results indicate that the proposed algorithm performs effectively under different noise conditions and also results in a lower average loudness error.

Recent work in audio information retrieval has demonstrated the effectiveness of combining semantic information, such as descriptive, tags with acoustic content. However, these methods largely ignore the possibility of tag queries that do not yet exist in the database and the possibility of similar terms. In this work, we propose a network structure integrating similarity between semantic tags, content-based similarity between environmental audio recordings, and the collective sound descriptions provided by a user community. We then demonstrate the effectiveness of our approach by comparing the use of existing similarity measures for incorporating new vocabulary into an audio annotation and retrieval system.

Automatically annotating or tagging unlabeled audio files has several applications, such as database organization and recommender systems. We are interested in the case where the system is trained using clean high-quality audio files, but most of the files that need to be automatically tagged during the test phase are heavily compressed and noisy, for instance if they were captured on a mobile device. In this situation we assume the audio files follow a covariate shift model in the acoustic feature space, i.e., the feature distributions are different in the training and test phases, but the conditional distribution of labels given features remains unchanged. Our method uses a specially designed audio similarity measure as input to a set of weighted logistic regressors, which attempt to alleviate the influence of covariate shift. Results on a freely available database of sound files contributed and labeled by non-expert users, demonstrate effective automatic tagging performance.

We propose a method for characterizing sound activity in fixed spaces through segmentation, indexing, and retrieval of continuous audio recordings. Regardingsegmentation, we present a dynamic Bayesian network (DBN) that jointly infers onsets and end times of the most prominent sound events in the space, along with an extension of the algorithm for covering large spaces with distributed microphone arrays. Each segmented sound event isindexedwith a hidden Markov model (HMM) that models the distribution of example-based queries that a user would employ toretrievethe event (or similar events). In order to increase the efficiency of the retrieval search, we recursively apply a modified spectral clustering algorithm to group similar sound events based on the distance between their corresponding HMMs. We then conduct a formal user study to obtain the relevancy decisions necessary for evaluation of our retrieval algorithm on both automatically and manually segmented sound clips. Furthermore, our segmentation and retrieval algorithms are shown to be effective in both quiet indoor and noisy outdoor recording conditions.

HandJabber is a movement-based interactive sound installation and research environment for exploring collaborative creative expression. HandJabber utilizes the emergence of semantic meanings from gesture and interpersonal behavior as a single interface for music performance by responding to an individual participant's hand and arm gestures, and reacting to the non-verbal interpersonal behavior between two participants. Within this context, we explore three major areas of non-verbal behavior, (a) metaphoric hand and arm gesture, (b) interpersonal space, and (c) body orientation. Our poster presents results from informal user and case studies.

In most real-world situations, a single microphone is insufficient for the characterization of an entire auditory scene. This often occurs in places such as office environments which consist of several interconnected spaces that are at least partially acoustically isolated from one another. To this end, we extend our previous work on segmentation of natural sounds to perform scene characterization using a sparse array of microphones, strategically placed to ensure that all parts of the environment are within range of at least one microphone. By accounting for which microphones are active for a given sound event, we perform a multi-channel segmentation that captures sound events occurring in any part of the space. The segmentation is inferred from a custom dynamic Bayesian network (DBN) that models how event boundaries influence changes in audio features. Example recordings illustrate the utility of our approach in a noisy office environment.

There has been much recent progress in the technical infrastructure necessary to continuously characterize and archive all sounds, or more precisely auditory streams, that occur within a given space or human life. Efficient and intuitive access, however, remains a considerable challenge. In specifically musical domains, i.e., melody retrieval, query-by-example (QBE) has found considerable success in accessing music that matches a specific query. We propose an extension of the QBE paradigm to the broad class of natural and environmental sounds, which occur frequently in continuous recordings. We explore several cluster-based indexing approaches, namely non-negative matrix factorization (NMF) and spectral clustering to efficiently organize and quickly retrieve archived audio using the QBE paradigm. Experiments on a test database compare the performance of the different clustering algorithms in terms of recall, precision, and computational complexity. Initial results indicate significant improvements over both exhaustive search schemes and traditional K- means clustering, and excellent overall performance in the example-based retrieval of environmental sounds.

We describe the realization of two interactive, mixed-reality installations arising from a partnership of K-12, university, and museum participants. Our goal was to apply emerging technologies to produce an innovative, hands-on arts learning experience within a conventional art museum. Suspended Animation, a Reflection on Calder is a mixed-reality installation created in response to a sculpture by Alexander Calder. Another Rack for Peto was created in response to a painting by John Frederick Peto. Both installations express formal aspects of the original artworks, and allow visitors to explore specific conceptual themes through their interactions. The project culminated in a six-month exhibition where the original artworks were presented alongside these new installations. We present data that the installations were well received by an audience of 25,000 visitors.

We propose a probabilistic generative model for extracting intended path shape qualities of an object moving under human control in real time. At each instant, we decide whether the object is moving in a straight, curved, or random path, or whether it has stopped moving. Our model incorporates sensor noise as well as human imperfections in the intended motion. As well as tracking the object's position, velocity, and motion direction, we compute the posterior probability of each shape quality hypothesis given all sensed-data in the horizon [t - N + 1,t]; the hypothesis maximizing this posterior is taken as the decision. The posterior is computed using the unscented Kalman filter (UKF), as our model is inherently nonlinear. The path-shape quality tracking is successfully embedded in a hybrid physical-digital interface where the position of an illuminated ball, sensed by a low-cost video camera array, triggers multimodal feedback in a mediated learning environment. We show successful results on a variety of real-world motion paths where the participant is given only verbal descriptions of how to move. Our generative model is further validated by user studies involving a simple color-based interaction, where participants discover shape quality controls as they interact.

We propose a probabilistic method for the joint segmentation and melody extraction for musical audio signals which arise from a monophonic score. The method operates on framewise short-time Fourier transform (STFT) peaks, enabling a computationally efficient inference of note onset, duration, and pitch attributes while retaining sufficient information for pitch determination and spectral change detection. The system explicitly models note events in terms of transient and steady-state regions as well as possible gaps between note events. In this way, the system readily distinguishes abrupt spectral changes associated with musical onsets from other abrupt change events. Additionally, the method may incorporate melodic context by modeling note-to-note dependences. The method is successfully applied to a variety of piano and violin recordings containing reverberation, effective polyphony due to legato playing style, expressive pitch variations, and background voices. While the method does not provide a sample-accurate segmentation, it facilitates the latter in subsequent processing by isolating musical onsets to frame neighborhoods and identifying possible pitch content before and after the true onset sample location

Many events of interest in human activity-based multimedia applications exhibit a high degree of temporal structure. This structure generates expectancies regarding the occurrence and location of subsequent events. In the context of switching state-space models, we develop a general Bayesian framework for representing temporal expectancies and fusing them with raw sense-data to improve both event segmentation and temporal structure identification. Furthermore, we develop a new cognitive model for event anticipation which adapts to incoming sense-data in real time. Comparative advantages of the proposed framework are realized in controlled experiments involving partially-observed, quasi-periodic event streams