We introduce the "Intonation" dataset of amateur vocal performances with a tendency for good intonation, collected from Smule, Inc. The dataset can be used for music information retrieval tasks such as autotuning, query by humming, and singing style analysis. It is available upon request on the Stanford CCRMA DAMP website.1We describe a semi-supervised approach to selecting the audio recordings from a larger collection of performances based on intonation patterns. The approach can be applied in other situations where a researcher needs to extract a subset of data samples from a large database. A comparison of the "Intonation" dataset and the remaining collection of performances shows that the two have different intonation behavior distributions.

Model evaluation plays a special role in interactive machine learning (IML) systems in which users rely on their assessment of a model's performance in order to determine how to improve it. A better understanding of what model criteria are important to users can therefore inform the design of user interfaces for model evaluation as well as the choice and design of learning algorithms. We present work studying the evaluation practices of end users interactively building supervised learning systems for real-world gesture analysis problems. We examine users' model evaluation criteria, which span conventionally relevant criteria such as accuracy and cost, as well as novel criteria such as unexpectedness. We observed that users employed evaluation techniques---including cross-validation and direct, real-time evaluation---not only to make relevant judgments of algorithms' performance and interactively improve the trained models, but also to learn to provide more effective training data. Furthermore, we observed that evaluation taught users about what types of models were easy or possible to build, and users sometimes used this information to modify the learning problem definition or their plans for using the trained models in practice. We discuss the implications of these findings with regard to the role of generalization accuracy in IML, the design of new algorithms and interfaces, and the scope of potential benefits of incorporating human interaction in the design of supervised learning systems.

It is challenging to navigate a dictionary consisting of thousands of entries in order to select appropriate words for building communication. This is particularly true for people with lexical access disorders like those present in aphasia. We make vocabulary navigation and word-finding easier by building a vocabulary network where links between words reflect human judgments of semantic relatedness. We report the results from a user study with people with aphasia that evaluated how our system (called ViVA) performs compared to a widely used vocabulary access system in which words are organized hierarchically into common categories and subcategories. The results indicate that word retrieval is significantly better with ViVA, but finding the first word to start a communication is still problematic and requires further investigation.

Auditory displays have been used in both human-machine and computer interfaces. However, the use of non-speech audio in assistive communication for people with language disabilities, or in other applications that employ visual representations, is still under-investigated. In this paper, we introduce SoundNet, a linguistic database that associates natural environmental sounds with words and concepts. A sound labeling study was carried out to verify SoundNet associations and to investigate how well the sounds evoke concepts. A second study was conducted using the verified SoundNet data to explore the power of environmental sounds to convey concepts in sentence contexts, compared with conventional icons and animations. Our results show that sounds can effectively illustrate (especially concrete) concepts and can be applied to assistive interfaces.

It is challenging to search a dictionary consisting of thousands of entries in order to select appropriate words for building written communication. This is true both for people trying to communicate in a foreign language who have not developed a full vocabulary, for school children learning to write, for authors who wish to be more precise and expressive, and especially for people with lexical access disorders. We make vocabulary navigation and word finding easier by augmenting a basic vocabulary with links between words based on human judgments of semantic similarity. In this paper, we report the results from a user study evaluating how our system named ViVA performs compared to a widely used assistive vocabulary in which words are organized hierarchically into common categories.

People with aphasia, a condition that impairs the ability to understand or generate written or spoken language, are aided by assistive technology that helps them communicate through a vocabulary of icons. These systems are akin to language translation systems, translating icon arrangements into spoken or written language and vice versa. However, these icon-based systems have little vocabulary breadth or depth, making it difficult for people with aphasia to apply their usage to multiple real world situations. Pictures from the web are numerous, varied, and easily accessible and thus, could potentially address the small size issues of icon-based systems. We present results from two studies that investigate this potential and demonstrate that images can be as effective as icons when used as a replacement for English language communication. The first study uses elderly subjects to investigate the efficacy of images vs. icons in conveying word meaning; the second study examines the retention of word-level meaning by both images and icons with a population of aphasics. We conclude that images collected from the web are as functional as icons in conveying information and thus, are feasible to use in assistive technology that supports people with aphasia.

The difficulties of navigating vocabulary in an assistive communication device are exacerbated for individuals with lexical access disorders like those due to aphasia. We present the design and implementation of a vocabulary network based on WordNet, a resource that attempts to model human semantic memory, that enables users to find words easily. To correct for the sparsity of links among words, we augment WordNet with additional connections derived from human judgments of semantic similarity collected in an online experiment. We evaluate the resulting system, the visual vocabulary for aphasia (ViVA), and describe its potential to adapt to a user's profile and enable faster search and improved navigation.

In this paper we study how verbs are visually conveyed in daily communication contexts for both young and old adults. Four visual modes are compared: a single static image, a panel of four static images, an animation, and a video clip. The results reveal age effects, as well as performance differences introduced by lexical verb properties and visual cues. We also suggest guidelines for visual verb creation.

In this paper, we introduce W2ANE, an Online Multimedia Language Assistant for individuals with aphasia, a language disorder that affects millions of people. W2ANE offers a rich online multimedia library (OMLA) supported by an adaptable and adaptive vocabulary scaffold (ViVA). The system, accessible over the Internet, provides a platform for applications such as looking up unknown words, constructing phrases for communication, practicing pronunciations, and accessing content. W2ANE also enables resource sharing and remote collaboration.

TAPESTREA is a sound design and composition framework that facilitates the creation of new sound from existing digital audio recordings, through interactive analysis, transformation and re-synthesis. During analysis, sound templates of different types are extracted using a variety of techniques. Each extracted template is transformed and synthesized independently, allowing specialized transformations on each template based on its type. The user interacts with TAPESTREA via a set of graphical interfaces that offer parametric control over every stage of analysis, transformation and re-synthesis. Synthesis is further controlled through ChucK scripts. These combined techniques form a workbench for completely transforming a sound scene, dynamically generating soundscapes, or creating musical tapestries by weaving together transformed elements from different recordings. Thus, TAPESTREA introduces a new paradigm for composition, sound design, and sonic sculpting tasks.

Icons and digital images used in augmentative and alternative communication (AAC) are not as effective in illustrating verbs, especially for people with cognitive degeneration or impairment. Realistic videos have possible advantages for conveying verbs, as verified in our studies with young and old adults comparing single image, multiple images, animations, and video clips. Videos are especially more effective for verbs that show concrete movements or actions. Based on our studies, we propose rules for filming video verb representations, exploring possible visual cues and other factors that may affect people's perception and interpretation.

We present two physics-based analysis, synthesis, and control systems for synthesizing hand clapping sounds. They both rely on the separation of the sound synthesis and event generation, and both are capable of producing individual hand-claps, or mimicking the asynchronous/synchronized applause of a group of clappers. The synthesis models consist of resonator filters, whose coefficients are derived from experimental measurements. The difference between these systems is mainly in the statistical event generation. While the first system allows an efficient parametric synthesis of large audiences, as well as flocking and synchronization by simple rules, the second one provides parametric extensions for synthesis of various clapping styles and enhanced control strategies. The synthesis and the control models of both systems are implemented as software running in real time at the audio sample rate, and they are available for download at at http://ccrma-www.stanford.edu/software/stk and http://www.acoustics.hut.fi/go/clapd

To recognize and understand a person’s emotion has been known as one of the most important issue in human-computer interaction. In this paper, we present a multimodal system that supports emotion recognition from both visual and acoustic feature analysis. Our main achievement is that with this bimodal method, we can effectively extend the recognized emotion categories compared to when only visual or acoustic feature analysis works alone. We also show that by carefully cooperating bimodal features, the recognition precision of each emotion category will exceed the limit set up by the single modality, both visual and acoustic. Moreover, we believe our system is closer to real human perception and experience and hence will make emotion recognition closer to practical application in the future.

In this paper, we describe ChucK - a programming language and programming model for writing precisely timed, concurrent audio synthesis and multimedia programs. Precise concurrent audio programming has been an unsolved (and ill-defined) problem. ChucK provides a concurrent programming model that solves this problem and significantly enhances designing, developing, and reasoning about programs with complex audio timing. ChucK employs a novel data-driven timing mechanism and a related time-based synchronization model, both implemented in a virtual machine. We show how these features enable precise, concurrent audio programming and provide a high degree of programmability in writing real-time audio and multimedia programs. As an extension, programmers can use this model to write code on-the-fly -- while the program is running. These features provide a powerful programming tool for building and experimenting with complex audio synthesis and multimedia programs.

Abstract The constantly increasing amount of audio available in digital form necessitates the development of software systems for analyzing and retrieving digital audio. In this work, we describe our efforts in developing such systems. More specifically, we describe the design philosophy behind our approach, the specific problems we try to solve, and how we evaluate the performance of our algorithms. Automatic music analysis and retrieval of non‐speech digital audio is a relatively new field, and the existing techniques are far from perfect. To improve the performance of the developed techniques, two main techniques are used: (1) integration of information from multiple analysis and retrieval algorithms and (2) the use of graphical user interfaces that enable the user to provide feedback to the design, development, and evaluation of the algorithms. All the developed algorithms and user interfaces are integrated under MARSYAS, a software framework for research in computer audition.

Musical genres are categorical labels created by humans to characterize pieces of music. A musical genre is characterized by the common characteristics shared by its members. These characteristics typically are related to the instrumentation, rhythmic structure, and harmonic content of the music. Genre hierarchies are commonly used to structure the large collections of music available on the Web. Currently musical genre annotation is performed manually. Automatic musical genre classification can assist or replace the human user in this process and would be a valuable addition to music information retrieval systems. In addition, automatic musical genre classification provides a framework for developing and evaluating features for any type of content-based analysis of musical signals. In this paper, the automatic classification of audio signals into an hierarchy of musical genres is explored. More specifically, three feature sets for representing timbral texture, rhythmic content and pitch content are proposed. The performance and relative importance of the proposed features is investigated by training statistical pattern recognition classifiers using real-world audio collections. Both whole file and real-time frame-based classification schemes are described. Using the proposed feature sets, classification of 61% for ten musical genres is achieved. This result is comparable to results reported for human musical genre classification.

This paper describes a technique for approximating sounds that are generated by the motions of solid objects. The technique builds on previous work in the field of physically based animation that uses deformable models to simulate the behavior of the solid objects. As the motions of the objects are computed, their surfaces are analyzed to determine how the motion will induce acoustic pressure waves in the surrounding medium. Our technique computes the propagation of those waves to the listener and then uses the results to generate sounds corresponding to the behavior of the simulated objects.

There is a huge amount of audio data available that is compressed using the MPEG audio compression standard. Sound analysis is based on the computation of short time feature vectors that describe the instantaneous spectral content of the sound. An interesting possibility is the calculation of features directly from compressed data. Since the bulk of the feature calculation is performed during the encoding stage this process has a significant performance advantage if the available data is compressed. Combining decoding and analysis in one stage is also very important for audio streaming applications. In this paper, we describe the calculation of features directly from MPEG audio compressed data. Two of the basic processes of analyzing sound are: segmentation and classification. To illustrate the effectiveness of the calculated features we have implemented two case studies: a general audio segmentation algorithm and a music/speech classifier. Experimental data is provided to show that the results obtained are comparable with sound analysis algorithms working directly with audio samples.