course Share on How to design, build, and perform with new musical interfaces Authors: Michael Lyons Ritsumeikan University Ritsumeikan UniversityView Profile , Sidney Fels University of British Columbia University of British ColumbiaView Profile Authors Info & Claims SA '17: SIGGRAPH Asia 2017 CoursesNovember 2017 Article No.: 9Pages 1–143https://doi.org/10.1145/3134472.3134473Published:27 November 2017Publication History 0citation151DownloadsMetricsTotal Citations0Total Downloads151Last 12 Months6Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Recent years have seen a significant increase of interest in rich multi-modal user interfaces going beyond conventional mouse/keyboard/screen interaction. The new interface technologies are broadly impacting music technology and culture. New musical interfaces use a variety of sensing (and actuating) modalities to receive and present information to users, and often require techniques from signal processing and machine learning in order to extract and fuse high level information from noisy, high dimensional signals over time. Hence they pose many interesting signal processing challenges while offering fascinating possibilities for new research. At the same time the richness of possibilities for new forms of musical interaction requires a new approach to the design of musical technologies and has implications for performance aesthetics and music pedagogy. This tutorial begins with a general and gentle introduction to the theory and practice of the design of new technologies for musical creation and performance. It continues with an overview of signal processing and machine learning methods which are needed for more advanced work in new musical interface design.

Advances in digital audio technologies have led to a situation where computers play a role in most music production and performance. Digital technologies offer unprecedented opportunities for the creation and manipulation of sound, however the flexibilty of these new technologies imply an often confusing array of choices for musical composers and performers. Some artists have faced this challenge by using computers directly to create music and leading to an explosion of new musical forms. However, most would agree that the computer is not a musical instrument, in the same sense as traditional instruments, and it is natural to ask 'how to play the computer' using interface technology appropriate for human brains and bodies. In 2001, we organized the first workshop on New Interfaces for Musical Expression (NIME), to attempt to answer this question by exploring connections with the better established field of human-computer interaction. This course summarizes what has been learned at NIME which has been held annually since that first workshop. We begin with an overview of the theory and practice of new musical interface design, asking what makes a good musical interface and whether there are any useful design principles or guidelines available. We will also discuss topics such as the mapping from human action to musical output, and control intimacy. Practical information about the tools for creating musical interfaces will be given, including an overview of sensors and microcontrollers, audio synthesis techniques, and communication protocols such as Open Sound Control (and MIDI). The remainder of the course will consist of several specific case studies representative of the major broad themes of the NIME conference, including augmented and sensor based instruments, mobile and networked music, and NIME pedagogy.

Advances in digital audio technologies have led to a situation where computers play a role in most music production and performance. Digital technologies offer unprecedented opportunities for the creation and manipulation of sound. However, the flexibilty of these new technologies implies an often confusing array of choices for musical composers and performers. Some artists have faced this challenge by using computers directly to create music, leading to an explosion of new musical forms. However, most would agree that the computer is not a musical instrument - in the same sense as traditional instruments - and it is natural to ask 'how to play the computer' using interface technology appropriate for human brains and bodies. A decade ago we organized the first workshop on New Interfaces for Musical Expression (NIME) to attempt to answer this question by exploring connections with the established field of human-computer interaction. This course summarizes what has been learned at NIME. We begin with an overview of the theory and practice of new musical interface design, asking what makes a good musical interface and whether there are any useful design principles or guidelines available. We will also discuss topics such as the mapping from human action to musical output, and control intimacy. Practical information about the tools for creating musical interfaces will be given, including an overview of sensors and microcontrollers, audio synthesis techniques, and communication protocols such as Open Sound Control and MIDI. The remainder of the course will consist of several specific case studies representative of the major broad themes of the NIME conference, including augmented and sensor based instruments, mobile and networked music, and NIME pedagogy.

Due to advances in digital audio technologies, computers now play a role in most music production and performance. Digital technologies offer unprecedented opportunities for creation and manipulation of sound, but the flexibilty of these new technologies implies an often confusing array of choices for musical composers and performers. Some artists are using computers directly to create music and generate an explosion of new musical forms. However, most would agree that the computer is not a musical instrument, in the same sense as traditional instruments, and it is natural to wonder "how to play the computer" using interface technology appropriate for human brains and bodies.

We describe the design and implementation of an interactive character animation interface. The system analyzes the attentive state and aspects of the affective behaviour of a viewer using input from a video camera and uses this to control the behaviour of a cartoon-like animated character. Using the interaction metaphor of a mime artist, we design the system to encourage viewer attention and interaction, with adaptation using an online reinforcement learning based on the viewer's attentive state. This work is ultimately aimed at developing a system to support the care of dementia sufferers.

The paper presents a novel approach to the peripheral display of information by applying audio effects to an arbitrary selection of music. We examine a specific instance: the communication of information about human affect, and construct a functioning prototype which captures behavioral activity level from the face and maps it to musical effects. Several audio effects are empirically evaluated as to their suitability for ambient display. We report measurements of the ambience, perceived affect, and pleasure of these effects. The findings support the hypothesis that musical effects are a promising method for ambient informational display.

We suggest that the real-time visual display of affective signals such as respiration, pulse, and skin conductivity can allow users of Web-based tutoring systems insight into each other's felt bodily experience. This allows remotely interacting users to increase their experience of empathy, or shared feeling and has the potential to enhance telelearning. We describe an implementation of such a system and present the results of a preliminary experiment in the context of a Web-based system for tutoring the writing of Chinese characters.

We present a novel multimodal interface which permits users to draw or paint using coordinated gestures of hand and mouth. A headworn camera captures an image of the mouth and the mouth cavity region is extracted by Fisher discriminant analysis of the pixel colour information. A normalized area parameter is read by a drawing or painting program to allow read-time gestural control of pen/brush parameters by mouth gesture while sketching with a digital pen/tablet. A new performance task, the Radius Control Task, is proposed as a means of systematic evaluation of performance of the interface. Data from preliminary experiments show that with some practice users can achieve single pixel radius control with ease. A trial of the system by a professional artist shows that it is ready for use as a novel tool for creative artistic expression.

No abstract available.

A method for automatically classifying facial images is proposed. Faces are represented using elastic graphs labelled with with 2D Gabor wavelet features. The system is trained from examples to classify faces on the basis of high-level attributes, such as sex, "race", and expression, using linear discriminant analysis (LDA). Use of the Gabor representation relaxes the requirement for precise normalization of the face: approximate registration of a facial graph is sufficient. LDA allows simple and rapid training from examples, as well as straightforward interpretation of the role of the input features for classification. The algorithm is tested on three different facial image datasets, one of which was acquired under relatively uncontrolled conditions, on tasks of sex, "race" and expression classification. Results of these tests are presented. The discriminant vectors may be interpreted in terms of the saliency of the input features for the different classification tasks, which we portray visually with feature saliency maps for node position as well as filter spatial frequency and orientation.

A method for automatically extracting symbolic-level information from single facial images is proposed. We combine a 2D Gabor wavelet labelled elastic graph representation with Fisher discriminant analysis. Use of this representation relaxes the requirement for exact normalization of the face. The algorithm is used to extract information about the sex, "race" and emotional expression of the face in the image. A visual interpretation of the discriminant vectors is provided.

We propose a method for automatically classifying facial images based on labeled elastic graph matching, a 2D Gabor wavelet representation, and linear discriminant analysis. Results of tests with three image sets are presented for the classification of sex, "race", and expression. A visual interpretation of the discriminant vectors is provided.

Article Avatar creation using automatic face processing Share on Authors: Michael Lyons ATR Human Information Processing Research Laboratory, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, Japan ATR Human Information Processing Research Laboratory, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, JapanView Profile , Andre Plante ATR Media Integration and Communication Laboratory, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, Japan ATR Media Integration and Communication Laboratory, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, JapanView Profile , Sebastien Jehan ATR Human Information Processing Research Laboratory, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, Japan ATR Human Information Processing Research Laboratory, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, JapanView Profile , Seiki Inoue ATR Media Integration and Communication Laboratory, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, Japan ATR Media Integration and Communication Laboratory, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, JapanView Profile , Shigeru Akamatsu ATR Human Information Processing Research Laboratory, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, Japan ATR Human Information Processing Research Laboratory, 2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-02, JapanView Profile Authors Info & Claims MULTIMEDIA '98: Proceedings of the sixth ACM international conference on MultimediaSeptember 1998 Pages 427–434https://doi.org/10.1145/290747.290816Online:01 September 1998Publication History 17citation614DownloadsMetricsTotal Citations17Total Downloads614Last 12 Months20Last 6 weeks2 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access