In this workshop, participants will try their hand at a variety of tangible, embodied, and embedded sensing and feedback technologies including vibrotactile instruments, expressive mechatronics, gesturally modulated fields of light, sound, mist; realtime steerable immersive atmospheres. Working through hands-on experience by theme, participants will be introduced to compositional and experimental methodologies. In the second half of the workshop, participants will compose together some simple "ecosystems" using the Synthesis Center's hardware-software media choreography architecture (sc), in the iStage experimental theater-scale blackbox space.

The integration of new digital and physical fabrication tools with fine arts has the potential to provide new outlets for artistic expression, while at the same time raising questions about the role of material and process in artistic practice. In this work, we present Lithobox, a system that translates the traditional ceramic and lighting technique of lithophanes into a means of creating illuminated 3D models through a creative approach that utilizes both digital and tangible construction. Through work sessions with nine artists, we explored how the Lithobox fabrication process impacted the way artists manifest design ideas and engage in creative exploration in crafting. At the TEI arts track, we plan to show our system and the physical lithophanes from our work with artists. The attendees will likely discuss the design, material, and artistic aspects of our exhibit. From these discussions, our goal is to gain insight into beneficial directions for integrating digital technology into traditional fine arts practices.

For people with Parkinson's disease (PD), gait and postural impairments can significantly affect their ability to perform activities of daily living. Presentation of appropriate cues has been shown to improve gait in PD. Based on this, a treadmill-based system and experimental paradigm were developed to determine if people with PD can utilize real-time feedback (RTFB) of step length or back angle (uprightness) to improve gait and posture. Eleven subjects (mean age 67 ± 8 years) with mild-to-moderate PD (Hoehn and Yahr stage I-III) were evaluated regarding their ability to successfully utilize RTFB of back angle or step length during quiet standing and treadmill walking tasks during a single session in their medication-on state. Changes in back angle and step length due to feedback were compared using Friedman nonparametric tests with Wilcoxon Signed-Rank tests for post-hoc comparisons. Improvements in uprightness were observed as an increase in back angle during quiet standing (p = 0.005) and during treadmill walking (p = 0.005) with back angle feedback when compared to corresponding tasks without feedback. Improvements in gait were also observed as an increase in step length (p = 0.005) during step length feedback compared to tasks without feedback. These results indicate that people with mild-to-moderate PD can utilize RTFB to improve upright posture and gait. Future work will investigate the long-term effects of this RTFB paradigm and the development of systems for clinical or home-based use.

This paper presents an interactive home-based adaptive mixed reality system (HAMRR) for upper extremity stroke rehabilitation. This home-based system is an extension of a previously designed and currently implemented clinical system. The goal of HAMRR is to restore motor function to chronic stroke survivors by providing an engaging long-term reaching task therapy at home. The HAMMR system tracks movement of the wrist and torso, and provides real-time, post-trial, and post-set multimodal feedback to encourage the stroke survivor to self-assess his or her movement and engage in active learning of new movement strategies. This experiential media system uses a computational adaptation scheme to create a continuously challenging and unique multi-year therapy experience through the use of multiple, integrated audio and visual feedback streams. Novel design features include creating an over-arching story for the participant, the ability of the system to adapt the feedback over multiple time scales, and the ability for this system to integrate into any home.

The Interaction Stations Exhibit features screen-based, interactive works that align with the new conference themes, and integrate into the physical setting of the conference center. In this paper we describe our motivation for this format, as well as the works selected and larger connections.

There has been an increased interest in using interactive multimedia as a mechanism for physical rehabilitation, including the direct usage of off-the-shelf games and gaming systems. Although, these games have been shown to have physical and psychosocial benefits in the elderly; they were limited when participants had significant physical and/or cognitive disabilities. The limited success in the use of gaming systems to promote rehabilitation may largely be due to the fact that the games being played on them were not designed with rehabilitation in mind and thus they do not necessarily target specific the symptoms or conditions of the disorder. However, if designed correctly and for rehabilitation, interactive multimedia and gaming systems have significant rehabilitation potential. We believe for a system to effective in promoting rehabilitation, its approach must be based in the clinical techniques used to treat the symptoms and the principles of neural science behind the disorder. In this paper, we describe and approach to Parkinson's rehabilitation using this methodology.

Mediated XicanIndio Resolana is an interactive rhetorical space where multiple participants engage in three rounds of verbal discourse surrounding a special topic (Figure 1.1). In this space, discourse is coupled with the manipulation of digital media through cultural interaction protocols and symbolic gestures. Our aim is to use this system to conduct social and cultural work in middle and high school classrooms. As a design team that includes Chicano and Native American designers, we seek to express respectful and appropriate design sensibilities that derive from indigenous epistemologies. To achieve this we have sought critical feedback through our cross-cultural partnerships with members of various indigenous communities.

HandJabber is a movement-based interactive sound installation and research environment for exploring collaborative creative expression. HandJabber utilizes the emergence of semantic meanings from gesture and interpersonal behavior as a single interface for music performance by responding to an individual participant's hand and arm gestures, and reacting to the non-verbal interpersonal behavior between two participants. Within this context, we explore three major areas of non-verbal behavior, (a) metaphoric hand and arm gesture, (b) interpersonal space, and (c) body orientation. Our poster presents results from informal user and case studies.

In this paper, we introduce the semantic network model (SNM), a generalization of the hidden Markov model (HMM) that uses factorization of state transition probabilities to reduce training requirements, increase the efficiency of gesture recognition and on-line learning, and allow more precision in gesture modeling. We demonstrate the advantages both formally and experimentally, using examples such as full-body multimodal gesture recognition via optical motion capture and a pressure sensitive floor, as well as mouse/pen gesture recognition. Our results show that our algorithm performs much better than the traditional approach in situations where training samples are limited and/or the precision of the gesture model is high.

This paper presents a novel real-time, multi-modal biofeedback system for stroke patient therapy. The problem is important as traditional mechanisms of rehabilitation are monotonous, and do not incorporate detailed quantitative assessment of recovery in addition to traditional clinical schemes. We have been working on developing an experiential media system that integrates task dependent physical therapy and cognitive stimuli within an interactive, multimodal environment. The environment provides a purposeful, engaging, visual and auditory scene in which patients can practice functional therapeutic reaching tasks, while receiving different types of simultaneous feedback indicating measures of both performance and results. There are three contributions of this paper - (a) identification of features and goals for the functional task (b) The development of sophisticated feedback (auditory and visual) mechanisms that match the semantics of action of the task. We additionally develop novel action-feedback coupling mechanisms. (c) New metrics to validate the ability of the system to promote learnability, stylization and engagement. We have validated the system for nine subjects with excellent results.

This paper presents a novel real-time, multi-modal biofeedback system for stoke patient therapy. The problem is important as traditional mechanisms of rehabilitation are monotonous, and do not incorporate detailed quantitative assessment of recovery in addition to traditional clinical schemes. We have been working on developing an experiential media system that integrates task dependent physical therapy and cognitive stimuli within an interactive, multimodal environment. The environment provides a purposeful, engaging, visual and auditory scene in which patients can practice functional therapeutic reaching tasks, while receiving different types of simultaneous feedback indicating measures of both performance and results. There are two contributions of this paper - (a) identification of features and goals for the functional task, (b) the development of sophisticated feedback (auditory and visual) mechanisms that match the semantics of action of the task.

Movement-based interactive dance has recently attracted great interest in the performing arts. While utilizing motion capture technology, the goal of this project was to design the necessary real-time motion analysis engine, staging, and communication systems for the completion of a movement-based interactive multimedia dance performance. The movement analysis engine measured the correlation of dance movement between three people wearing similar sets of retro-reflective markers in a motion capture volume. This analysis provided the framework for the creation of an interactive dance piece, Lucidity, which will be described in detail. Staging such a work also presented additional challenges. These challenges and our proposed solutions will be discussed. We conclude with a description of the final work and a summary of our future research objectives.

Many events of interest in human activity-based multimedia applications exhibit a high degree of temporal structure. This structure generates expectancies regarding the occurrence and location of subsequent events. In the context of switching state-space models, we develop a general Bayesian framework for representing temporal expectancies and fusing them with raw sense-data to improve both event segmentation and temporal structure identification. Furthermore, we develop a new cognitive model for event anticipation which adapts to incoming sense-data in real time. Comparative advantages of the proposed framework are realized in controlled experiments involving partially-observed, quasi-periodic event streams

In this paper, we report a real-time gesture driven interactive system with multimodal feedback for performing arts, especially dance. The system consists of two major parts., a gesture recognition engine and a multimodal feedback engine. The gesture recognition engine provides real-time recognition of the performer's gesture based on the 3D marker coordinates from a marker-based motion capture system. According to the recognition results, the multimodal feedback engine produces associated visual and audio feedback to the performer. This interactive system is simple to implement and robust to errors in 3D marker data. Satisfactory interactive dance performances have been successfully created and presented using the reported system