Growing evidence suggests that games can be a highly effective and motivating framework for learning. Digital games are increasingly prevalent in today's classrooms, and game-based learning is emerging as a successful means of reaching students who often struggle to succeed through more traditional approaches. For example, students with autism face a unique set of challenges and goals in the context of classroom learning. Principles of well-designed games offer a powerful framework for designing new types of learning experiences for students with autism. Moreover, recent innovations in computing interfaces have also lead to new forms of socio-collaborative experiences with digital media. This article introduces our recent case study in applying game-based learning in Special Education. We will present the learning scenario goals and implementation, focusing on how this work successfully applied well-designed game principles to this context. We briefly describe our evaluation methodology and preliminary results.

Mediated XicanIndio Resolana is an interactive rhetorical space where multiple participants engage in three rounds of verbal discourse surrounding a special topic (Figure 1.1). In this space, discourse is coupled with the manipulation of digital media through cultural interaction protocols and symbolic gestures. Our aim is to use this system to conduct social and cultural work in middle and high school classrooms. As a design team that includes Chicano and Native American designers, we seek to express respectful and appropriate design sensibilities that derive from indigenous epistemologies. To achieve this we have sought critical feedback through our cross-cultural partnerships with members of various indigenous communities.

In this art submission we describe a collection of twelve multimedia art works that combine sound, images and animation. These works are the result of collaborative creative practice between media artists and students in the context of two high school Language Arts classes for non-Native English speakers. The works were created, in part, by the students physically acting out aspects of the composition within SMALLab, a mixed reality environment for multimodal composition. We call the works "Embodied Poetry" because of this physical action aspect. The title of the collection, Stratification, refers to the layers of meaning within human language. The theme of the collection is English Idioms and the works incorporate both literal and figurative meanings. Examples of idioms include "bringing home the bacon," "fighting a losing battle", and "cutting through red tape." These works were created in an everyday educational setting, with the goal of improving student learning. Nevertheless, they should be viewed as works of art. For us, the artistic contribution of this work includes the pieces themselves and also the guided creative process by which they were created. Thus, for the exhibit we will showcase the works themselves and a documentary describing the process.

In this video presentation, we introduce the Situated Multimedia Arts Learning Lab [SMALLab], a mixed-reality learning environment that supports interactive engagement through full body 3D movements and gestures within a collaborative, computationally mediated space. The video begins by describing the holistic approach to embodied and mediated learning developed by our transdisciplinary research team, grounded in understandings derived from research in the learning sciences, digital media and human computer interaction. We then outline the three core tenets of effective learning exemplified by our research -- embodiment, multimodality and collaboration. The video next demonstrates the design and functionality of the physical and digital components of SMALLab. We conclude by illustrating our partner collaborations with K12 teachers and students with four scenarios depicting Geography, Physics, Language Arts and Chemistry learning modules.

We describe the realization of two interactive, mixed-reality installations arising from a partnership of K-12, university, and museum participants. Our goal was to apply emerging technologies to produce an innovative, hands-on arts learning experience within a conventional art museum. Suspended Animation, a Reflection on Calder is a mixed-reality installation created in response to a sculpture by Alexander Calder. Another Rack for Peto was created in response to a painting by John Frederick Peto. Both installations express formal aspects of the original artworks, and allow visitors to explore specific conceptual themes through their interactions. The project culminated in a six-month exhibition where the original artworks were presented alongside these new installations. We present data that the installations were well received by an audience of 25,000 visitors.

Many students' learning styles are not well served by traditional classroom educational approaches. While interactive media tools can be effective in addressing shortcomings in traditional approaches, few projects provide practical solutions that support collaborative, discovery-based educational models while simultaneously addressing different student learning styles.

In this paper, we develop a novel real-time, interactive, automatic multimodal exploratory environment that dynamically adapts the media presented, to user context. There are two key contributions of this paper – (a) development of multimodal user-context model and (b) modeling the dynamics of the presentation to maximize coherence. We develop a novel user-context model comprising interests, media history, interaction behavior and tasks, that evolves based on the specific interaction. We also develop novel metrics between media elements and the user context. The presentation environment dynamically adapts to the current user context. We develop an optimal media selection and display framework that maximizes coherence, while constrained by the user-context, user goals and the structure of the knowledge in the exploratory environment. The experimental results indicate that the system performs well. The results also show that user-context models significantly improve presentation coherence.

In this paper, we present our efforts towards creating interfaces for networked media exploration and collaborative annotation. The problem is important since online social networks are emerging as conduits for exchange of everyday experiences. These networks do not currently provide media-rich communication environments. Our approach has two parts -- collaborative annotation, and a media exploration framework. The collaborative annotation takes place through a web based interface, and provides to each user personalized recommendations, based on media features, and by using a common sense inference toolkit. We develop three media exploration interfaces that allow for two-way interaction amongst the participants -- (a) spatio-temporal evolution, (b) event cones and (c) viewpoint centric interaction. We also analyze the user activity to determine important people and events, for each user. We also develop subtle visual interface cues for activity feedback. Preliminary user studies indicate that the system performs well and is well liked by the users.

Visualization and knowledge of detailed pressure information can play a vital role in multimodal sensing of human movement. We have designed a high-resolution pressure sensing floor prototype with a sensor density of one sensor per square centimeter that can provide real-time information about the location of the performer on the floor as well as the amount of pressure being exerted on the floor. Hardware and software have been developed for detecting, collecting, transmitting and rendering a graphical representation of the pressure data gathered from the sensors. This prototype can be easily reconfigured to cover large floor areas, and integrates closely with video, audio and motion-based sensing technologies to facilitate robust multimodal sensing. In our demonstrations, we show that the system accurately captures and transmits pressure information, and we illustrate how this forms a basis for a variety of applications including use in rehabilitation, virtual reality, entertainment, and children activity centers.

Interactions is an interactive multimedia installation designed and realized by the author. The installation utilizes two neural network artist agents that act as virtual artists to manipulate a body of images, texts, and sounds collected from the internet as directed by audience participants. The piece addresses issues including competition in the arts, machine learning in media, the role of popular acceptance in art, and the relationship between raw materials and style in creating media.

This paper describes work on a composed instrument, the digital rainstick. The instrument is an acoustic rainstick that is augmented by five sensors that send real time performance data to the computer. A generative mechanism creates sound outputs that extend the sonic and expressive means of the rainstick, while collaborating with the performer to generate a compelling musical form that unfolds over the duration of the piece. The design and implementation of the hardware and software tools are first described, followed by a discussion of the results of this work.