This article presents the Software Architecture for Immersipresence (SAI) framework for the design, analysis and implementation of interactive software systems. SAI defines an architectural style whose data and processing models capture temporal properties of computational primitives. SAI adopts an asynchronous concurrent processing model in order to allow designing for optimal system latency and throughput. The SAI style is modular to enable scalability, and to facilitate incremental system design, integration and evolution, as well as code development, testing and reuse. A graph-based notation for architectural designs aims to communicate system architectures in an intuitive and scalable manner. The open-source Modular Flow Scheduling Middleware (MFSM) provides a multi-threaded, cross-platform implementation of SAI's primitives. The SAI/MFSM framework has been used in the design and implementation of numerous interactive systems in research, education and artistic performance settings. This article illustrates the definition and use of SAI with examples from computer vision.

We present Mimi4x, an interactive audio-visual installation that engages members of the general public in high-level structural decision-making in musical improvisation. Through a MIDI controller, the user manages four instances of the Mimi 1.5 improvisation system. The visuals in each Mimi system animate the generation of new music, depicting the recombination of musical material, and the flow of generated music in time. A composer prepares original music material for each Mimi to use in its improvisation. Through Mimi4x, the user is directly involved in the improvisation/composition process, and takes on some of the role of the composer. The user chooses which oracles will perform and when, and can adjust the recombination rate for, and playback volume of, each oracle during performance. This simple method of interaction is accessible to all users regardless of their musical background, yet gives the user significant and satisfying influence over the music generated.

This poster reports on the introduction of class-wide projects in computer science courses at different levels in the curriculum, as a means to fostering collaborative teamwork and creativity.

Immersive, interactive applications grouped under the concept of Immersipresence require on-line processing and mixing of multimedia data streams and structures. One critical issue seldom addressed is the integration of different solutions to technical challenges, developed independently in separate fields, into working systems, that operate under hard performance constraints. In order to realize the Immersipresence vision, a consistent, generic approach to system integration is needed, that is adapted to the constraints of research development. This paper introduces SAI, a new software architecture model for designing, analyzing and implementing applications performing distributed, asynchronous parallel processing of generic data streams. SAI provides a universal framework for the distributed implementation of algorithms and their easy integration into complex systems that exhibit desirable software engineering qualities such as efficiency, scalability, extensibility, reusability and interoperability. The SAI architectural style and its properties are described. The use of SAI and of its supporting open source middleware (MFSM) is illustrated with integrated, distributed interactive systems.

We present the design and construction of a handheld mirror simulation device. The perception of the world reflected through a mirror depends on the viewer's position with respect to the mirror and the 3-D geometry of the world. In order to simulate a real mirror on a computer screen, images of the observed world, consistent with the viewer's position, must be synthesized and displayed in real- time. Our system is build around a LCD screen manipulated by the user, a single camera fixed on the screen, and a tracking device. The continuous input video stream and tracker data is used to synthesize, in real-time, a continuous video stream displayed on the LCD screen. The synthesized video stream is a close approximation of what the user would see on the screen surface if it were a real mirror. Our system provides a generic interface for applications involving rich, first-person interaction, such as the virtual daguerreotype.

We present MuSA.RT, Opus 1, a multimodal interactive system for music analysis and visualization using the Spiral Array model. Real-time MIDI input from a live performance is processed, analyzed and mapped to the 3D model, revealing tonal structures such as pitches, chords and keys. A user can concurrently navigate through the Spiral Array space using a gamepad or set the camera control to automatic pilot. The interaction among and concurrent processing of the different data streams is made possible through the Modular Flow Scheduling Middleware.

A person is immersed when they feel part of an environment they experience and influence. While virtual immersion systems are usually designed on a case by case basis, and are not easily reusable or scalable, our goal is to specify and develop a framework for the design and integration of immersive systems. We address the issues raised in the design and implementation of the middleware components necessary for immersion, in a generic, extensible, modular architecture for Integrated Media Systems we have developed for efficient generic concurrent processing of data streams. Design principles are illustrated on specific visual immersion components. Utilization of these components is demonstrated with a real-time immersive interactive application.

We discuss the issues and challenges of generic object recognition. We argue that high-level, volumetric part-based descriptions are essential in the process of recognizing objects that might never have been observed before, and for which no exact geometric model is available. We discuss the representation scheme and its relationships to the three main tasks to solve: extracting descriptions from real images, under a wide variety of viewing conditions; learning new objects by storing their description in a database; and recognizing objects by matching their description to that of similar previously observed objects.

Immersive, interactive applications require on-line processing and mixing of multimedia data. In order to realize the Immersipresence vision, we propose a generic, extensible, modular multimedia system software architecture. We describe here the Flow Scheduling Framework (FSF), that constitutes the core of its middleware layer. The FSF is an extensible set of classes that provide basic synchronization functionality and composition mechanisms to develop data-stream processing components. In this dataflow approach, applications are implemented by specifying data streams and their path through processing nodes, where they can undergo various manipulations. We describe the details of the FSF data and processing model that supports stream synchronization in a concurrent processing framework. We illustrate the FSF concepts with a real-time video stream processing application.