This paper presents a Multi-Modal EnvironmentAware Network (MEAN-RIR), which uses an encoder-decoder framework to predict room impulse response (RIR) based on multi-level environmental information from audio, visual, and textual sources. Specifically, reverberant speech capturing room acoustic properties serves as the primary input, which is combined with panoramic images and text descriptions as supplementary inputs. Each input is processed by its respective encoder, and the outputs are fed into cross-attention modules to enable effective interaction between different modalities. The MEAN-RIR decoder generates two distinct components: the first component captures the direct sound and early reflections, while the second produces masks that modulate learnable filtered noise to synthesize the late reverberation. These two components are mixed to reconstruct the final RIR. The results show that MEANRIR significantly improves RIR estimation, with notable gains in acoustic parameters.

Many augmented reality (AR) applications require a seamless blending of real and virtual content as key to increased immersion and improved user experiences. Photorealistic and non-photorealistic rendering (NPR) are two ways to achieve this goal. Compared with photorealistic rendering, NPR stylizes both the real and virtual content and makes them indistinguishable. Maintaining temporal coherence is a key challenge in NPR. We propose a NPR framework with support for temporal coherence by leveraging model-space information. Our systems targets painterly rendering styles of NPR. There are three major steps in this rendering framework for creating coherent results: tensor field creation, brush anchor placement, and brush stroke reshaping. To achieve temporal coherence for the final rendered results, we propose a new projection-based surface sampling algorithm which generates anchor points on model surfaces. The 2D projections of these samples are uniformly distributed in image space for optimal brush stroke placement. We also propose a general method for averaging various properties of brush stroke textures, such as their skeletons and colors, to further improve the temporal coherence. We apply these methods to both static and animated models to create a painterly rendering style for AR. Compared with existing image space algorithms our method renders AR with NPR effects with a high degree of coherence.

A seamless blending of the real and virtual worlds is key to increased immersion and improved user experiences for augmented reality (AR). Photorealistic and non-photorealistic rendering (NPR) are two ways to achieve this goal. Non-photorealistic rendering creates an abstract version of both the real and virtual world by stylization to make them indistinguishable. We presented a painterly rendering algorithm for AR applications. This algorithm paints composed AR video frames with bump-mapping curly brushstrokes. Tensor fields are created for each frame to define direction for brushstrokes. The anchor point of a brushstroke is tracked or warped from frame to frame. Brushstrokes are also reshaped to provide better temporal coherence. The major difference between our algorithm and existing NPR work in general graphics and AR/VR areas is we use feature points across composed AR video frames to maintain coherence in the rendering.

We present an intelligent photo slideshow system that automatically analyzes thematic information about the photo collection and utilizes such information to generate compositions and transitions in two modes: story-telling mode and person-highlighting mode. In the story-telling mode the system groups photos by a theme-based clustering algorithm and multiple photos in each theme cluster are seamlessly tiled on a slide. Multiple tiling layouts are generated for each theme cluster and the slideshow is animated by intra-cluster transitions. In the person-highlighting mode, the system first recognizes faces from photos and creates photo clusters for individuals. It then uses face areas as ROI (Regions of Interests) and creates various content-based transitions to highlight individuals in a cluster. With an emphasis on photo content, our system creates slideshows with more fluid, dynamic and meaningful structure compared to existing systems.

The "Reminiscing View" is a visualization system for browsing hierarchically organized photo and video-clip collections. The current Reminiscing View prototype supports browsing of hierarchical temporal event clusters. Users simply upload their pictures and video clips to a server and their collection is automatically clustered into hierarchical temporal events. They can then browse events in a chronologically ordered spiral or grid view and explore details of each event. They can also filter the displayed media by date, text-based search, and sharing group. The addition of a remote control interface to the system allows users to browse their collection in a ldquolean backrdquo setting such as a living room environment. This paper describes the system and interface of the Reminiscing View. It also presents results and implications of a user study conducted on an earlier version of the system.

Registration errors between the physical world and computer- generated objects are a central problem in Augmented Reality (AR) systems. Some existing AR systems have demonstrated how to dynamically estimate registration errors based on estimates of spatial errors in the system. Using these error estimates, these systems also demonstrated a number of ways of ameliorating the effects of registration error. One central part of this previous work was the creation and use of error regions around objects; unfortunately, the analytic methods used only created accurate regions for simple convex objects. In this paper, we present a simple and stable algorithm for generating the uncertainty regions for complex objects, including non-convex objects and objects with interior holes. We demonstrate how our approach can be used to create a set of more accurate error-based highlights in the presence of registration error, and also be used as a general highlighting mechanism.

Non-photorealistic rendering (NPR) is an attractive approach for seamlessly blending virtual and physical content in Augmented Reality (AR) applications. Simple NRP techniques, that use information from a single rendered image, have been demonstrated in real-time AR systems. More complex NRP techniques require visual coherence across multiple frames of video, and typical offline algorithms are expensive and/or require global knowledge of the video sequence. To use such techniques in real-time AR, fast algorithms must be developed that do not require information past the currently rendered frame. This paper presents a watercolor-like NPR style for AR applications with some degree of visual coherence.