We propose a framework for learning neural scene representations directly from images, without 3D supervision. Our key insight is that 3D structure can be imposed by ensuring that the learned representation transforms like a real 3D scene. Specifically, we introduce a loss which enforces equivariance of the scene representation with respect to 3D transformations. Our formulation allows us to infer and render scenes in real time while achieving comparable results to models requiring minutes for inference. In addition, we introduce two challenging new datasets for scene representation and neural rendering, including scenes with complex lighting and backgrounds. Through experiments, we show that our model achieves compelling results on these datasets as well as on standard ShapeNet benchmarks.

Repeated stress during adolescence has been shown to decrease cognitive function while increasing rates of anxiety and depression. Eighty-one percent of teens report stress stemming from their school environment and yet, schools are struggling to manage the increase of teen mental health needs. Virtual-reality (VR) has been shown effective at treating post-traumatic stress disorder, phobias, and pain due to its immersive experience. The following paper presents a participatory, human-centered design process utilizing teens as co-designers to develop a VR interaction intended to reduce stress.

We propose a novel interactive system to simplify the process of indoor 3D CAD room modeling. Traditional room modeling methods require users to measure room and furniture dimensions, and manually select models that match the scene from large catalogs. Users then employ a mouse and keyboard interface to construct walls and place the objects in their appropriate locations. In contrast, our system leverages the sensing capabilities of a 3D aware mobile device, recent advances in object recognition, and a novel augmented reality user interface, to capture indoor 3D room models in-situ. With a few taps, a user can mark the surface of an object, take a photo, and the system retrieves and places a matching 3D model into the scene, from a large online database. User studies indicate that this modality is significantly quicker, more accurate, and requires less effort than traditional desktop tools.

We present an interactive system to capture CAD-like 3D models of indoor scenes, on a mobile device. To overcome sensory and computational limitations of the mobile platform, we employ an in situ, semi-automated approach and harness the user's high-level knowledge of the scene to assist the reconstruction and modeling algorithms. The modeling proceeds in two stages: (1) The user captures the 3D shape and dimensions of the room. (2) The user then uses voice commands and an augmented reality sketching interface to insert objects of interest, such as furniture, artwork, doors and windows. Our system recognizes the sketches and add a corresponding 3D model into the scene at the appropriate location. The key contributions of this work are the design of a multi-modal user interface to effectively capture the user's semantic understanding of the scene and the underlying algorithms that process the input to produce useful reconstructions.

In this paper, we present a novel smartphone application designed to easily capture, visualize and reconstruct homes, offices and other indoor scenes. Our application leverages data from smartphone sensors such as the camera, accelerometer, gyroscope and magnetometer to help model the indoor scene. The output of the system is two-fold; first, an interactive visual tour of the scene is generated in real time that allows the user to explore each room and transition between connected rooms. Second, with some basic interactive photogrammetric modeling the system generates a 2D floor plan and accompanying 3D model of the scene, under a Manhattan-world assumption. The approach does not require any specialized equipment or training and is able to produce accurate floor plans.

The cultural heritage of a region is conveyed by both tangible physical artifacts and intangible aspects in the form of stories, dance styles, rituals, etc. Hitherto, the task of creating digital representations for each of these aspects has been addressed in isolation, i.e. using specific media most suited to the artifact such as video, audio, three-dimensional (3D) models, scanning, etc. The challenge of bringing together these separate elements to create a coherent story, however, has remained unaddressed until recently. In this paper we present a unified digital framework that enables the integration of disparate representations of heritage elements into a holistic entity. Our approach results in a compelling and engaging narration that affords a unified user experience. Our solution supports both active (user-controlled explorations) and passive (watching pre-orchestrated narrations) user interactions.