We present "Push-That-There", an interaction method and system enabling multimodel object-level user interaction with multi-robot system to autonomously and collectively manipulate objects on tabletop surfaces, inspired by "Put-That-There". Rather than requiring users to instruct individual robots, users directly specify how they want the objects to be moved, and the system responds by autonomously moving objects via our generalizable multi-robot control algorithm. The system is combined with various user instruction modalities, including gestures, GUI, tangible manipulation, and speech, allowing users to intuitively create object-level instruction. We outline a design space, highlight interaction design opportunities facilitated by "Push-That-There", and provide an evaluation to assess our system's technical capabilities. While other recent HCI research has studied interaction using multi-robot system (e.g. Swarm UIs), our contribution is in the design and technical implementation of intuitive object-level interaction for multi-robot system that allows users to work at a high level, rather than needing to focus on the movements of individual robots.

We introduce DrawTalking, an approach to building and controlling interactive worlds by sketching and speaking while telling stories. It emphasizes user control and flexibility, and gives programming-like capability without requiring code. An early open-ended study with our prototype shows that the mechanics resonate and are applicable to many creative-exploratory use cases, with the potential to inspire and inform research in future natural interfaces for creative exploration and authoring.

Terrain generation and authoring in Virtual Reality (VR) offers unique benefits, including 360-degree views, improved spatial perception, immersive and intuitive design experience and natural input modalities. Yet even in VR it can be challenging to integrate natural input modalities, preserve artistic controls and lower the effort of landscape prototyping. To tackle these challenges, we present our VR-based terrain generation and authoring system, which utilizes hand tracking and a generative model to allow users to quickly prototype natural landscapes, such as mountains, mesas, canyons and volcanoes. Via positional hand tracking and hand gesture detection, users can use their hands to draw mid-air strokes to indicate desired shapes for the landscapes. A Conditional Generative Adversarial Network trained by using real-world terrains and their height maps then helps to generate a realistic landscape which combines features of training data and the mid-air strokes. In addition, users can use their hands to further manipulate their mid-air strokes to edit the landscapes. In this paper, we explore this design space and present various scenarios of terrain generation. Additionally, we evaluate our system across a diverse user base that varies in VR experience and professional background. The study results indicate that our system is feasible, user-friendly and capable of fast prototyping.

While using VR, efficient text entry is a challenge: users cannot easily locate standard physical keyboards, and keys are often out of reach, e.g. when standing. We present TapGazer, a text entry system where users type by tapping their fingers in place. Users can tap anywhere as long as the identity of each tapping finger can be detected with sensors. Ambiguity between different possible input words is resolved by selecting target words with gaze. If gaze tracking is unavailable, ambiguity is resolved by selecting target words with additional taps. We evaluated TapGazer for seated and standing VR: seated novice users using touchpads as tap surfaces reached 44.81 words per minute (WPM), 79.17% of their QWERTY typing speed. Standing novice users tapped on their thighs with touch-sensitive gloves, reaching 45.26 WPM (71.91%). We analyze TapGazer with a theoretical performance model and discuss its potential for text input in future AR scenarios.

Exploring large virtual environments, such as cities, is a central task in several domains, such as gaming and urban planning. VR systems can greatly help this task by providing an immersive experience; however, a common issue with viewing and navigating a city in the traditional sense is that users can either obtain a local or a global view, but not both at the same time, requiring them to continuously switch between perspectives, losing context and distracting them from their analysis. In this article, our goal is to allow users to navigate to points of interest without changing perspectives. To accomplish this, we design an intuitive navigation interface that takes advantage of the strong sense of spatial presence provided by VR. We supplement this interface with a perspective that warps the environment, called UrbanRama, based on a cylindrical projection, providing a mix of local and global views. The design of this interface was performed as an iterative process in collaboration with architects and urban planners. We conducted a qualitative and a quantitative pilot user study to evaluate UrbanRama and the results indicate the effectiveness of our system in reducing perspective changes, while ensuring that the warping doesn't affect distance and orientation perception.

Communication software such as Clubhouse and Zoom has evolved to be an integral part of many people’s daily lives. However, due to network bandwidth constraints and concerns about privacy, cameras in video conferencing are often turned off by participants. This leads to a situation in which people can only see each others’ profile images, which is essentially an audio-only experience. Even when switched on, video feeds do not provide accurate cues as to who is talking to whom. This paper introduces GazeChat, a remote communication system that visually represents users as gaze-aware 3D profile photos. This satisfies users’ privacy needs while keeping online conversations engaging and efficient. GazeChat uses a single webcam to track whom any participant is looking at, then uses neural rendering to animate all participants’ profile images so that participants appear to be looking at each other. We have conducted a remote user study (N=16) to evaluate GazeChat in three conditions: audio conferencing with profile photos, GazeChat, and video conferencing. Based on the results of our user study, we conclude that GazeChat maintains the feeling of presence while preserving more privacy and requiring lower bandwidth than video conferencing, provides a greater level of engagement than to audio conferencing, and helps people to better understand the structure of their conversation.

Among its many promising applications, Virtual Reality (VR) can simulate diverse real-life scenarios and therefore help experimenters assess individuals’ gait performance (i.e., walking) under controlled functional contexts. VR-based gait assessment may provide low-risk, reproducible and controlled virtual environments, enabling experimenters to investigate underlying causes for imbalance by manipulating experimental conditions such as multi-sensory loads, mental processing loads (cognitive load), and/or motor tasks. We present a low-cost novel VR gait assessment system that simulates virtual obstacles, visual, auditory, and cognitive loads while using motion tracking to assess participants’ walking performance. The system utilizes in-situ spatial visualization for trial playback and instantaneous outcome measures which enable experimenters and participants to observe and interpret their performance. The trial playback can visualize any moment in the trial with embodied graphic segments including the head, waist, and feet. It can also replay two trials at the same time frame for trial-to-trial comparison, which helps visualize the impact of different experimental conditions. The outcome measures, i.e., the metrics related to walking performance, are calculated in real-time and displayed as data graphs in VR. The system can help experimenters get specific gait information on balance performance beyond a typical clinical gait test, making it clinically relevant and potentially applicable to gait rehabilitation. We conducted a feasibility study with physical therapy students, research graduate students, and licensed physical therapists. They evaluated the system and provided feedback on the outcome measures, the spatial visualizations, and the potential use of the system in the clinic. The study results indicate that the system was feasible for gait assessment, and the immediate spatial visualization features were seen as clinically relevant and useful. Limitations and considerations for future work are discussed.

Writing or sketching on whiteboards is an essential part of collaborative discussions in business meetings, reading groups, design sessions, and interviews. However, prior work in collaborative virtual reality (VR) systems has rarely explored the design space of multi-user layouts and interaction modes with virtual whiteboards. In this paper, we present CollaboVR, a reconfigurable framework for both co-located and geographically dispersed multi-user communication in VR. Our system unleashes users’ creativity by sharing freehand drawings, converting 2D sketches into 3D models, and generating procedural animations in real-time. To minimize the computational expense for VR clients, we leverage a cloud architecture in which the computational expensive application (Chalktalk) is hosted directly on the servers, with results being simultaneously streamed to clients. We have explored three custom layouts – integrated, mirrored, and projective – to reduce visual clutter, increase eye contact, or adapt different use cases. To evaluate CollaboVR, we conducted a within-subject user study with 12 participants. Our findings reveal that users appreciate the custom configurations and real-time interactions provided by CollaboVR. We have open sourced CollaboVR at https://github.com/snowymo/CollaboVR to facilitate future research and development of natural user interfaces and real-time collaborative systems in virtual and augmented reality.

The virtual reality ecosystem has gained momentum in the gaming, entertainment, and enterprise markets, but is hampered by limitations in concurrent user count, throughput, and accessibility to mass audiences. Based on our analysis of the current state of the virtual reality ecosystem and relevant aspects of traditional media, we propose a set of design hypotheses for practical and effective seated virtual reality experiences of scale. Said hypotheses manifest in the Collective Audience Virtual Reality Nexus (CAVRN), a framework and management system for large-scale (30+ user) virtual reality deployment in a theater-like physical setting. A mixed methodology study of CAVE, an experience implemented using CAVRN, generated rich insights into the proposed hypotheses. We discuss the implications of our findings on content design, audience representation, and audience interaction.

Virtual Reality enables users to explore content whose physics are only limited by our creativity. Such limitless environments provide us with many opportunities to explore innovative ways to support productivity and collaboration. We present Spacetime, a scene editing tool built from the ground up to explore the novel interaction techniques that empower single user interaction while maintaining fluid multi-user collaboration in immersive virtual environment. We achieve this by introducing three novel interaction concepts: the Container, a new interaction primitive that supports a rich set of object manipulation and environmental navigation techniques, Parallel Objects, which enables parallel manipulation of objects to resolve interaction conflicts and support design workflows, and Avatar Objects, which supports interaction among multiple users while maintaining an individual users' agency. Evaluated by professional Virtual Reality designers, Spacetime supports powerful individual and fluid collaborative workflows.

Efficient simulation of the Navier-Stokes equations for fluid flow is a long standing problem in applied mathematics, for which state-of-the-art methods require large compute resources. In this work, we propose a data-driven approach that leverages the approximation power of deep-learning with the precision of standard solvers to obtain fast and highly realistic simulations. Our method solves the incompressible Euler equations using the standard operator splitting method, in which a large sparse linear system with many free parameters must be solved. We use a Convolutional Network with a highly tailored architecture, trained using a novel unsupervised learning framework to solve the linear system. We present real-time 2D and 3D simulations that outperform recently proposed data-driven methods; the obtained results are realistic and show good generalization properties.

clayodor (\klei-o-dor\) is a clay-like malleable material that changes smell based on user manipulation of its shape. This work explores the tangibility of shape changing materials to capture smell, an ephemeral and intangible sensory input. We present the design of a proof-of-concept prototype, and discussions on the challenges of navigating smell though form.

We present Mechanical Force Redistribution (MFR): a method of sensing which creates an anti-aliased image of forces applied to a surface. This technique mechanically focuses the force from a surface onto adjacent discrete forcels (force sensing cells) by way of protrusions (small bumps or pegs), allowing for high-accuracy interpolation between adjacent discrete forcels. MFR works with any force transducing technique or material, including force variable resistive inks, piezoelectric materials and capacitive force plates. MFR sensors can be tiled such that the signal is continuous across contiguous tiles. By minimizing active materials and computational complexity, MFR makes large-format interactive walls, collaborative tabletops and high-resolution floor tiles possible and economically feasible.

We present a novel method for real-time continuous pose recovery of markerless complex articulable objects from a single depth image. Our method consists of the following stages: a randomized decision forest classifier for image segmentation, a robust method for labeled dataset generation, a convolutional network for dense feature extraction, and finally an inverse kinematics stage for stable real-time pose recovery. As one possible application of this pipeline, we show state-of-the-art results for real-time puppeteering of a skinned hand-model.

In this paper we describe ClayVision, a new quasi-immersive urban navigation system that rethinks the design conventions of existing Augmented Reality (AR) applications, by aggressively incorporating knowledge from non-Computer Science fields - namely Information Design and Urban Planning. Instead of the prevailing approach of pasting "information bubbles" onto the existing urban scenery, ClayVision communicates through real-time 3D transformations of city elements. In other words, the system dynamically probes and reassembles the city into a better-designed copy of the original, that is both easier to navigate and tailored to suit the user's needs and preferences. We provide extensive discussions that cover the technical details of the system, the types of city-morphing operations that can be effectively applied, and what people's experiences will be in the newly "elastic" city.

What will the interface between people and computers look like in five years? In ten years? In twenty five years? Will we still have screens? Keyboards? Will we all be seeing Princess Leia in a beam of light? Based on current trends and inspired guesswork, we will go together on a tour of the future.

Abstract Procedural noise functions are widely used in computer graphics, from off‐line rendering in movie production to interactive video games. The ability to add complex and intricate details at low memory and authoring cost is one of its main attractions. This survey is motivated by the inherent importance of noise in graphics, the widespread use of noise in industry and the fact that many recent research developments justify the need for an up‐to‐date survey. Our goal is to provide both a valuable entry point into the field of procedural noise functions, as well as a comprehensive view of the field to the informed reader. In this report, we cover procedural noise functions in all their aspects. We outline recent advances in research on this topic, discussing and comparing recent and well‐established methods. We first formally define procedural noise functions based on stochastic processes and then classify and review existing procedural noise functions. We discuss how procedural noise functions are used for modelling and how they are applied to surfaces. We then introduce analysis tools and apply them to evaluate and compare the major approaches to noise generation. We finally identify several directions for future work.

Recently, there has been great interest in multi-touch interfaces. Such devices have taken the form of camera-based systems such as Microsoft Surface [de los Reyes et al. 2007] and Perceptive Pixel's FTIR Display [Han 2005] as well as hand-held devices using capacitive sensors such as the Apple iPhone [Jobs et al. 2008]. However, optical systems are inherently bulky while most capacitive systems are only practical in small form factors and are limited in their application since they respond only to human touch and are insensitive to variations in pressure [Westerman 1999]. We have created the UnMousePad, a flexible and inexpensive multitouch input device based on a newly developed pressure-sensing principle called Interpolating Force Sensitive Resistance. IFSR sensors can acquire high-quality anti-aliased pressure images at high frame rates. They can be paper-thin, flexible, and transparent and can easily be scaled to fit on a portable device or to cover an entire table, floor or wall. The UnMousePad can sense three orders of magnitude of pressure variation, and can be used to distinguish multiple fingertip touches while simultaneously tracking pens and styli with a positional accuracy of 87 dpi, and can sense the pressure distributions of objects placed on its surface. In addition to supporting multi-touch interaction, IFSR is a general pressure imaging technology that can be incorporated into shoes, tennis racquets, hospital beds, factory assembly lines and many other applications. The ability to measure high-quality pressure images at low cost has the potential to dramatically improve the way that people interact with machines and the way that machines interact with the world.

We describe a new technique for measuring the bidirectional texture function (BTF) of a surface that requires no mechanical movement, can measure surfaces in situ under arbitrary lighting conditions, and can be made small, portable and inexpensive. The enabling innovation is the use of a tapered kaleidoscope, which allows a camera to view the same surface sample simultaneously from many directions. Similarly, the surface can be simultaneously illuminated from many directions, using only a single structured light source. We describe the techniques of construction and measurement, and we show experimental results.

Two deficiencies in the original Noise algorithm are corrected: second order interpolation discontinuity and unoptimal gradient computation. With these defects corrected, Noise both looks better and runs faster. The latter change also makes it easier to define a uniform mathematical reference standard.

We present a display device which solves a long-standing problem: to give a true stereoscopic view of simulated objects, without artifacts, to a single unencumbered observer, while allowing the observer to freely change position and head rotation. Based on a novel combination of temporal and spatial multiplexing, this technique will enable artifact-free stereo to become a standard feature of display screens, without requiring the use of special eyewear. The availability of this technology may significantly impact CAD and CHI applications, as well as entertainment graphics. The underlying algorithms and system architecture are described, as well as hardware and software aspects of the implementation.

Nested User Interface Components combine the concepts of Zooming User Interfaces (ZUIs) with recursive nesting of active graphical user interface widgets. The resulting system of recursively nesting interface components has a number of desirable properties. The level of detail of the view of any widget component and its children, as well as the responsiveness of that component to the user's actions, can be tuned to the current visible size of that component on the screen.

No abstract available.

Improv is a system for the creation of real−time behavior−based animated actors. There have been several recent efforts to build network distributed autonomous agents. But in general these efforts do not focus on the author’s view. To create rich interactive worlds inhabited by believable animated actors, authors need the proper tools. Improv provides tools to create actors that respond to users and to each other in real−time, with personalities and moods consistent with the author’s goals and intentions. Improv consists of two subsystems. The first subsystem is an Animation Engine that uses procedural techniques to enable authors to create layered, continuous, non−repetitive motions and smooth transitions between them. The second subsystem is a Behavior Engine that enables authors to create sophisticated rules governing how actors communicate, change, and make decisions. The combined system provides an integrated set of tools for authoring the minds and bodies of interactive actors. The system uses an english−style scripting language so that creative experts who are not primarily programmers can create powerful interactive applications.

Article Free Access Share on Live paint: painting with procedural multiscale textures Authors: Ken Perlin Media Research Laboratory, Courant Institute of Mathematical Sciences, New York University Media Research Laboratory, Courant Institute of Mathematical Sciences, New York UniversityView Profile , Luiz Velho IMPA, Instituto de Matemática Pura e Aplicada IMPA, Instituto de Matemática Pura e AplicadaView Profile Authors Info & Claims SIGGRAPH '95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniquesSeptember 1995 Pages 153–160https://doi.org/10.1145/218380.218437Published:15 September 1995Publication History 27citation917DownloadsMetricsTotal Citations27Total Downloads917Last 12 Months66Last 6 weeks19 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Building on principles from prior work on procedural texture synthesis (K. Perlin, 1985), we are able to create remarkably lifelike, responsively animated characters in real time. Rhythmic and stochastic noise functions are used to define time varying parameters that drive computer generated puppets. Because we are conveying just the "texture" of motion, we are able to avoid computation of dynamics and constraint solvers. The subjective impression of dynamics and other subtle influences on motion can be conveyed with great visual realism by properly tuned expressions containing pseudo random noise functions. For example, we can make a character appear to be dynamically balancing herself, to appear nervous, or to be gesturing in a particular way. Each move has an internal rhythm, and transitions between moves are temporally constrained so that "impossible" transitions are precluded. For example, if while the character is walking we specify a dance turn, the character will always step into the turn onto the correct weight bearing foot. An operator can make a character perform a properly connected sequence of actions, while conveying particular moods and attitudes, merely by pushing buttons at a high level. Potential uses of such high level "textural" approaches to computer graphic simulation include role playing games, simulated conferences, "clip animation", graphical front ends for MUDs, and synthetic performances.>

Article Pad: an alternative approach to the computer interface Share on Authors: Ken Perlin View Profile , David Fox View Profile Authors Info & Claims SIGGRAPH '93: Proceedings of the 20th annual conference on Computer graphics and interactive techniquesSeptember 1993 Pages 57–64https://doi.org/10.1145/166117.166125Online:01 September 1993Publication History 281citation2,436DownloadsMetricsTotal Citations281Total Downloads2,436Last 12 Months58Last 6 weeks9 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We introduce the concept of a Pixel Stream Editor. This forms the basis for an interactive synthesizer for designing highly realistic Computer Generated Imagery. The designer works in an interactive Very High Level programming environment which provides a very fast concept/implement/view iteration cycle.Naturalistic visual complexity is built up by composition of non-linear functions, as opposed to the more conventional texture mapping or growth model algorithms. Powerful primitives are included for creating controlled stochastic effects. We introduce the concept of "solid texture" to the field of CGI.We have used this system to create very convincing representations of clouds, fire, water, stars, marble, wood, rock, soap films and crystal. The algorithms created with this paradigm are generally extremely fast, highly realistic, and asynchronously parallelizable at the pixel level.