Unlike large and dangerous industrial robots at production lines in factories that are strictly fenced off, collaborative robots are smaller and safer and can be installed adjacent to human workers and collaborate with them. However, controlling and teaching new moves to collaborative robots can be difficult and time-consuming when using existing methods, such as pressing buttons on a teaching pendant and physically grabbing and moving the robot via direct teaching. We present Robot Telekinesis, a novel robot interaction technique that lets the user remotely control the movement of the end effector of a robot arm with unimanual and bimanual hand gestures that closely resemble handling a physical object. Through formal evaluation, we show that using a teaching pendant is slow and confusing and that direct teaching is fast and intuitive but physically demanding. Robot Telekinesis is as fast and intuitive as direct teaching without the need for physical contact or physical effort.

Hand motion and pen drawing can be intuitive and expressive inputs for professional digital 3D authoring. However, their inherent limitations have hampered wider adoption. 3D sketching using hand motion is rapid but rough, and 3D sketching using pen drawing is delicate but tedious. Our new 3D sketching workflow combines these two in a complementary manner. The user makes quick hand motions in the air to generate approximate 3D shapes, and uses them as scaffolds on which to add details via pen-based 3D sketching on a tablet device. Our air scaffolding technique and corresponding algorithm extract only the intended shapes from unconstrained hand motions. Then, the user sketches 3D ideas by defining sketching planes on these scaffolds while appending new scaffolds, as needed. A user study shows that our progressive and iterative workflow enables more agile 3D sketching compared to ones using either hand motion or pen drawing alone.

In augmented and virtual reality (AR and VR), there may be many 3D planar windows with 2D texts, images, and videos on them. However, managing the position, orientation, and scale of such a window in an immersive 3D workspace can be difficult. Projective Windows strategically uses the absolute and apparent sizes of the window at various stages of the interaction to enable the grabbing, moving, scaling, and releasing of the window in one continuous hand gesture. With it, the user can quickly and intuitively manage and interact with windows in space without any controller hardware or dedicated widget. Through an evaluation, we demonstrate that our technique is performant and preferable, and that projective geometry plays an important role in the design of spatial user interfaces.

Technological advances and socioeconomic disruptions such as self-driving cars, car-sharing services and artificial intelligence assistance may fundamentally alter interactions inside the future car. However, existing design tools and processes geared toward static physical authoring are ill-equipped for such interaction design. We propose a new design workflow that combines experience prototyping methods typically used by the user interface and product design communities with 3D sketching and haptic helper techniques to help automotive designers ideate, prototype, experience and evaluate multi-sensory interactions in a collaborative manner. Using our workflow, designers use 3D sketching to quickly and expressively author 3D shape and motion ideas in space; augment them with tactile and other sensory feedback through physical proxies and other available gadgets; and immediately enact and immersively experience them to progressively explore and develop them.