Unlike large and dangerous industrial robots at production lines in factories that are strictly fenced off, collaborative robots are smaller and safer and can be installed adjacent to human workers and collaborate with them. However, controlling and teaching new moves to collaborative robots can be difficult and time-consuming when using existing methods, such as pressing buttons on a teaching pendant and physically grabbing and moving the robot via direct teaching. We present Robot Telekinesis, a novel robot interaction technique that lets the user remotely control the movement of the end effector of a robot arm with unimanual and bimanual hand gestures that closely resemble handling a physical object. Through formal evaluation, we show that using a teaching pendant is slow and confusing and that direct teaching is fast and intuitive but physically demanding. Robot Telekinesis is as fast and intuitive as direct teaching without the need for physical contact or physical effort.

Hand motion and pen drawing can be intuitive and expressive inputs for professional digital 3D authoring. However, their inherent limitations have hampered wider adoption. 3D sketching using hand motion is rapid but rough, and 3D sketching using pen drawing is delicate but tedious. Our new 3D sketching workflow combines these two in a complementary manner. The user makes quick hand motions in the air to generate approximate 3D shapes, and uses them as scaffolds on which to add details via pen-based 3D sketching on a tablet device. Our air scaffolding technique and corresponding algorithm extract only the intended shapes from unconstrained hand motions. Then, the user sketches 3D ideas by defining sketching planes on these scaffolds while appending new scaffolds, as needed. A user study shows that our progressive and iterative workflow enables more agile 3D sketching compared to ones using either hand motion or pen drawing alone.

In augmented and virtual reality (AR and VR), there may be many 3D planar windows with 2D texts, images, and videos on them. However, managing the position, orientation, and scale of such a window in an immersive 3D workspace can be difficult. Projective Windows strategically uses the absolute and apparent sizes of the window at various stages of the interaction to enable the grabbing, moving, scaling, and releasing of the window in one continuous hand gesture. With it, the user can quickly and intuitively manage and interact with windows in space without any controller hardware or dedicated widget. Through an evaluation, we demonstrate that our technique is performant and preferable, and that projective geometry plays an important role in the design of spatial user interfaces.

Background modeling in complex scenes is a challenging problem. In this paper, a novel background subtraction method is proposed to address it. First, the textures are modeled with local compact binary patterns (LCBP), which have excellent robustness, strong discriminative power, and fast computation speed. To make LCBP more effective to appearance changes in complex scenarios, spatiotemporal local compact binary patterns (STLCBP) are then considered in which spatial texture information and temporal motion information are combined together. Multiple color spaces are also presented to separate foreground pixels more accurately from the background. To our knowledge, this is the first time that LCBP have been used for background modeling. Extensive experimental results on a widely used dataset clearly show that the proposed method outperforms other state-of-the-art methods and works effectively in complex scenes.

Technological advances and socioeconomic disruptions such as self-driving cars, car-sharing services and artificial intelligence assistance may fundamentally alter interactions inside the future car. However, existing design tools and processes geared toward static physical authoring are ill-equipped for such interaction design. We propose a new design workflow that combines experience prototyping methods typically used by the user interface and product design communities with 3D sketching and haptic helper techniques to help automotive designers ideate, prototype, experience and evaluate multi-sensory interactions in a collaborative manner. Using our workflow, designers use 3D sketching to quickly and expressively author 3D shape and motion ideas in space; augment them with tactile and other sensory feedback through physical proxies and other available gadgets; and immediately enact and immersively experience them to progressively explore and develop them.

We present SketchingWithHands, a 3D sketching system that incorporates a hand-tracking sensor. The system enables product designers to easily capture desired hand postures from a first-person point of view at any time and to use the captured hand information to explore handheld product concepts by 3D sketching while keeping the proper scale and usage of the products. Based on the analysis of design practices and drawing skills in the art and design literature, we suggest novel ideas for efficiently acquiring hand postures (palm-pinning widget, front and center mirrors, responsive spangles), for quickly creating and easily adjusting sketch planes (modified tick-triggered, orientable and shiftable sketch planes), for appropriately starting 3D sketching products with hand information (hand skeleton, grip axis), and for practically increasing user throughput (intensifier, rough and precise erasers)---all of which are coherently and consistently integrated in our system. A user test by ten industrial design students and an in-depth discussion show that our system is both useful and usable in designing handheld products.