In this study, we explore a new way to complementarily utilize the immersive visual output of VR and the physical haptic input of a smartphone. In particular, we focus on interacting with distant virtual objects using a smartphone in a through-plane manner and present a novel selection technique that overcomes the binocular parallax that occurs in such an arrangement. In our proposed technique, when a user in the stereoscopic viewing mode needs to perform a distant selection, the user brings the fingertip near the screen of the mobile device, triggering a smoothly animated transition to the monoscopic touching mode. Using a novel proof-of-concept implementation that utilizes a transparent acrylic panel, we conducted a user study and found that the proposed technique is significantly quicker, more precise, more direct, and more intuitive compared to the ray casting baseline. Subsequently, we created VR applications that explore the rich and interesting use cases of the proposed technique.

Designing articulated objects is challenging because, unlike with static objects, it requires complex decisions to be made regarding the form, parts, rig, poses, and motion. We present a novel 3D sketching system for rapidly authoring concepts of articulated objects for the early stages of design, when designers make such decisions. Compared to existing CAD software, which focuses on slowly but elaborately producing models consisting of precise surfaces and volumes, our system focuses on quickly but roughly producing models consisting of key curves through a small set of coherent pen and multi-touch gestures. We found that professional designers could easily learn and use our system and author compelling concepts in a short time, showing that 3D sketching can be extended to designing articulated objects and is generally applicable in film, animation, game, and product design.

Unlike large and dangerous industrial robots at production lines in factories that are strictly fenced off, collaborative robots are smaller and safer and can be installed adjacent to human workers and collaborate with them. However, controlling and teaching new moves to collaborative robots can be difficult and time-consuming when using existing methods, such as pressing buttons on a teaching pendant and physically grabbing and moving the robot via direct teaching. We present Robot Telekinesis, a novel robot interaction technique that lets the user remotely control the movement of the end effector of a robot arm with unimanual and bimanual hand gestures that closely resemble handling a physical object. Through formal evaluation, we show that using a teaching pendant is slow and confusing and that direct teaching is fast and intuitive but physically demanding. Robot Telekinesis is as fast and intuitive as direct teaching without the need for physical contact or physical effort.

Hand motion and pen drawing can be intuitive and expressive inputs for professional digital 3D authoring. However, their inherent limitations have hampered wider adoption. 3D sketching using hand motion is rapid but rough, and 3D sketching using pen drawing is delicate but tedious. Our new 3D sketching workflow combines these two in a complementary manner. The user makes quick hand motions in the air to generate approximate 3D shapes, and uses them as scaffolds on which to add details via pen-based 3D sketching on a tablet device. Our air scaffolding technique and corresponding algorithm extract only the intended shapes from unconstrained hand motions. Then, the user sketches 3D ideas by defining sketching planes on these scaffolds while appending new scaffolds, as needed. A user study shows that our progressive and iterative workflow enables more agile 3D sketching compared to ones using either hand motion or pen drawing alone.

In augmented and virtual reality (AR and VR), there may be many 3D planar windows with 2D texts, images, and videos on them. However, managing the position, orientation, and scale of such a window in an immersive 3D workspace can be difficult. Projective Windows strategically uses the absolute and apparent sizes of the window at various stages of the interaction to enable the grabbing, moving, scaling, and releasing of the window in one continuous hand gesture. With it, the user can quickly and intuitively manage and interact with windows in space without any controller hardware or dedicated widget. Through an evaluation, we demonstrate that our technique is performant and preferable, and that projective geometry plays an important role in the design of spatial user interfaces.

Technological advances and socioeconomic disruptions such as self-driving cars, car-sharing services and artificial intelligence assistance may fundamentally alter interactions inside the future car. However, existing design tools and processes geared toward static physical authoring are ill-equipped for such interaction design. We propose a new design workflow that combines experience prototyping methods typically used by the user interface and product design communities with 3D sketching and haptic helper techniques to help automotive designers ideate, prototype, experience and evaluate multi-sensory interactions in a collaborative manner. Using our workflow, designers use 3D sketching to quickly and expressively author 3D shape and motion ideas in space; augment them with tactile and other sensory feedback through physical proxies and other available gadgets; and immediately enact and immersively experience them to progressively explore and develop them.

Human communication significantly relies on the expressivity of their body movements. Based on these body language experiences, humans tend to extract meanings even from movements of objects. This paper begins with the above human tendencies to create a design method that can help product designers make their products move to communicate. As a research vehicle, we created a robotic torso prototype and utilized it to collaborate with movement experts, and listed up possible expressive movement components. We then built a mapping matrix that links these movements to general product messages. A method which utilizes this mapping matrix was developed to help designers determine a set of effective movements that can communicate specific product messages. Lastly, a design workshop was conducted to identify the usefulness of the proposed method. We expect the procedures and findings of this study to help researchers and designers approach affective user experience through product movement design.

Binocular parallax is a problem for any interaction system that has a transparent display and objects behind it, as users will see duplicated and overlapped images. In this note, we propose a quantitative measure called Binocular Selectability Discriminant (BSD) to predict the ability of the user to perform selection task in such a setup. In addition, we propose a technique called Binocular Cursor (BC) which takes advantage of this duplicating and overlapping phenomenon, rather than being hampered by it, to resolve binocular selection ambiguity by visualizing the correct selection point. An experiment shows that selection with BC is not slower than monocular selection, and that it can be significantly more precise, depending on the design of BC.

Following the recent remarkable successes of crowdsourcing, there have been attempts to apply it to design. However a design problem is often too complex and difficult to break down into simpler, distributable tasks as required by the conventional crowdsourcing model. In this paper, we present Crowd vs. Crowd (CvC), a novel design crowdsourcing method, where several design teams made up of designers and crowd compete with each other. In each team, a designer coordinates effective communication between the crowd members and takes responsibility for the final design output, and the crowd contributes at different stages of design. We conducted an initial evaluation of CvC in comparison with other collaborative design methods, and found that: CvC can attract more people to participate; the crowd can make useful contribution in CvC; CvC can produce competent design outputs. We then applied CvC to two real-life design problems: first, designing a new logo for a university department; second, for a small tech company. With quantitative and qualitative analyses on these applications, we observed that the elements of competition and collaboration helped to sustain the crowd's motivation to participate, and to produce quality design outcomes with higher level of satisfaction for the stakeholders.

We present PhantomPen, a direct pen input device whose pen head is virtualized onto the tablet display surface and visually connected to a graspable pen barrel in order to achieve digital drawing free from pen occlusion and visual parallax. As the pen barrel approaches the display, the virtual pen head smoothly appears as if the rendered virtual pen head and the physical pen barrel are in unity. The virtual pen head provides visual feedback by changing its virtual form according to pen type, color, and thickness while the physical pen tip, hidden in the user's sight, provides tactile feedback. Three experiments were carefully designed based on an analysis of drawings by design professionals and observations of design drawing classes. With these experiments that simulate natural drawing we proved significant performance advantages of PhantomPen. PhantomPen was at least as usable as the normal stylus in basic line drawing, and was 17 % faster in focus region drawing (26% faster in extreme focus region drawing). PhantomPen also reduced error rate by 40 % in a typical drawing setup where users have to manage a complex combination of pen and stroke properties.