In this study, we explore a new way to complementarily utilize the immersive visual output of VR and the physical haptic input of a smartphone. In particular, we focus on interacting with distant virtual objects using a smartphone in a through-plane manner and present a novel selection technique that overcomes the binocular parallax that occurs in such an arrangement. In our proposed technique, when a user in the stereoscopic viewing mode needs to perform a distant selection, the user brings the fingertip near the screen of the mobile device, triggering a smoothly animated transition to the monoscopic touching mode. Using a novel proof-of-concept implementation that utilizes a transparent acrylic panel, we conducted a user study and found that the proposed technique is significantly quicker, more precise, more direct, and more intuitive compared to the ray casting baseline. Subsequently, we created VR applications that explore the rich and interesting use cases of the proposed technique.

Designing articulated objects is challenging because, unlike with static objects, it requires complex decisions to be made regarding the form, parts, rig, poses, and motion. We present a novel 3D sketching system for rapidly authoring concepts of articulated objects for the early stages of design, when designers make such decisions. Compared to existing CAD software, which focuses on slowly but elaborately producing models consisting of precise surfaces and volumes, our system focuses on quickly but roughly producing models consisting of key curves through a small set of coherent pen and multi-touch gestures. We found that professional designers could easily learn and use our system and author compelling concepts in a short time, showing that 3D sketching can be extended to designing articulated objects and is generally applicable in film, animation, game, and product design.

Unlike large and dangerous industrial robots at production lines in factories that are strictly fenced off, collaborative robots are smaller and safer and can be installed adjacent to human workers and collaborate with them. However, controlling and teaching new moves to collaborative robots can be difficult and time-consuming when using existing methods, such as pressing buttons on a teaching pendant and physically grabbing and moving the robot via direct teaching. We present Robot Telekinesis, a novel robot interaction technique that lets the user remotely control the movement of the end effector of a robot arm with unimanual and bimanual hand gestures that closely resemble handling a physical object. Through formal evaluation, we show that using a teaching pendant is slow and confusing and that direct teaching is fast and intuitive but physically demanding. Robot Telekinesis is as fast and intuitive as direct teaching without the need for physical contact or physical effort.

Hand motion and pen drawing can be intuitive and expressive inputs for professional digital 3D authoring. However, their inherent limitations have hampered wider adoption. 3D sketching using hand motion is rapid but rough, and 3D sketching using pen drawing is delicate but tedious. Our new 3D sketching workflow combines these two in a complementary manner. The user makes quick hand motions in the air to generate approximate 3D shapes, and uses them as scaffolds on which to add details via pen-based 3D sketching on a tablet device. Our air scaffolding technique and corresponding algorithm extract only the intended shapes from unconstrained hand motions. Then, the user sketches 3D ideas by defining sketching planes on these scaffolds while appending new scaffolds, as needed. A user study shows that our progressive and iterative workflow enables more agile 3D sketching compared to ones using either hand motion or pen drawing alone.

In augmented and virtual reality (AR and VR), there may be many 3D planar windows with 2D texts, images, and videos on them. However, managing the position, orientation, and scale of such a window in an immersive 3D workspace can be difficult. Projective Windows strategically uses the absolute and apparent sizes of the window at various stages of the interaction to enable the grabbing, moving, scaling, and releasing of the window in one continuous hand gesture. With it, the user can quickly and intuitively manage and interact with windows in space without any controller hardware or dedicated widget. Through an evaluation, we demonstrate that our technique is performant and preferable, and that projective geometry plays an important role in the design of spatial user interfaces.

Technological advances and socioeconomic disruptions such as self-driving cars, car-sharing services and artificial intelligence assistance may fundamentally alter interactions inside the future car. However, existing design tools and processes geared toward static physical authoring are ill-equipped for such interaction design. We propose a new design workflow that combines experience prototyping methods typically used by the user interface and product design communities with 3D sketching and haptic helper techniques to help automotive designers ideate, prototype, experience and evaluate multi-sensory interactions in a collaborative manner. Using our workflow, designers use 3D sketching to quickly and expressively author 3D shape and motion ideas in space; augment them with tactile and other sensory feedback through physical proxies and other available gadgets; and immediately enact and immersively experience them to progressively explore and develop them.

We present SketchingWithHands, a 3D sketching system that incorporates a hand-tracking sensor. The system enables product designers to easily capture desired hand postures from a first-person point of view at any time and to use the captured hand information to explore handheld product concepts by 3D sketching while keeping the proper scale and usage of the products. Based on the analysis of design practices and drawing skills in the art and design literature, we suggest novel ideas for efficiently acquiring hand postures (palm-pinning widget, front and center mirrors, responsive spangles), for quickly creating and easily adjusting sketch planes (modified tick-triggered, orientable and shiftable sketch planes), for appropriately starting 3D sketching products with hand information (hand skeleton, grip axis), and for practically increasing user throughput (intensifier, rough and precise erasers)---all of which are coherently and consistently integrated in our system. A user test by ten industrial design students and an in-depth discussion show that our system is both useful and usable in designing handheld products.

With the ever increasing amount of digital information, users desire more screen real estate to process daily desktop computing work, and might well benefit from using a large high-resolution display for information management. Unfortunately, we know very little about users' behaviors when using such a display for daily computing, and current user interfaces are mainly designed for normal-sized desktop monitors, which might not well suit a large display. In this article, we first present a longitudinal study that investigates how users manage overflowing digital information on a wall-sized display in a personal desktop computing context by comparing it with single and dual desktop monitors. Results showed users' unanimous preferences of working on a large display and revealed large-display users' unique activity patterns of managing windows. Guided by the study results, we designed a set of interaction techniques that provide greater flexibility in managing multiple windows. They include facile methods for selecting, moving, and resizing multiple windows using an active window boundary called Fringe, rearranging selected windows using multi- and single-window marking menus, packing/unpacking the selected windows using easily activated icons, and freely adjusting the order of overlapping windows with a Jab-to-Lift operation. We coherently integrated these techniques with traditional operations in a large-display window management prototype called WallTop. Two rounds of usability testing showed that users can quickly and easily learn the new interaction techniques and apply them to realistic window management tasks on a large display with increased efficiency.

Human communication significantly relies on the expressivity of their body movements. Based on these body language experiences, humans tend to extract meanings even from movements of objects. This paper begins with the above human tendencies to create a design method that can help product designers make their products move to communicate. As a research vehicle, we created a robotic torso prototype and utilized it to collaborate with movement experts, and listed up possible expressive movement components. We then built a mapping matrix that links these movements to general product messages. A method which utilizes this mapping matrix was developed to help designers determine a set of effective movements that can communicate specific product messages. Lastly, a design workshop was conducted to identify the usefulness of the proposed method. We expect the procedures and findings of this study to help researchers and designers approach affective user experience through product movement design.

Binocular parallax is a problem for any interaction system that has a transparent display and objects behind it, as users will see duplicated and overlapped images. In this note, we propose a quantitative measure called Binocular Selectability Discriminant (BSD) to predict the ability of the user to perform selection task in such a setup. In addition, we propose a technique called Binocular Cursor (BC) which takes advantage of this duplicating and overlapping phenomenon, rather than being hampered by it, to resolve binocular selection ambiguity by visualizing the correct selection point. An experiment shows that selection with BC is not slower than monocular selection, and that it can be significantly more precise, depending on the design of BC.

Following the recent remarkable successes of crowdsourcing, there have been attempts to apply it to design. However a design problem is often too complex and difficult to break down into simpler, distributable tasks as required by the conventional crowdsourcing model. In this paper, we present Crowd vs. Crowd (CvC), a novel design crowdsourcing method, where several design teams made up of designers and crowd compete with each other. In each team, a designer coordinates effective communication between the crowd members and takes responsibility for the final design output, and the crowd contributes at different stages of design. We conducted an initial evaluation of CvC in comparison with other collaborative design methods, and found that: CvC can attract more people to participate; the crowd can make useful contribution in CvC; CvC can produce competent design outputs. We then applied CvC to two real-life design problems: first, designing a new logo for a university department; second, for a small tech company. With quantitative and qualitative analyses on these applications, we observed that the elements of competition and collaboration helped to sustain the crowd's motivation to participate, and to produce quality design outcomes with higher level of satisfaction for the stakeholders.

Advances in ubicomp technology are enabling the devel-opment of products that move in affective ways. However, there is insufficient empirical knowledge to encourage such designs. As research through design, we built three proto-types of standing-type kinetic products to conduct user experience (UX) field studies with visceral, behavioral, and reflective perspectives. Tasks, the users' body reactions, and their feelings were measured and interpreted to uncover features of a desirable UX with moving products. The find-ings and discussions contribute to the ubicomp community by expanding the design space for moving products and inspiring the community with practical applications.

In this paper we introduce CheekTouch, an affective audio-tactile communication technique that transmits multi-finger touch gestures applied on a sender's mobile phone to a receiver's cheek in real time during a call. We made a pair of CheekTouch prototypes each with a multi-touch screen and vibrotactile display to enable bidirectional touch delivery. We observed four romantic couples in their twenties using our prototype system in a lab setting over five consecutive days, and analyzed how CheekTouch affected their non-verbal and emotional communication. The results of the user study showed that CheekTouch could effectively support audio-tactile communication in various ways - persuading, conveying status, delivering information, emphasizing emotion/words, calling for attention, and being playful.

We present an air-contained display medium that can be directly deformed and spatially moved by various physical interaction techniques for interactive games. We first investigated familiar objects in our everyday lives that allow users to easily anticipate the idea of exertion interaction. We then introduce a novel concept of interactive medium, dubbed Inflated Roly-Poly, which consists of an inflated body with a roly-poly structure. This device receives physical input, provides passive haptic feedback and allows spatial interaction. We discuss a number of interaction techniques with game applications on Inflated Roly-Poly that presents an engaging experience through full-body interaction. Finally, we conducted an experience workshop with four participants. The workshop proved that an inflated screen coupled with a roly-poly structure exceeds the capabilities of the rigid touch screens in terms of engagement in physical interaction.

We present PhantomPen, a direct pen input device whose pen head is virtualized onto the tablet display surface and visually connected to a graspable pen barrel in order to achieve digital drawing free from pen occlusion and visual parallax. As the pen barrel approaches the display, the virtual pen head smoothly appears as if the rendered virtual pen head and the physical pen barrel are in unity. The virtual pen head provides visual feedback by changing its virtual form according to pen type, color, and thickness while the physical pen tip, hidden in the user's sight, provides tactile feedback. Three experiments were carefully designed based on an analysis of drawings by design professionals and observations of design drawing classes. With these experiments that simulate natural drawing we proved significant performance advantages of PhantomPen. PhantomPen was at least as usable as the normal stylus in basic line drawing, and was 17 % faster in focus region drawing (26% faster in extreme focus region drawing). PhantomPen also reduced error rate by 40 % in a typical drawing setup where users have to manage a complex combination of pen and stroke properties.

Tiled-monitor large displays are widely used in various application domains. However, how their interior bezels affect user performance and behavior has not been fully understood. We conducted three controlled experiments to investigate effects of tiled-monitor interior bezels on visual search, straight-tunnel steering, and target selection tasks. The conclusions of our paper are: 1) interior bezels do not affect visual search time nor error rate; however, splitting objects across bezels is detrimental to search accuracy, 2) interior bezels are detrimental to straight-tunnel steering, but not to target selection. In addition, we discuss how inte-rior bezels affect user behaviors, and suggest guidelines for effectively using tiled-monitor large displays and designing user interfaces suited to them.

We present EverybodyLovesSketch, a gesture-based 3D curve sketching system for rapid ideation and visualization of 3D forms, aimed at a broad audience. We first analyze traditional perspective drawing in professional practice. We then design a system built upon the paradigm of ILoveSketch, a 3D curve drawing system for design professionals. The new system incorporates many interaction aspects of perspective drawing with judicious automation to enable novices with no perspective training to proficiently create 3D curve sketches. EverybodyLovesSketch supports a number of novel interactions: tick-based sketch plane selection, single view definition of arbitrary extrusion vectors, multiple extruded surface sketching, copy-and-project of 3D curves, freeform surface sketching, and an interactive perspective grid. Finally, we present a study involving 49 high school students (with no formal artistic training) who each learned and used the system over 11 days, which provides detailed insights into the popularity, power and usability of the various techniques, and shows our system to be easily learnt and effectively used, with broad appeal.

We present ILoveSketch, a 3D curve sketching system that captures some of the affordances of pen and paper for professional designers, allowing them to iterate directly on concept 3D curve models. The system coherently integrates existing techniques of sketch-based interaction with a number of novel and enhanced features. Novel contributions of the system include automatic view rotation to improve curve sketchability, an axis widget for sketch surface selection, and implicitly inferred changes between sketching techniques. We also improve on a number of existing ideas such as a virtual sketchbook, simplified 2D and 3D view navigation, multi-stroke NURBS curve creation, and a cohesive gesture vocabulary. An evaluation by a professional designer shows the potential of our system for deployment within a real design process.

This paper presents tangible interaction techniques for fine-tuning one-to-one scale NURBS curves on a large display for automotive design. We developed a new graspable handle with a transparent groove that allows designers to manipulate virtual curves on a display screen directly. The use of the proposed handle leads naturally to a rich vocabulary of terms describing interaction techniques that reflect existing shape styling methods. A user test raised various issues related to the graspable user interface, two-handed input, and large-display interaction.