As VR expands into public spaces, new challenges emerge around spontaneous interactions between bystanders and unfamiliar VR users. While current VR systems often prioritize user awareness of their physical surroundings, they overlook the social dynamics affecting nearby bystanders. We conducted a deception-based study (N=80) examining how interface availability influences bystanders’ comfort, confidence, and hesitation when interrupting VR users. We compared traditional static interruption interfaces (e.g., button on screen) with a proactive proxy that actively approached bystanders upon detecting interruption intent. Static interfaces, due to insufficient cueing, frequently caused bystander discomfort, leading to hesitant physical interruptions or complete communication avoidance. In contrast, the proactive proxy implicitly conveyed social permission, significantly enhancing bystanders’ comfort and confidence. Our findings provide empirical insights into how bystanders assess availability and initiate interruptions with unfamiliar VR users in shared spaces, offering design implications for VR systems that support bystander agency and comfort during these interactions.

Virtual Reality (VR) offers portable and flexible workspaces. However, enabling efficient and comfortable interactions without external input devices remains challenging. We propose leveraging redirected input to enable comfortable and touch-like interaction for quick and intuitive control. Our design study revealed that while touch interaction performs well with direct input, its performance degrades significantly under input redirection. In contrast, using pinch improves redirected input by providing self-haptic feedback and reducing input dimensionality, thereby compensating for spatial discrepancies. Based on these findings, we introduce Redirected Pinch, a bare-hand interaction technique that combines input redirection with pinch confirmation. It creates a virtual plane at waist height, remapping hand movements on the plane to a vertical window, with pinch gestures used for confirmation. A user study demonstrated that Redirected Pinch achieves a strong balance of accuracy, efficiency, comfort, and sense of agency across fundamental interactions.

Live streaming has become increasingly popular, driven by the desire for direct and real-time interactions between streamers and viewers. However, current text-based interactions and pre-defined emojis limit expressiveness, especially when referring to specific stream moments. We propose ThingMoji, a type of user-captured cut-outs to enhance user expression and foster more effective communication between streamers and their audience in the comment section. ThingMojis are unique digital icons created by users by capturing snapshots and annotating specific areas at any point during the stream. We developed StreamThing, a live-streaming platform integrated with ThingMojis, to explore their use during object-focused live streaming contexts. In a user study with three in-the-wild deployments reveals the expressive use of ThingMojis in diverse live-streaming scenarios with rich visual contents. Our findings show that ThingMojis enable viewers to reference specific objects, express emotions, and create shared visual narratives. Streamers found ThingMojis valuable for facilitating on-the-fly communication around visual content and fostering playful interactions. The study also uncovered challenges in ThingMoji comprehension, issues for long-term uses of ThingMojis, and potential concerns regarding misuse. Based on these insights, we discussed new opportunities for supporting object-focused communication during live streaming environments.

Sketching on virtual objects in Virtual Reality (VR) can be challenging due to the lack of a physical surface that constrains the movement and provides haptic feedback for contact and movement. While using a flat physical drawing surface has been proposed, it creates a significant discrepancy between the physical and virtual surfaces when sketching on non-planar virtual objects. We propose using a dynamic shape display that physically mimics the shape of a virtual surface, allowing users to sketch on a virtual surface as if they are sketching on a physical object’s surface. We demonstrate this using VRScroll, a shape-changing device that features seven independently controlled flaps to imitate the shape of a virtual surface automatically. Our user study showed that participants exhibited higher precision when tracing simple shapes with the dynamic shape display and produced clearer sketches. We also provided several design implications for dynamic shape displays aimed at enabling precise sketching in VR.

In this paper, we present MoiréTag—a novel tag-like device that magnifies displacement without active components for indirect sensing of subtle tangible interactions. The device consists of two overlapping layers of stripe patterns with distinct pattern frequencies. These layers create Moiré fringes that can move faster than the actual movement of a layer. Using a customized image processing pipeline, we show that MoiréTag can reliably detect sub-mm movement in real-time (mean error = 0.043 mm) under varying lighting conditions, camera angles, and camera distances. We also demonstrate five applications of MoiréTag to showcase its potential as a low-cost solution to capture and monitor small changes in movement and other physical properties, such as force and volume, by converting them into displacement.

Turn-taking is one of the biggest interactivity challenges in multiparty remote meetings. One contributing factor is that current videoconferencing tools lack support for proxemic cues; i.e., spatial cues that humans use to enact their social relations and intentions. While more recent tools provide support for proxemic metaphors, they often focus on approach and leave-taking rather than turn-taking. In this paper, we present OpenMic, a videoconferencing system that utilizes proxemic metaphors for conversational floor management by providing 1) a Virtual Floor that serves as a fixed-feature space for users to be aware of others’ intention to talk, and 2) Malleable Mirrors, which are video and screen feeds that can be continuously moved and resized for conversational floor transitions. Our exploratory user study found that these system features can aid the conversational flow in multiparty video meetings. With this work, we show potential for embedding proxemic metaphors to support conversational floor management in videoconferencing systems.

In video meetings, individuals may wish to share various physical objects with remote participants, such as physical documents, design prototypes, and personal belongings. However, our formative study discovered that this poses several challenges, including difficulties in referencing a remote user’s physical objects, the limited visibility of the object, and the friction of properly framing and orienting an object to the camera. To address these challenges, we propose ThingShare, a video-conferencing system designed to facilitate the sharing of physical objects during remote meetings. With ThingShare, users can quickly create digital copies of physical objects in the video feeds, which can then be magnified on a separate panel for focused viewing, overlaid on the user’s video feed for sharing in context, and stored in the object drawer for reviews. Our user study demonstrated that ThingShare made initiating object-centric conversations more efficient and provided a more stable and comprehensive view of shared objects.

Tasks that involve locating objects and then moving hands to those specific locations, such as using touchscreens or grabbing objects on a desk, are challenging for the visually impaired. Over the years, audio guidance and haptic feedback have been a staple in hand navigation based assistive technologies. However, these methods require the user to interpret the generated directional cues and then manually perform the hand motions. In this paper, we present automated hand-based spatial guidance to bridge the gap between guidance and execution, allowing visually impaired users to move their hands between two points automatically, without any manual effort. We implement this concept through FingerRover, an on-finger miniature robot that carries the user’s finger to target points. We demonstrate the potential applications that can benefit from automated hand-based spatial guidance. Our user study shows the potential of our technique in improving the interaction capabilities of people with visual impairments.

Robots operate under certain forces that affect their behavior. Consider, a drone meant to deliver packages must hold its pose as long as it operates under its weight and wind limits. Validating that such a drone handles external forces correctly is key to ensuring its safety. Nevertheless, validating the system's behavior under the effect of such forces can be difficult and costly. For example, checking the effects of different wind magnitudes may require waiting for the matching outdoor conditions or requiring wind tunnels. Checking the effects of different package sizes and shapes may require many slow and laborious iterations, and validating the combinations of wind gusts and package configurations is often hard to replicate. This work introduces a framework to overcome such challenges by mimicking external forces exercised on a drone with limited cost, setup, and space. The framework consists of a haptic suit device with directional propellers that can be mounted onto a drone, a synthesizer to transform intended forces into setpoints for the suit's directional propellers, and a controller for the suit to meet those setpoints. We conduct a study to assess the framework's capabilities under multiple scenarios involving various forces. Our findings show that the haptic suit framework can recreate real-world forces on the drone with acceptable precision.

Wearable devices allow quick and convenient interactions for controlling mobile computers. However, these interactions are often device-dependent, and users cannot control devices in a way they are familiar with if they do not wear the same wearable device. This paper proposes a new method, UnifiedSense, to enable device-dependent gestures even when the device that detects such gestures is missing by utilizing sensors on other wearable devices. UnifiedSense achieves this without explicit gesture training for different devices, by training its recognition model while users naturally perform gestures. The recognizer uses the gestures detected on the primary device (i.e., a device that reliably detects gestures) as labels for training samples and collects sensor data from all other available devices on the user. We conducted a technical evaluation with data collected from 15 participants with four types of wearable devices. It showed that UnifiedSense could correctly recognize 5 gestures (5 gestures × 5 configurations) with an accuracy of 90.9% (SD = 1.9%) without the primary device present.

QR codes have been used as an inexpensive means to connect users to digital platforms such as websites and mobile applications. However, despite their ubiquity, QR codes are limited in purpose and can only redirect users to the URL contained within it, thereby making their use heavily network dependent which can be unsuitable for use in ephemeral scenarios and areas with limited connectivity. In this paper, we introduce Frappé, a framework capable of deploying ultra lightweight UIs to mobile devices directly through QR codes, without requiring any network connectivity. This is achieved by decomposing the UI into metadata and storing it inside the QR code, while offloading the UI functionality to API calls. We also introduce enFrappé, a WYSIWYG tool for building Frappé UIs. We demonstrate the lightweight nature of our framework through a technical evaluation, whereas the usability of our UI builder tool is demonstrated through a user study.

People often form small conversation groups during physical gatherings to have ad-hoc and informal conversations. As these groups are loosely defined, others can often overhear and join the conversation. However, current video-conferencing tools only allow for strict boundaries between small conversation groups, inhibiting fluid group formations and between-group conversations. This isolates small-group conversations from others and leads to inefficient transitions between conversations. We present FluidMeet, a virtual breakout meeting system that employs flexible conversation boundaries and cross-group conversation visualizations to enable fluid conversation group formations and ad-hoc, informal conversations. FluidMeet enables out-group members to overhear group conversations while allowing conversation groups to control their shared level of context. Users within conversation groups can also quickly switch between in-group and private conversations. A study of FluidMeet showed that it encouraged users to break group boundaries, made them feel less isolated in group conversations, and facilitated communication across different groups.

Automated vehicles promise a future where drivers can engage in non-driving tasks without hands on the steering wheels for a prolonged period. Nevertheless, automated vehicles may still need to occasionally hand the control back to drivers due to technology limitations and legal requirements. While some systems determine the need for driver takeover using driver context and road condition to initiate a takeover request, studies show that the driver may not react to it. We present DeepTake, a novel deep neural network-based framework that predicts multiple aspects of takeover behavior to ensure that the driver is able to safely take over the control when engaged in non-driving tasks. Using features from vehicle data, driver biometrics, and subjective measurements, DeepTake predicts the driver’s intention, time, and quality of takeover. We evaluate DeepTake performance using multiple evaluation metrics. Results show that DeepTake reliably predicts the takeover intention, time, and quality, with an accuracy of 96%, 93%, and 83%, respectively. Results also indicate that DeepTake outperforms previous state-of-the-art methods on predicting driver takeover time and quality. Our findings have implications for the algorithm development of driver monitoring and state detection.

Completing tasks on smartwatches often requires multiple gestures due to the small size of the touchscreens and the lack of sufficient number of touch controls that are easily accessible with a finger. We propose to increase the number of functions that can be triggered with the touch gesture by enabling a smartwatch to identify which finger is being used. We developed MagTouch, a method that uses a magnetometer embedded in an off-the-shelf smartwatch. It measures the magnetic field of a magnet fixed to a ring worn on the middle finger. By combining the measured magnetic field and the touch location on the screen, MagTouch recognizes which finger is being used. The tests demonstrated that MagTouch can differentiate among the three fingers used to make contacts at a success rate of 95.03%.

We present Plane, Ray, and Point, a set of interaction techniques that utilizes shape constraints to enable quick and precise object alignment and manipulation in virtual reality. Users create the three types of shape constraints, Plane, Ray, and Point, by using symbolic gestures. The shape constraints are used like scaffoldings and limit and guide the movement of virtual objects that collide or intersect with them. The same set of gestures can be performed with the other hand, which allow users to further control the degrees of freedom for precise and constrained manipulation. The combination of shape constraints and bimanual gestures yield a rich set of interaction techniques to support object transformation. An exploratory study conducted with 3D design experts and novice users found the techniques to be useful in 3D scene design workflows and easy to learn and use.

We present PseudoBend, a haptic feedback technique that creates the illusion that a rigid device is being stretched, bent, or twisted. The method uses a single 6-DOF force sensor and a vibrotactile actuator to render grain vibrations to simulate the vibrations produced during object deformation based on the changes in force or torque exerted on a device. Because this method does not require any moving parts aside from the vibrotactile actuator, devices designed using this method can be small and lightweight. Psychophysical studies conducted using a prototype that implements this method confirmed that the method could be used to successfully create the illusion of deformation and could also change users' perception of stiffness by changing the virtual stiffness parameters.

We present a new handheld haptic device, Thor's Hammer, which uses propeller propulsion to generate ungrounded, 3-DOF force feedback. Thor's Hammer has six motors and propellers that generates strong thrusts of air without the need for physical grounding or heavy air compressors. With its location and orientation tracked by an optimal tracking system, the system can exert forces in arbitrary directions regardless of the device's orientation. Our technical evaluation shows that Thor's Hammer can apply up to 4 N of force in arbitrary directions with less than 0.11 N and 3.9° of average magnitude and orientation errors. We also present virtual reality applications that can benefit from the force feedback provided by Thor's Hammer. Using these applications, we conducted a preliminary user study and participants felt the experience more realistic and immersive with the force feedback.

Despite gaining traction in North America, live streaming has not reached the popularity it has in China, where live- streaming has a tremendous impact on the social behaviors of users. To better understand this socio-technological phenomenon, we conducted a mixed methods study of live streaming practices in China. We present the results of an online survey of 527 live streaming users, focusing on their broadcasting or viewing practices and the experiences they find most engaging. We also interviewed 14 active users to explore their motivations and experiences. Our data revealed the different categories of content that was broadcasted and how varying aspects of this content engaged viewers. We also gained insight into the role reward systems and fan group-chat play in engaging users, while also finding evidence that both viewers and streamers desire deeper channels and mechanisms for interaction in addition to the commenting, gifting, and fan groups that are available today.

We make touch input by physically colliding an end effector (e.g., a body part or a stylus) with a touch surface. Prior studies have examined the use of kinematic variables of collision between objects, such as position, velocity, force, and impact. However, the nature of the collision can be understood more thoroughly by considering the known physical relationships that exist between directly measurable variables (i.e., kinetics). Based on this collision kinetics, this study proposes a novel touch technique called FDSense. By simultaneously observing the force and contact area measured from the touchpad, FDSense allows estimation of the Young's modulus and stiffness of the object being contacted. Our technical evaluation showed that FDSense could effectively estimate the Young's modulus of end effectors made of various materials, and the stiffness of each part of the human hand. Two applications using FDSense were demonstrated, for digital painting and digital instruments, where the result of the expression varies significantly depending on the elasticity of the end effector. In a following informal study, participants assessed the technique positively.

Knowledge-sharing live streams are distinct from traditional educational videos, at least because of the large concurrently-viewing audience and the real-time discussions between viewers and the streamer. Though this creates unique opportunities for interactive learning, it also brings a challenge for creating a useful archive for post hoc learning. This paper presents the results of interviews with knowledge sharing streamers, their moderators, and viewers to understand current experiences and needs for sharing and learning knowledge through live streaming. Based on those findings, we built StreamWiki, a tool which leverages the viewers during live streams to produce useful archives of the interactive learning experience. On StreamWiki, moderators initiate tasks that viewers complete by conducting microtasks, such as writing a summary, commenting, and voting for informative comments. As a result, a summary document is built in real time. Through the tests of our prototype with streamers and viewers, we found that StreamWiki could help understanding the content and the context of the stream, during the stream and for post hoc learning.

While smart glasses make information more accessible in mobile scenarios, entering text on these devices is still difficult. In this paper, we suggest using a smartwatch as an indirect input device for smart glasses text entry. With the watch-glasses combination, users do not need to lift the arm to touch the glasses nor need to carry a special external input device. To prove the feasibility of the suggested combination, we implemented two text entry methods: a modified version of SwipeBoard, which we adapted for the suggested combination, and HoldBoard, which we newly designed and implemented specifically for the suggested combination. We evaluated the performances of the two text entry methods through two user studies, and could show that they are faster than prior art for smart glasses text entry in a seated condition. A further study showed that they are competitive with the prior art also in a walking condition.

Touchscreens continue to advance including progress towards sensing fingers proximal to the display. We explore this emerging pre-touch modality via a self-capacitance touchscreen that can sense multiple fingers above a mobile device, as well as grip around the screen's edges. This capability opens up many possibilities for mobile interaction. For example, using pre-touch in an anticipatory role affords an "ad-lib interface" that fades in a different UI--appropriate to the context--as the user approaches one-handed with a thumb, two-handed with an index finger, or even with a pinch or two thumbs. Or we can interpret pre-touch in a retroactive manner that leverages the approach trajectory to discern whether the user made contact with a ballistic vs. a finely-targeted motion. Pre-touch also enables hybrid touch + hover gestures, such as selecting an icon with the thumb while bringing a second finger into range to invoke a context menu at a convenient location. Collectively these techniques illustrate how pre-touch sensing offers an intriguing new back-channel for mobile interaction.

The QWERTY keyboard has been a de facto standard for computer text entry and continues to be one for mobile text entry such as for smartphones. It is not clear, however, that it will continue to be an option for text entry for much smaller devices such as smartwatches. In a series of user experiments, we examined the performance of the QWERTY keyboard when it is reduced to fit a small smartwatch screen. At the same time, we examined whether the ZoomBoard and the SplitBoard, which are QWERTY keyboards augmented by zooming and panning strategies, respectively, would be effective in comparison with a plain QWERTY keyboard. In Experiment 1, we evaluated the text entry performance of new users on the three QWERTY keyboards. In Experiment 2, we evaluated the relative performance of the three keyboards for three different screen sizes. In Experiment 3, we further observed how the keyboard performance changed when used in a mobile situation. Main results are: (i) users could adapt to a plain QWERTY keyboard even in the smallest screen cases. (ii) The SplitBoard consistently showed a better performance than other keyboards in all tested sizes. (iii) The SplitBoard showed a better performance than other keyboards in a mobile condition (treadmill) and was preferred most by participants.

Text entry on a smartwatch is a challenging problem due to the device's limited screen area. In this paper, we introduce the SplitBoard, which is a soft keyboard designed for a smartwatch. As the user flicks left or right on the keyboard, it switches between the left and right halves of a QWERTY keyboard. We report the results of two user experiments where the SplitBoard was compared to an ordinary QWERTY keyboard, the ZoomBoard, SlideBoard, and Qwerty-like keypad. We measured the initial performance with new users for each method. The SplitBoard outperformed all other techniques in the experiments. The SplitBoard is expected to be a viable option for smartwatch text entry because of its light processing requirements, good performance, and immediate learnability.

In recent years, touch screens have emerged and matured as the main input interface for mobile and tablet computers calling for extended touch input possibilities. In this paper, we explore the use of consecutive distant taps to expand the touch screen input vocabulary. We analyzed time intervals and distances between consecutive taps during common applications on a tablet and verified that consecutive distant taps can be used conflict-free with existing touch gestures. We designed the two interaction techniques Ta-tap and Ta-Ta-tap that utilize consecutive distant taps. Ta-tap uses two consecutive distant taps to invoke alternative touch operations for multi-touch emulation, whereas Ta-Ta-tap uses a series of consecutive distant taps to define a spatial gesture. We verified the feasibility of both interaction techniques through a series of experiments and a user study. The high recognition rate of Ta-tap and Ta-Ta-tap gestures, the few conflicts with existing gestures, and the positive feedback from the participants assert the potential of consecutive distant taps as a new design space to enrich touch screen interactions.

In this paper, we explore the possibility of a long touchpad that utilizes the entire area below the keyboard of a laptop computer. An essential prerequisite for such a touchpad is a robust palm rejection method, which we satisfy using a proximity-sensing touchpad. We developed LongPad, a proximity-sensing optical touchpad that is as wide as a laptop keyboard, and implemented a palm rejection algorithm that utilizes proximity images from LongPad. In a user study conducted, we observed that LongPad rejected palm touches almost perfectly while participants were repeating typing and pointing tasks. We also summarize the new design space enabled by LongPad and demonstrate a few of the interaction techniques it facilitates.

The possibility of using shear forces is being explored recently as a method to enrich touch screen interaction. However, most of the related studies are restricted to the case of single-point shear forces, possibly owing to the difficulty of independently sensing shear forces at multiple touch points. In this paper, we propose indirect methods to estimate shear forces using the movement of contact areas. These methods enable multi-point shear force estimation, where the estimation is done for each finger independently. We show the feasibility of these methods through an informal user study with a demo application utilizing these methods.

Our goal is to show that it is possible to automatically infer social relationship types among people who stay together in an organization by analyzing communication patterns. We collected indoor co-location data and instant messenger data from 22 participants for one month. Based on the data, we designed and explored several indicators which are considered to be useful for mining social relationship types. We applied machine learning techniques using the indicators and found that it is possible to develop an intelligent method to infer social relationship types.

We introduce an interaction technique that increases the touch screen input vocabulary by distinguishing a strong tap from a gentle tap without the use of additional hardware. We have designed and validated an algorithm that detects different types of screen touches by combining data from the built-in accelerometer with position data from the touch screen. The proposed technique allows a touch screen input to contain not only the position of a finger contact, but also its type, i.e., whether the contact is a 'Tap' or a 'ForceTap.' To verify the feasibility of the proposed technique we have implemented our detection algorithm in experiments that test cases of single-handed, two-handed, immersive, and on the move usage. Based on the experimental results, we investigate the advantages of using two types of touch inputs and discuss emerging issues. Finally, we suggest a design guideline for applying the proposed technique to touch screen applications, and present possible application scenarios.

Six-degrees-of-freedom (6-DOF) trackers, which were mainly for professional computer applications, are now in demand by everyday consumer applications. With the requirements of consumer electronics in mind, we designed an optical 6-DOF tracker where a few photo-sensors can track the position and orientation of an LED cluster. The operating principle of the tracker is basically source localization by solving an inverse problem. We implemented a prototype system for a TV viewing environment, verified the feasibility of the operating principle, and evaluated the basic performance of the prototype system in terms of accuracy and speed. We also examined its application possibility to different environments, such as a tabletop computer, a tablet computer, and a mobile spatial interaction environment.

Force gestures are touch screen gestures augmented by the normal and tangential forces on the screen. In order to study the feasibility of the force gestures on a mobile touch screen, we implemented a prototype touch screen device that can sense the normal and tangential forces of a touch gesture on the screen. We also designed two example applications, a web browser and an e-book reader, that utilize the force gestures for their primary actions. We conducted a user study with the prototype and the applications to study the characteristics of the force gestures and the effectiveness of their mapping to the primary actions. In the user study we could also discover interesting usability issues and collect useful user feedback about the force gestures and their mapping to GUI actions.