In this paper, we explore a novel approach that leverages retrofitting to create sensor-powered smart car cabins. We propose that retrofitting offers a promising way to complement and extend the capabilities of built-in smart cabin sensors provided by car manufacturers. To understand how retrofitting solutions should be designed, we conducted a two-phase study. First, through semi-structured interviews with 18 participants, we examined challenges with built-in smart cabin sensors and identified opportunities where retrofitting could address these limitations. Second, through probe-based participatory design sessions with 15 participants, we identified user requirements and expectations for effective retrofit solutions. Based on our findings, we present a set of design recommendations to guide the future development of retrofit methods for smart car cabins.

The development of smart textile interfaces is hindered by the inclusion of rigid hardware components and batteries within the fabric, which pose challenges in terms of manufacturability, usability, and environmental concerns related to electronic waste. To mitigate these issues, we propose a smart textile interface and its wireless sensing system to eliminate the need for ICs, batteries, and connectors embedded into textiles. Our technique is established on the integration of multi-resonant circuits in smart textile interfaces, and utilizing near-field electromagnetic coupling between two coils to facilitate wireless power transfer and data acquisition from smart textile interface. A key aspect of our system is the development of a mathematical model that accurately represents the equivalent circuit of the sensing system. Using this model, we developed a novel algorithm to accurately estimate sensor signals based on changes in system impedance. Through simulation-based experiments and a user study, we demonstrate that our technique effectively supports multiple textile sensors of various types.

InkBrush is a new sketch-based 3D drawing tool for creating 3D ink paintings using free-form 3D ink strokes. It offers a digital calligraphy brush and various editing tools to generate realistic ink-like brush strokes with attributes like hairy edges, ink drips, and scattered dots. Users can adjust parameters such as moisture, color, darkness, dryness, and stroke style to customize the appearance of the brush strokes. The development of InkBrush was guided by a design study involving artists and designers. It was developed as a plugin for Blender, a popular 3D modeling tool, and its effectiveness and usability were evaluated through a user study involving 75 participants. Preliminary feedback from the participants was overwhelmingly positive, indicating that InkBrush was intuitive and easy to use. Following this, we also sought in-depth assessments from experts in ink painting and 3D design. Their evaluations further demonstrated the effectiveness of InkBrush.

We present a new approach to address the challenges associated with maintaining the functionality of triboelectric vibration sensors in smart plywood during woodworking operations involving nails and screws. The current state-of-the-art sensor design employs non-overlapping electrodes, which unfortunately leads to significant compromises in terms of signal strength and clarity, particularly in real-world scenarios that involve electromagnetic (EM) interference. To overcome these limitations, we propose a method that enables the woodworker to manually isolate short-circuited electrodes. This method facilitates the creation of sensors using overlapping electrodes, while also incorporating EM shielding, thereby resulting in a substantial improvement in the sensor’s robustness when detecting user activities. To validate the effectiveness of our proposed approach, we conducted a series of experiments, which not only shed light on the drawbacks of non-overlapping electrode designs but also demonstrated the significant improvements achieved through our method.

Tagnoo is a computational plywood augmented with RFID tags, aimed at empowering woodworkers to effortlessly create room-scale smart environments. Unlike existing solutions, Tagnoo does not necessitate technical expertise or disrupt established woodworking routines. This battery-free and cost-effective solution seamlessly integrates computation capabilities into plywood, while preserving its original appearance and functionality. In this paper, we explore various parameters that can influence Tagnoo’s sensing performance and woodworking compatibility through a series of experiments. Additionally, we demonstrate the construction of a small office environment, comprising a desk, chair, shelf, and floor, all crafted by an experienced woodworker using conventional tools such as a table saw and screws while adhering to established construction workflows. Our evaluation confirms that the smart environment can accurately recognize 18 daily objects and user activities, such as a user sitting on the floor or a glass lunchbox placed on the desk, with over 90% accuracy.

Shadow puppetry, a culturally rich storytelling art, faces challenges transitioning to the digital realm. Creators in the early design phase struggle with crafting intricate patterns, textures, and basic animations while adhering to stylistic conventions - hindering creativity, especially for novices. This paper presents Lumina, a tool to facilitate the early Chinese shadow puppet design stage. Lumina provides contour templates, animations, scene editing tools, and machine-generated traditional puppet patterns. These features liberate creators from tedious tasks, allowing focus on the creative process. Developed based on a formative study with puppet creators, the web-based Lumina enables wide dissemination. An evaluation with 18 participants demonstrated Lumina’s effectiveness and ease of use, with participants successfully creating designs spanning traditional themes to contemporary and science-fiction concepts.

Tools supporting immersive live video conferencing (VC) have gained popularity recently across diverse application domains. A core component of the experience is augmenting video communication with multimodal interactive media. While many direct-manipulation techniques for VC communication have been proposed in existing literature, the usability and preferences for these techniques have never been formally studied. In this paper, we examine how embodied interaction democratizes content authoring, and propose a rehearsal-to-performance (RtP) framework along with a VC system, Clio, that enables performers to directly interact with their media using voice, gesture, and external devices such as tablets. We evaluate existing operation-to-modality mappings for VC communication, as well as describe novel mappings not present in the literature. A series of studies demonstrate modality preferences and potentials for incorporating real-time direct-manipulation tools to create expressive augmented VC performances.

User interactions with smartwatches are limited due to one-finger usage on small touch screens. However, smartwatch interaction capabilities can be extended beyond the screen into the around-device space by leveraging multi-finger interactions. In this paper, we explore suitable finger postures and gestures in mid-air and on the back of the palm to extend interaction capabilities on the unmodified commodity smartwatch while ensuring their learnability through discoverability. We conduct a design study to find a set of finger postures and gestures suitable for interacting with off-the-shelf smartwatches. An application is then designed to detect the posture-gesture set with on-device dual cameras while examining their classification accuracy. To facilitate learnability through discoverability of the gesture-posture set, we explore ‘context-aware-hints’ that shows a sub-set of gesture-posture and their associated actions based on the usage context. Results from a user study show that the guidance helps users to discover and learn them quickly and accurately.

NFC (Near-Field Communication) has been widely applied for human-computer interaction (HCI). However, the short sensing distance of NFC requires the users to initiate the tasks with extra effort mostly using their hands, so it is inconvenient to use NFC in hands-busy scenarios. This paper presents an investigation of body-centric interactions between the NFC device users and their surroundings. The exploration is based on the recent development of near-field enabled clothing, which can passively extend an NFC-enabled device’s reading distance to users’ body landmarks. We present an accessible method for fabricating flexible, extensible, and scalable NFC extenders on clothing pieces, and an easy-to-use toolkit for facilitating designers to realize the interactive experiences. The method and toolkit were tested in technical experiments and in a co-creation workshop. The elicited design outcomes and the further exploratory makings generated knowledge for future research and embodied interaction design opportunities.

iWood is interactive plywood that can sense vibration based on triboelectric effect. As a material, iWood survives common woodworking operations, such as sawing, screwing, and nailing and can be used to create furniture and artifacts. Things created using iWood inherit its sensing capability and can detect a variety of user input and activities based on their unique vibration patterns. Through a series of experiments and machine simulations, we carefully chose the size of the sensor electrodes, the type of triboelectric materials, and the bonding method of the sensor layers to optimize the sensitivity and fabrication complexity. The sensing performance of iWood was evaluated with 4 gestures and 12 daily activities carried out on a table, nightstand, and cutting board, all created using iWood. Our result suggested over 90% accuracies for activity and gesture recognition.

We study phrase-gesture typing, a gesture typing method that allows users to type short phrases by swiping through all the letters of the words in a phrase using a single, continuous gesture. Unlike word-gesture typing, where text needs to be entered word by word, phrase-gesture typing enters text phrase by phrase. To demonstrate the usability of phrase-gesture typing, we implemented a prototype called PhraseSwipe. Our system is composed of a frontend interface designed specifically for typing through phrases and a backend phrase-level gesture decoder developed based on a transformer-based neural language model. Our decoder was trained using five million phrases of varying lengths of up to five words, chosen randomly from the Yelp Review Dataset. Through a user study with 12 participants, we demonstrate that participants could type using PhraseSwipe at an average speed of 34.5 WPM with a Word Error Rate of 1.1%.

In this paper, we propose the designs for low cost and 3D-printable add-on components to adapt existing breadboards, circuit components and electronics tools for blind or low vision (BLV) users. Through an initial user study, we identified several barriers to entry for beginners with BLV in electronics and circuit prototyping. These barriers guided the design and development of our add-on components. We focused on developing adaptations that provide additional information about the specific component pins and breadboard holes, modify tools to make them easier to use for users with BLV, and expand non-visual feedback (e.g., audio, tactile) for tasks that require vision. Through a second user study, we demonstrated that our adaptations can effectively overcome the accessibility barriers in breadboard circuit prototyping for users with BLV.

Back-of-device interaction is a promising approach to interacting on smartphones. In this paper, we create a back-of-device command and text input technique called BackSwipe, which allows a user to hold a smartphone with one hand, and use the index finger of the same hand to draw a word-gesture anywhere at the back of the smartphone to enter commands and text. To support BackSwipe, we propose a back-of-device word-gesture decoding algorithm which infers the keyboard location from back-of-device gestures, and adjusts the keyboard size to suit the gesture scales; the inferred keyboard is then fed back into the system for decoding. Our user study shows BackSwipe is feasible and a promising input method, especially for command input in the one-hand holding posture: users can enter commands at an average accuracy of 92% with a speed of 5.32 seconds/command. The text entry performance varies across users. The average speed is 9.58 WPM with some users at 18.83 WPM; the average word error rate is 11.04% with some users at 2.85%. Overall, BackSwipe complements the extant smartphone interaction by leveraging the back of the device as a gestural input surface.

In this paper, we propose a reconfigurable electrode, RElectrode, using a microfluidic technique that can change the geometry and material properties of the electrode to satisfy the needs for sensing a variety of different types of user input through touch/touchless gestures, pressure, temperature, and distinguish between different types of objects or liquids. Unlike the existing approaches, which depend on the specific-shaped electrode for particular sensing (e.g., coil for inductive sensing), RElectrode enables capacity, inductance, resistance/pressure, temperature, pH sensings all in a single package. We demonstrate the design and fabrication of the microfluidic structure of our RElectrode, evaluate its sensing performance through several studies, and provide some unique applications. RElectrode demonstrates technical feasibility and application values of integrating physical and biochemical properties of microfluidics into novel sensing interfaces.

We present Project Tasca, a pocket-based textile sensor that detects user input and recognizes everyday objects that a user carries in the pockets of a pair of pants (e.g., keys, coins, electronic devices, or plastic items). By creating a new fabric-based sensor capable of detecting in-pocket touch and pressure, and recognizing metallic, non-metallic, and tagged objects inside the pocket, we enable a rich variety of subtle, eyes-free, and always-available input, as well as context-driven interactions in wearable scenarios. We developed our prototype by integrating four distinct types of sensing methods, namely: inductive sensing, capacitive sensing, resistive sensing, and NFC in a multi-layer fabric structure into the form factor of a jeans pocket. Through a ten-participant study, we evaluated the performance of our prototype across 11 common objects including hands, 8 force gestures, and 30 NFC tag placements. We yielded a 92.3% personal cross-validation accuracy for object recognition, 96.4% accuracy for gesture recognition, and a 100% accuracy for detecting NFC tags at close distance . We conclude by demonstrating the interactions enabled by our pocket-based sensor in several applications.

Guqin is a plucked seven-string traditional Chinese musical instrument that exists for over 3,000 years. However, as an Intangible World Cultural Heritage, the inheritance of Guqin and its culture in modern society is in deep danger. According to our study with 1,006 Chinese worldwide, Guqin as an instrument is not well-known and barely accessible. To better promote Guqin, we developed two interactive systems: VirGuqin and MRGuqin. VirGuqin was developed using a low-cost motion tracking device and was tested in a museum. 89% of 308 participants expressed an increase in interest in learning Guqin after using our system. MRGuqin was developed as a mixed reality learning environment to reduce the entry barrier to Guqin, and was tested by 16 participants, allowing them to learn Guqin significantly faster and perform better than the current practice. Our study demonstrates how technology can be used to help the inheritance of this dying art.

In this paper, we present ArmMenu, a command input approach for distant displays. ArmMenu has a circular interface like pie menus and menu selection is performed by proprioception-based lateral arm movements. We implemented ArmMenu with an off-the-shelf body tracking device (Kinect) and conducted two experiments to validate its efficacy. In the first experiment, we explored the design space of ArmMenu by varying the number of menu items, with exposed or hidden menu modes. Users can operate up to 8-item menus with high selection accuracy (>98%). ArmMenu was fast and accurate even with the hidden menu mode. The second experiment compared the performance of ArmMenu and touchless marking menus. While having similar selection accuracy, ArmMenu was faster and more preferable by users. Our studies consequently demonstrate ArmMenu's effectiveness for command input on distant displays.

We present a novel haptic and audio feedback device that allows blind and visually impaired (BVI) users to understand circuit diagrams. TangibleCircuits allows users to interact with a 3D printed tangible model of a circuit which provides audio tutorial directions while being touched. Our system comprises an automated parsing algorithm which extracts 3D printable models as well as an audio interfaces from a Fritzing diagram. To better understand the requirements of designing technology to assist BVI users in learning hardware computing, we conducted a series of formative inquiries into the accessibility limitations of current circuit tutorial technologies. In addition, we derived insights and design considerations gleaned from conducting a formal comparative user study to understand the effectiveness of TangibleCircuits as a tutorial system. We found that BVI users were better able to understand the geometric, spatial and structural circuit information using TangibleCircuits, as well as enjoyed learning with our tool.

Zippers are common in a wide variety of objects that we use daily. This work investigates how we can take advantage of such common daily activities to support seamless interaction with technology. We look beyond simple zipper-sliding interactions explored previously to determine how to weave foreground and background interactions into a vocabulary of natural usage patterns. We begin by conducting two user studies to understand how people typically interact with zippers. The findings identify several opportunities for zipper input and sensing, which inform the design of Zippro, a self-contained prototype zipper slider, which we evaluate with a standard jacket zipper. We conclude by demonstrating several applications that make use of the identified foreground and background input methods.

We propose a new sensing technique for one-dimensional touch input workable on an interactive thread of less than 0.4 mm thick. Our technique locates up to two touches using impedance sensing with a spacing resolution unachievable by the existing methods. Our approach is also unique in that it locates a touch based on a mathematical model describing the change in thread impedance in relation to the touch locations. This allows the system to be easily calibrated by the user touching a known location(s) on the thread. The system can thus quickly adapt to various environmental settings and users. A system evaluation showed that our system could track the slide motion of a finger with an average error distance of 6.13 mm and 4.16 mm using one and five touches for calibration, respectively. The system could also distinguish between single touch and two concurrent touches with an accuracy of 99% and could track two concurrent touches with an average error distance of 8.55 mm. We demonstrate new interactions enabled by our sensing approach in several unique applications.

Bezel-based gestures expand the interaction space of touch-screen devices (e.g., smartphones and smartwatches). Existing works have mainly focused on bezel-initiated swipe (BIS) on square screens. To investigate the usability of BIS on round smartwatches, we design six different circular bezel layouts, by dividing the bezel into 6, 8, 12, 16, 24, and 32 segments. We evaluate the user performance of BIS on these layouts in an eyes-free situation. The results show that the performance of BIS is highly orientation dependent, and varies significantly among users. Using the Support-Vector-Machine (SVM) model significantly increases the accuracy on 6-, 8-, 12-, and 16-segment layouts. We then compare the performance of personal and general SVM models, and find that personal models significantly improve the accuracy for 8-, 12-, 16-, and 24-segment layouts. Lastly, we discuss the potential smartwatch applications enabled by the BIS.

We present Fabriccio, a touchless gesture sensing technique developed for interactive fabrics using Doppler motion sensing. Our prototype was developed using a pair of loop antennas (one for transmitting and the other for receiving), made of conductive thread that was sewn onto a fabric substrate. The antenna type, configuration, transmission lines, and operating frequency were carefully chosen to balance the complexity of the fabrication process and the sensitivity of our system for touchless hand gestures, performed at a 10 cm distance. Through a ten-participant study, we evaluated the performance of our proposed sensing technique across 11 touchless gestures as well as 1 touch gesture. The study result yielded a 92.8% cross-validation accuracy and 85.2% leave-one-session-out accuracy. We conclude by presenting several applications to demonstrate the unique interactions enabled by our technique on soft objects.

We present a bimanual text input method on a miniature fingertip keyboard, that invisibly resides on the first segment of a user's index finger on both hands. Text entry can be carried out using the thumb-tip to tap the tip of the index finger. The design of our keyboard layout followed an iterative process, where we first conducted a study to understand the natural expectation of the handedness of the keys in a QWERTY layout for users. Among a choice of 67,108,864 design variations, we identified 1295 candidates offering a good satisfaction for user expectations. Based on these results, we computed an optimized bimanual keyboard layout, while considering the joint optimization problems of word ambiguity and movement time. Our user evaluation revealed that participants achieved an average text entry speed of 23.4 WPM.

Haptic simulation of hand tools like wrenches, pliers, scissors and syringes are beneficial for finely detailed skill training in VR, but designing for numerous hand tools usually requires an expert-level knowledge of specific mechanism and protocol. This paper presents HapLinkage, a prototyping framework based on linkage mechanism, that provides typical motion templates and haptic renderers to facilitate proxy design of virtual hand tools. The mechanical structures can be easily modified, for example, to scale the size, or to change the range of motion by selectively changing linkage lengths. Resistant, stop, release, and restoration force feedback are generated by an actuating module as part of the structure. Additional vibration feedback can be generated with a linear actuator. HapLinkage enables easy and quick prototypting of hand tools for diverse VR scenarios, that embody both of their kinetic and haptic properties. Based on interviews with expert designers, it was confirmed that HapLinkage is expressive in designing haptic proxy of hand tools to enhance VR experiences. It also identified potentials and future development of the framework.

We present Capacitivo, a contact-based object recognition technique developed for interactive fabrics, using capacitive sensing. Unlike prior work that has focused on metallic objects, our technique recognizes non-metallic objects such as food, different types of fruits, liquids, and other types of objects that are often found around a home or in a workplace. To demonstrate our technique, we created a prototype composed of a 12 x 12 grid of electrodes, made from conductive fabric attached to a textile substrate. We designed the size and separation between the electrodes to maximize the sensing area and sensitivity. We then used a 10-person study to evaluate the performance of our sensing technique using 20 different objects, which yielded a 94.5% accuracy rate. We conclude this work by presenting several different application scenarios to demonstrate unique interactions that are enabled by our technique on fabrics.

Members of the blind and visually impaired community rely heavily on tactile illustrations - raised line graphics on paper that are felt by hand - to understand geometric ideas in school textbooks, depict a story in children's books, or conceptualize exhibits in museums. However, these illustrations often fail to achieve their goals, in large part due to the lack of understanding in how 3D shapes can be represented in 2D projections. This paper describes a new technique to design tactile illustrations considering the needs of blind individuals. Successful illustration design of 3D objects presupposes identification and combination of important information in topology and geometry. We propose a twofold approach to improve shape understanding. First, we introduce a part-based multi-projection rendering strategy to display geometric information of 3D shapes, making use of canonical viewpoints and removing reliance on traditional perspective projections. Second, curvature information is extracted from cross sections and embedded as textures in our illustrations.

We propose autocomplete for the design and development of virtual breadboard circuits using software prototyping tools. With our system, a user inserts a component into the virtual breadboard, and it automatically provides a user with a list of suggested components. These suggestions complete or ex- tend the electronic functionality of the inserted component to save the user's time and reduce circuit error. To demon- strate the effectiveness of autocomplete, we implemented our system on Fritzing, a popular open source breadboard circuit prototyping software, used by novice makers. Our autocomplete suggestions were implemented based upon schematics from datasheets for standard components, as well as how components are used together from over 4000 circuit projects from the Fritzing community. We report the results of a controlled study with 16 participants, evaluating the effectiveness of autocomplete in the creation of virtual breadboard circuits, and conclude by sharing insights and directions for future research.

We propose a new type of haptic output for foreground interactions on an interactive chair, where input is carried out explicitly in the foreground of the user's consciousness. This type of force output restricts a user's motion by modulating the resistive force when rotating a seat, tilting the backrest, or rolling the chair. These interactions are useful for many applications in a ubiquitous computing environment, ranging from immersive VR games to rapid and private query of information for people who are occupied with other tasks (e.g. in a meeting). We carefully designed and implemented our proposed haptic force output on a standard office chair and determined the recognizability of five force profiles for rotating, tilting, and rolling the chair. We present the result of our studies, as well as a set of novel interaction techniques enabled by this new force output for chairs.

Interacting with a smartphone using touch input and speech output is challenging for visually impaired people in mobile and public scenarios, where only one hand may be available for input (e.g., while holding a cane) and using the loudspeaker for speech output is constrained by environmental noise, privacy, and social concerns. To address these issues, we propose EarTouch, a one-handed interaction technique that allows the users to interact with a smartphone using the ear to perform gestures on the touchscreen. Users hold the phone to their ears and listen to speech output from the ear speaker privately. We report how the technique was designed, implemented, and evaluated through a series of studies. Results show that EarTouch is easy, efficient, fun and socially acceptable to use.

Instructors of hardware computing face many challenges including maintaining awareness of student progress, allocating their time adequately between lecturing and helping individual students, and keeping students engaged even while debugging problems. Based on formative interviews with 5 electronics instructors, we found that many circuit style behaviors could help novice users prevent or efficiently debug common problems. Drawing inspiration from the software engineering practice of coding style, these circuit style behaviors consist of best-practices and guidelines for implementing circuit prototypes that do not interfere with the functionality of the circuit, but help a circuit be more readable, less error-prone, and easier to debug. To examine if these circuit style behaviors could be peripherally enforced, aid an in-person instructor's ability to facilitate a workshop, and not monopolize instructor's attention, we developed CircuitStyle, a teaching aid for in-person hardware computing workshops. To evaluate the effectiveness of our tool, we deployed our system in an in-person maker-space workshop. The instructor appreciated CircuitStyle's ability to provide a broad understanding of the workshop's progress and the potential for our system to help instructors of various backgrounds better engage and understand the needs of their classroom.

We present Tessutivo, a contact-based inductive sensing technique for contextual interactions on interactive fabrics. Our technique recognizes conductive objects (mainly metallic) that are commonly found in households and workplaces, such as keys, coins, and electronic devices. We built a prototype containing six by six spiral-shaped coils made of conductive thread, sewn onto a four-layer fabric structure. We carefully designed the coil shape parameters to maximize the sensitivity based on a new inductance approximation formula. Through a ten-participant study, we evaluated the performance of our proposed sensing technique across 27 common objects. We yielded 93.9% real-time accuracy for object recognition. We conclude by presenting several applications to demonstrate the unique interactions enabled by our technique.

Force feedback is said to be the next frontier in virtual reality (VR). Recently, with consumers pushing forward with untethered VR, researchers turned away from solutions based on bulky hardware (e.g., exoskeletons and robotic arms) and started exploring smaller portable or wearable devices. However, when it comes to rendering inertial forces, such as when moving a heavy object around or when interacting with objects with unique mass properties, current ungrounded force feedback devices are unable to provide quick weight shifting sensations that can realistically simulate weight changes over 2D surfaces. In this paper we introduce Aero-plane, a force-feedback handheld controller based on two miniature jet propellers that can render shifting weights of up to 14 N within 0.3 seconds. Through two user studies we: (1) characterize the users' ability to perceive and correctly recognize different motion paths on a virtual plane while using our device; and, (2) tested the level of realism and immersion of the controller when used in two VR applications (a rolling ball on a plane, and using kitchen tools of different shapes and sizes). Lastly, we present a set of applications that further explore different usage cases and alternative form-factors for our device.

We propose blending the virtual and physical worlds for prototyping circuits using physical proxies. With physical proxies, real-world components (e.g. a motor, or light sensor) can be used with a virtual counterpart for a circuit designed in software. We demonstrate this concept in Proxino, and elucidate the new scenarios it enables for makers, such as remote collaboration with physically distributed electronics components. We compared our hybrid system and its output with designs of real circuits to determine the difference through a system evaluation and observed minimal differences. We then present the results of an informal study with 9 users, where we gathered feedback on the effectiveness of our system in different working conditions (with a desktop, using a mobile, and with a remote collaborator). We conclude by sharing our lessons learned from our system and discuss directions for future research that blend physical and virtual prototyping for electronic circuits.

In this paper, we propose and investigate a new text entry technique using micro thumb-tip gestures. Our technique features a miniature QWERTY keyboard residing invisibly on the first segment of the user's index finger. Text entry can be carried out using the thumb-tip to tap the tip of the index finger. The keyboard layout was optimized for eyes-free input by utilizing a spatial model reflecting the users' natural spatial awareness of key locations on the index finger. We present our approach of designing and optimizing the keyboard layout through a series of user studies and computer simulated text entry tests over 1,146,484 possibilities in the design space. The outcome is a 2×3 grid with the letters highly confining to the alphabetic and spatial arrangement of QWERTY. Our user evaluation showed that participants achieved an average text entry speed of 11.9 WPM and were able to type as fast as 13.3 WPM towards the end of the experiment.

Interacting with media and games is a challenging user experience on smartwatches due to their small screens. We propose using lateral force feedback to enhance these experiences. When virtual objects on the smartwatch display visually collide or push the edge of the screen, we add haptic feedback so that the user also feels the impact. This addition creates the illusion of a virtual object that is physically hitting or pushing the smartwatch, from within the device itself. Using this approach, we extend virtual space and scenes into a 2D physical space. To create realistic lateral force feedback, we first examined the minimum change in force magnitude that is detectable by users in different directions and weight levels, finding an average JND of 49% across all tested conditions, with no significant effect of weight and force direction. We then developed a proof-of-concept hardware prototype called Jetto and demonstrated its unique capabilities through a set of impact-enhanced videos and games. Our preliminary user evaluations indicated the concept was welcomed and is regarded as a worthwhile addition to smartwatch output and media experiences.

We present WrisText - a one-handed text entry technique for smartwatches using the joystick-like motion of the wrist. A user enters text by whirling the wrist of the watch hand, towards six directions which each represent a key in a circular keyboard, and where the letters are distributed in an alphabetical order. The design of WrisText was an iterative process, where we first conducted a study to investigate optimal key size, and found that keys needed to be 55º or wider to achieve over 90% striking accuracy. We then computed an optimal keyboard layout, considering a joint optimization problem of striking accuracy, striking comfort, word disambiguation. We evaluated the performance of WrisText through a five-day study with 10 participants in two text entry scenarios: hand-up and hand-down. On average, participants achieved a text entry speed of 9.9 WPM across all sessions, and were able to type as fast as 15.2 WPM by the end of the last day.

Smart-rings are ideal for subtle and always-available haptic notifications due to their direct contact with the skin. Previous researchers have highlighted the feasibility of haptic technology in smart-rings and their promise in delivering noticeable stimulations by poking a limited set of planar locations on the finger. However, the full potential of poking as a mechanism to deliver richer and more expressive information on the finger is overlooked. With three studies and a total of 76 participants, we informed the design of PokeRing, a smart-ring capable of delivering information via stimulating eight different locations around the index finger's proximal phalanx. We report our evaluation of the performance of PokeRing in semi-realistic wearable conditions, (standing and walking), and its effective usage for information transfer with twenty-one spatio-temporal patterns designed by six interaction designers in a workshop. Finally, we present three applications that exploit PokeRing's notification usages.

We present Indutivo, a contact-based inductive sensing technique for contextual interactions. Our technique recognizes conductive objects (metallic primarily) that are commonly found in households and daily environments, as well as their individual movements when placed against the sensor. These movements include sliding, hinging, and rotation. We describe our sensing principle and how we designed the size, shape, and layout of our sensor coils to optimize sensitivity, sensing range, recognition and tracking accuracy. Through several studies, we also demonstrated the performance of our proposed sensing technique in environments with varying levels of noise and interference conditions. We conclude by presenting demo applications on a smartwatch, as well as insights and lessons we learned from our experience.

In this paper, we propose using the auricle - the visible part of the ear - as a means of expressive output to extend body language to convey emotional states. With an initial exploratory study, we provide an initial set of dynamic and static auricular postures. Using these results, we examined the relationship between emotions and auricular postures, noting that dynamic postures involving stretching the top helix in fast (e.g., 2Hz) and slow speeds (1Hz) conveyed intense and mild pleasantness while static postures involving bending the side or top helix towards the center of the ear were associated with intense and mild unpleasantness. Based on the results, we developed a prototype (called Orrechio) with miniature motors, custom-made robotic arms and other electronic components. A preliminary user evaluation showed that participants feel more comfortable using expressive auricular postures with people they are familiar with, and that it is a welcome addition to the vocabulary of human body language.

We present a self-powered module for gesture recognition that utilizes small, low-cost photodiodes for both energy harvesting and gesture sensing. Operating in the photovoltaic mode, photodiodes harvest energy from ambient light. In the meantime, the instantaneously harvested power from individual photodiodes is monitored and exploited as clues for sensing finger gestures in proximity. Harvested power from all photodiodes are aggregated to drive the whole gesture-recognition module including the micro-controller running the recognition algorithm. We design robust, lightweight algorithm to recognize finger gestures in the presence of ambient light fluctuations. We fabricate two prototypes to facilitate user's interaction with smart glasses and smart watch. Results show 99.7%/98.3% overall precision/recall in recognizing five gestures on glasses and 99.2%/97.5% precision/recall in recognizing seven gestures on the watch. The system consumes 34.6 µW/74.3 µW for the glasses/watch and thus can be powered by the energy harvested from ambient light. We also test system's robustness under varying light intensities, light directions, and ambient light fluctuations, where the system maintains high recognition accuracy (> 96%) in all tested settings.

We propose and explore actuating a smartwatch face to enable extended interactions. Five face movements are defined: rotation, hinging, translation, rising, and orbiting. These movements are incorporated into interaction techniques to address limitations of a fixed watch face. A 20-person study uses concept videos of a passive low fidelity prototype to confirm the usefulness of the actuated interaction techniques. A second 20-person study uses 3D rendered animations to access social acceptability and perceived comfort for different actuation dynamics and usage contexts. Finally, we present Cito, a high-fidelity proof-of-concept hardware prototype that investigates technical challenges.

Chronic disease is one of the most pressing health challenges facing the United States (and an increasing set of other countries). The onset or progression of diseases like obesity, diabetes, and metabolic disorder are strongly related to eating behavior, and scientists are still trying to fully understand the complex mixture of diet, exercise, genetics, sociocultural context, and physical environment that lead to these diseases. Health science, however, has no effective means for automatically measuring eating behavior in free-living conditions. The Auracle aims to be a wearable earpiece that detects eating behavior, to be fielded by health-science researchers in their efforts to study eating behavior and ultimately to develop interventions useful to individuals striving to address chronic disease related to eating.

We present Pyro, a micro thumb-tip gesture recognition technique based on thermal infrared signals radiating from the fingers. Pyro uses a compact, low-power passive sensor, making it suitable for wearable and mobile applications. To demonstrate the feasibility of Pyro, we developed a self-contained prototype consisting of the infrared pyroelectric sensor, a custom sensing circuit, and software for signal processing and machine learning. A ten-participant user study yielded a 93.9% cross-validation accuracy and 84.9% leave-one-session-out accuracy on six thumb-tip gestures. Subsequent lab studies demonstrated Pyro's robustness to varying light conditions, hand temperatures, and background motion. We conclude by discussing the insights we gained from this work and future research questions.

Smart rings have a unique form factor suitable for many applications, however, they offer little opportunity to provide the user with natural output. We propose passive kinesthetic force feedback as a novel output method for rotational input on smart rings. With this new output channel, friction force profiles can be designed, programmed, and felt by a user when they rotate the ring. This modality enables new interactions for ring form factors. We demonstrate the potential of this new haptic output method though Frictio, a prototype smart ring. In a controlled experiment, we determined the recognizability of six force profiles, including Hard Stop, Ramp-Up, Ramp-Down, Resistant Force, Bump, and No Force. The results showed that participants could distinguish between the force profiles with 94% accuracy. We conclude by presenting a set of novel interaction techniques that Frictio enables, and discuss insights and directions for future research.

The small screen size of a smartwatch limits user experience when watching or interacting with media. We propose a supplementary tactile feedback system to enhance the user experience with a method unique to the smartwatch form factor. Our system has a deformable surface on the back of the watch face, allowing the visual scene on screen to extend into 2.5D physical space. This allows the user to watch and feel virtual objects, such as experiencing a ball bouncing against the wrist. We devised two controlled experiments to analyze the influence of tactile display resolution on the illusion of virtual object presence. Our first study revealed that on average, a taxel can render virtual objects between 70% and 138% of its own size without shattering the illusion. From the second study, we found visual and haptic feedback can be separated by 4.5mm to 16.2mm for the tested taxels. Based on the results, we developed a prototype (called RetroShape) with 4×4 10mm taxels using micro servo motors, and demonstrated its unique capability through a set of tactile-enhanced games and videos. A preliminary user evaluation showed that participants welcome RetroShape as a useful addition to existing smartwatch output.

We motivate, design, and prototype a modular smartphone designed to make temporary device lending trustworthy and convenient. The concept is that the phone can be separated into pieces, so a child, friend, or even stranger can begin an access-controlled interaction with one piece, while the own-er retains another piece to continue their tasks and monitor activity. This is grounded in a survey capturing attitudes towards device lending, and an exploratory study probing how people might lend pieces of different kinds of modular smartphones. Design considerations are generated for a hardware form factor and software interface to support different lending scenarios. A functional prototype combining three smartphones into a single modular device is described and used to demonstrate a lending interaction design. A usability test validates the concept using the prototype.

Gestural interaction has become increasingly popular, as enabling technologies continue to transition from research to retail. The mobility of miniaturized (and invisible) technologies introduces new uses for gesture recognition. This paper investigates single-hand microgestures (SHMGs), detailed gestures in a small interaction space. SHMGs are suitable for the mobile and discrete nature of interactions for ubiquitous computing. However, there has been a lack of end-user input in the design of such gestures. We performed a user-elicitation study with 16 participants to determine their preferred gestures for a set of referents. We contribute an analysis of 1,632 gestures, the resulting gesture set, and prevalent conceptual themes amongst the elicited gestures. These themes provide a set of guidelines for gesture designers, while informing the designs of future studies. With the increase in hand-tracking and electronic devices in our surroundings, we see this as a starting point for designing gestures suitable to portable ubiquitous computing.

Doppio is a reconfigurable smartwatch with two touch sensitive display faces. The orientation of the top relative to the base and how the top is attached to the base, creates a very large interaction space. We define and enumerate possible configurations, transitions, and manipulations in this space. Using a passive prototype, we conduct an exploratory study to probe how people might use this style of smartwatch interaction. With an instrumented prototype, we conduct a controlled experiment to evaluate the transition times between configurations and subjective preferences. We use the combined results of these two studies to generate a set of characteristics and design considerations for applying this interaction space to smartwatch applications. These considerations are illustrated with a proof-of-concept hardware prototype demonstrating how Doppio interactions can be used for notifications, private viewing, task switching, temporary information access, application launching, application modes, input, and sharing the top.

We propose and study a new input modality, WristWhirl, that uses the wrist as an always-available joystick to perform one-handed continuous input on smartwatches. We explore the influence of the wrist's bio-mechanical properties for performing gestures to interact with a smartwatch, both while standing still and walking. Through a user study, we examine the impact of performing 8 distinct gestures (4 directional marks, and 4 free-form shapes) on the stability of the watch surface. Participants were able to perform directional marks using the wrist as a joystick at an average rate of half a second and free-form shapes at an average rate of approximately 1.5secs. The free-form shapes could be recognized by a $1 gesture recognizer with an accuracy of 93.8% and by three human inspectors with an accuracy of 85%. From these results, we designed and implemented a proof-of-concept device by augmenting the watchband using an array of proximity sensors, which can be used to draw gestures with high quality. Finally, we demonstrate a number of scenarios that benefit from one-handed continuous input on smartwatches using WristWhirl.

Physiotherapy patients exercising at home alone are at risk of re-injury since they do not have corrective guidance from a therapist. To explore solutions to this problem, we designed Physio@Home, a prototype that guides people through prerecorded physiotherapy exercises using real-time visual guides and multi-camera views. Our design addresses several aspects of corrective guidance, including: plane and range of movement, joint positions and angles, and extent of movement. We evaluated our design, com-paring how closely people could follow exercise movements under various feedback conditions. Participants were most accurate when using our visual guide and multi-views. We provide suggestions for exercise guidance systems drawn from qualitative findings on visual feedback complexity.

Distributed display environments (DDEs) allow use of various specialized devices but challenge designers to provide a clean flow of data across multiple displays. Upcoming consumer-ready head-worn displays (HWDs) can play a central role in unifying the interaction experience in such ecosystems. In this paper, we report on the design and development of Gluey, a user interface that acts as a 'glue' to facilitate seamless input transitions and data movement across displays. Based on requirements we refine for such an interface, Gluey leverages inherent headworn display attributes such as field-of-view tracking and an always-available canvas to redirect input and migrate content across multiple displays, while minimizing device switching costs. We implemented a functional prototype integrating Gluey's numerous interaction possibilities. From our experience in this integration and from user evaluation results, we identify the open challenges in using HWDs to unify the interaction experience in DDEs.

Rich interaction with high-resolution wall displays is not limited to remotely pointing at targets. Other relevant types of interaction include virtual navigation, text entry, and direct manipulation of control widgets. However, most techniques for remotely acquiring targets with high precision have studied remote pointing in isolation, focusing on pointing efficiency and ignoring the need to support these other types of interaction. We investigate high-precision pointing techniques capable of acquiring targets as small as 4 millimeters on a 5.5 meters wide display while leaving up to 93 % of a typical tablet device's screen space available for task-specific widgets. We compare these techniques to state-of-the-art distant pointing techniques and show that two of our techniques, a purely relative one and one that uses head orientation, perform as well or better than the best pointing-only input techniques while using a fraction of the interaction resources.

Mobile devices are endowed with significant sensing capabilities. However, their ability to 'see' their surroundings, during active use, is limited. We present Surround-See, a self-contained smartphone equipped with an omni-directional camera that enables peripheral vision around the device to augment daily mobile tasks. Surround-See provides mobile devices with a field-of-view collinear to the device screen. This capability facilitates novel mobile tasks such as, pointing at objects in the environment to interact with content, operating the mobile device at a physical distance and allowing the device to detect user activity, even when the user is not holding it. We describe Surround-See's architecture, and demonstrate applications that exploit peripheral 'seeing' capabilities during active use of a mobile device. Users confirm the value of embedding peripheral vision capabilities on mobile devices and offer insights for novel usage methods.

The human hand can naturally coordinate multiple finger joints, and simultaneously tilt, press and roll a pen to write or draw. For this reason, digital pens are now embedded with auxiliary input sensors to capture these actions. Prior research on auxiliary input channels has mainly investigated them in isolation of one another. In this work, we explore the coordinated use of two auxiliary channels, a class of interaction techniques we refer to as a-coord input. Through two separate experiments, we explore the design space of a-coord input. In the first study we identify if users can successfully coordinate two auxiliary channels. We found a strong degree of coordination between channels. In a second experiment, we evaluate the effectiveness of a-coord input in a task with multiple steps, such as multi-parameter selection and manipulation. We find that a-coord input facilitates coordination even with a complex, aforethought sequential task. Overall our results indicate that users can control at least two auxiliary input channels in conjunction which can facilitate a number of common tasks can on the pen.

Tabletop systems provide a versatile space for collaboration, yet, in many cases, are limited by the inability to differentiate the interactions of simultaneous users. We present See Me, See You, a lightweight approach for discriminating user touches on a vision-based tabletop. We contribute a valuable characterization of finger orientation distributions of tabletop users. We exploit this biometric trait with a machine learning approach to allow the system to predict the correct position of users as they touch the surface. We achieve accuracies as high as 98% in simple situations and above 92% in more challenging conditions, such as two-handed tasks. We show high acceptance from users, who can self-correct prediction errors without significant costs. See Me, See You is a viable solution for providing simple yet effective support for multi-user application features on tabletops.

Observational studies indicate that most people use one hand to interact with their mobile devices. Interaction on the back-of-devices (BoD) has been proposed to enhance one-handed input for various tasks, including selection and gesturing. However, we do not possess a good understanding of some fundamental issues related to one-handed BoD input. In this paper, we attempt to fill this gap by conducting three studies. The first study explores suitable selection techniques; the second study investigates the performance and suitability of the two main modes of cursor movement: Relative and Absolute; and the last study examines solutions to the problem of reaching the lower part of the device. Our results indicate that for BoD interaction, relative input is more efficient and accurate for cursor positioning and target selection than absolute input. Based on these findings provide guidelines for designing BoD interactions for mobile devices.

We present Magic Finger, a small device worn on the fingertip, which supports always-available input. Magic Finger inverts the typical relationship between the finger and an interactive surface: with Magic Finger, we instrument the user's finger itself, rather than the surface it is touching. Magic Finger senses touch through an optical mouse sensor, enabling any surface to act as a touch screen. Magic Finger also senses texture through a micro RGB camera, allowing contextual actions to be carried out based on the particular surface being touched. A technical evaluation shows that Magic Finger can accurately sense 22 textures with an accuracy of 98.9%. We explore the interaction design space enabled by Magic Finger, and implement a number of novel interaction techniques that leverage its unique capabilities.

Manipulating 3D objects on a tabletop is inherently problematic. Tabletops lack a third degree of freedom and thus require novel solutions to support even the simplest 3D manipulations. Our solution is TZee - a passive tangible widget that enables natural interactions with 3D objects by exploiting the lighting properties of diffuse illumination (DI) multi-touch tabletops. TZee is assembled from stacked layers of acrylic glass to extend the tabletop's infrared light slightly above the surface without supplemental power. With TZee, users can intuitively scale, translate and rotate objects in all three dimensions, and also perform more sophisticated gestures, like "slicing" a volumetric object, that have not been possible with existing tabletop interaction schemes. TZee is built with affordable and accessible materials, and one tabletop surface can easily support multiple TZees. Moreover, since TZee is transparent, there are numerous possibilities to augment interactions with feedback, helpful hints, or other visual enhancements. We discuss several important design considerations and demonstrate the value of TZee with several applications.

Although touch-screen laptops are increasing in popularity, users still do not comfortably rely on touch in these environments, as current software interfaces were not designed for being used by the finger. In this paper, we first demonstrate the benefits of using touch as a complementary input modality along with the keyboard and mouse or touchpad in a laptop setting. To alleviate the frustration users experience with touch, we then design two techniques, TouchCuts, a single target expansion technique, and ,i>TouchZoom,/i>, a multiple target expansion technique. Both techniques facilitate the selection of small icons, by detecting the finger proximity above the display surface, and expanding the target as the finger approaches. In a controlled evaluation, we show that our techniques improve performance in comparison to both the computer mouse and a baseline touch-based target acquisition technique. We conclude by discussing other application scenarios that our techniques support.

We introduce LensMouse, a novel device that embeds a touch-screen display -- or tangible 'lens' -- onto a mouse. Users interact with the display of the mouse using direct touch, whilst also performing regular cursor-based mouse interactions. We demonstrate some of the unique capabili-ties of such a device, in particular for interacting with auxil-iary windows, such as toolbars, palettes, pop-ups and dia-log-boxes. By migrating these windows onto LensMouse, challenges such as screen real-estate use and window man-agement can be alleviated. In a controlled experiment, we evaluate the effectiveness of LensMouse in reducing cursor movements for interacting with auxiliary windows. We also consider the concerns involving the view separation that results from introducing such a display-based device. Our results reveal that overall users are more effective with LenseMouse than with auxiliary application windows that are managed either in single or dual-monitor setups. We conclude by presenting other application scenarios that LensMouse could support.

Interaction patterns with handheld mobile devices are constantly evolving. Researchers observed that users prefer to interact with mobile device using one hand. However, only few interaction techniques support this mode of operation. We show that one-handed operations can be enhanced with coordinated interaction using for input the front and back of a mobile device, which we term as Dual-Surface interaction. We present some of the design rationale for introducing coordinated Dual-Surface interactions. We demonstrate that several tasks, including target selection, benefit from Dual-Surface input which allows users to rapidly select small targets in locations that are less accessible when interacting using the thumb with one-handed input. Furthermore, we demonstrate the benefits of virtual enhancements that are possible with behind-the-display relative input to perform complex tasks, such as steering. Our results show that Dual-Surface interactions offer numerous benefits that are not available with input on the front or the back alone.

Haptic interfaces are used increasingly in medical systems and related applications, but relatively little is known on the effectiveness of these interfaces. This paper reports a study on the perception of haptic force magnitude during hand movements. Discrimination thresholds were determined for a reference force of 1.5N in five different directions (0°, 45°, 90°, 135°, and 180°) with respect to the movement direction. We found that force discrimination thresholds detected were significantly higher during hand movement than those reported previously without hand movement, indicating that the perception of force magnitude is impaired by hand movement. The results also show there is no significant difference between the discrimination thresholds found for fast (28mm/s) and slow (14mm/s) hand movements. Finally, we found that the perception of force magnitude is impaired at a force direction of 45° with respect to the hand movement, indicating the existence of an oblique effect.

Many systems provide the user with a limited viewport of a larger graphical workspace. In these systems, the user often needs to find and select targets that are in the workspace, but not visible in the current view. Standard methods for navigating to the off-screen targets include scrolling, panning, and zooming; however, these are laborious when users cannot see a target's direction or distance. Techniques such as halos can provide awareness of targets, but actually getting to the target is still slow with standard navigation. To improve off-screen target selection, we developed a new technique called hop, which combines halos with a teleportation mechanism that shows proxies of distant objects. Hop provides both awareness of off-screen targets and fast navigation to the target context. A study showed that users are significantly faster at selecting off-screen targets with hopping than with two-level zooming or grab-and-drag panning, and it is clear that hop will be faster than either halos or proxy-based techniques (like drag-and-pop or vacuum filtering) by themselves. Hop both improves on halo-based navigation and extends the value of proxies to small-screen environments.

Multimedia content is becoming more widely used in applications resulting from the easy accessibility of high-speed networks in the public domain. An important component in multimedia content is 3D geometry, which in the past had low resolution due to acquisition, computational and network limitation, and was not able to approximate 3D surfaces realistically. Although processing speed and network capacity have been greatly increased in the last decade, the increase in demands for multimedia content surpass the increase in resources. Consequently, techniques for data simplification especially for 3D mesh data is inevitable in order to achieve shorter latency and satisfactory interactivity in applications. This paper presents a perceptual analysis to evaluate the visual quality associated with a change in level-of-detail. Our analysis is consistent to how the human visual system evaluates 3D objects in the real world and is based on the just-noticeable-difference methodology. Experimental results show that our approach presents an accurate estimation of visual quality and thus provides a systematic method to evaluate the performance of different simplification algorithms