A quintessential feature of human intelligence is the ability to create ad hoc conventions over time to achieve shared goals efficiently. We investigate how communication strategies evolve through repeated collaboration as people coordinate on shared procedural abstractions. To this end, we conducted an online unimodal study (n = 98) using natural language to probe abstraction hierarchies. In a follow-up lab study (n = 40), we examined how multimodal communication (speech and gestures) changed during physical collaboration. Pairs used augmented reality to isolate their partner’s hand and voice; one participant viewed a 3D virtual tower and sent instructions to the other, who built the physical tower. Participants became faster and more accurate by establishing linguistic and gestural abstractions and using cross-modal redundancy to emphasize key changes from previous interactions. Based on these findings, we extend probabilistic models of convention formation to multimodal settings, capturing shifts in modality preferences. Our findings and model provide building blocks for designing convention-aware intelligent agents situated in the physical world.

Target disambiguation is crucial in resolving input ambiguity in augmented reality (AR), especially for queries over distant objects or cluttered scenes on the go. Yet, visual feedforward techniques that support this process remain underexplored. We present Uncertain Pointer, a systematic exploration of feedforward visualizations that annotate multiple candidate targets before user confirmation, either by adding distinct visual identities (e.g., colors) to support disambiguation or by modulating visual intensity (e.g., opacity) to convey system uncertainty. First, we construct a pointer space of 25 pointers by analyzing existing placement strategies and visual signifiers used in target visualizations across 30 years of relevant literature. We then evaluate them through two online experiments (n = 60 and 40), measuring user preference, confidence, mental ease, target visibility, and identifiability across varying object distances and sparsities. Finally, from the results, we derive design recommendations in choosing different Uncertain Pointers based on AR context and disambiguation techniques.

This paper explores the feasibility of deliberately designing VR motion that diverges from users’ physical movements to turn mundane, everyday transportation motion (e.g., metros, trains, and cars) into more entertaining VR motion experiences, in contrast to prior car-based VR approaches that synchronize VR motion to physical car movement exactly. To gain insight into users’ preferences for veering rate and veering direction for turning (left/right) and pitching (up/down) during the three phases of acceleration (accelerating, cruising, and decelerating), we conducted a formative, perceptual study (n=24) followed by a VR experience evaluation (n=18), all conducted on metro trains moving in a mundane, straight-line motion. Results showed that participants preferred relatively high veering rates, and preferred pitching upward during acceleration and downward during deceleration. Furthermore, while veering decreased comfort as expected, it significantly enhanced immersion (p<.01) and entertainment (p<.001) and the overall experience, with comfort being considered, was preferred by 89% of participants.

Interior design aims to create aesthetically pleasing and functional environments within an architectural space. For a simple room, the preliminary design exploration currently takes multiple meetings and days of work for interior designers to incorporate homeowners’ personal preferences through layout, furnishings, form, colors, and materials. We present RoomDreaming, a generative AI-based approach designed to facilitate preliminary interior design exploration. It empowers owners and designers to rapidly and efficiently iterate through a broad range of AI-generated, photo-realistic design alternatives, each uniquely tailored to fit actual space layouts and individual design preferences. We conducted a series of formative and summative studies with a total of 18 homeowners and 20 interior designers to help design, improve, and evaluate RoomDreaming. Owners reported that RoomDreaming effectively increased the breadth and depth of design exploration with higher efficiency and satisfaction. Designers reported that one hour of collaborative designing with RoomDreaming yielded results comparable to several days of traditional owner-designer meetings, plus days to weeks worth of designer work to develop and refine designs.

Real-world impact, such as hitting a tennis ball and a baseball, generates instantaneous, directional impact forces. However, current ungrounded force feedback technologies, such as air jets and propellers, can only generate directional impulses that are 10x-10,000x weaker. We present SpinShot, a flywheel-based device with a solenoid-actuated stopper capable of generating directional impulse of 22Nm in 1ms, which is more than 10x stronger than prior ungrounded directional technologies. Furthermore, we present a novel force design that reverses the flywheel immediately after the initial impact, to significantly increase the perceived magnitude. We conducted a series of two formative, perceptual studies (n=16, 18), followed by a summative user experience study (n=16) that compared SpinShot vs. moving mass (solenoid) and vs. air jets in a VR baseball hitting game. Results showed that SpinShot significantly improved realism, immersion, magnitude (p < .01) compared to both baselines, but significantly reduced comfort vs. air jets primarily due to the 2.9x device weight. Overall, SpinShot was preferred by 63-75% of the participants.

Walking is a cyclic pattern of alternating footstep strikes, with each pair of steps forming a stride, and a series of strides forming a gait. We conduct a systematic examination of different kinds of intentional variations from a normal gait that could be used as input actions without interrupting overall walking progress. A design space of 22 candidate Gait Gestures is generated by adapting previous standing foot input actions and identifying new actions possible in a walking context. A formative study (n=25) examines movement easiness, social acceptability, and walking compatibility with foot movement logging to calculate temporal and spatial characteristics. Using a categorization of these results, 7 gestures are selected for a wizard-of-oz prototype demonstrating an AR interface controlled by Gait Gestures for ordering food and audio playback while walking. As a technical proof-of-concept, a gait gesture recognizer is developed and tested using the formative study data.

Impact events, which generate directional forces with extremely short impulse durations and large force magnitudes, are prevalent in both virtual reality (VR) games and real-world experiences. However, despite recent advancement in ungrounded force feedback technologies, such as air jet propulsion and propellers, these technologies remain 5-100x weaker and 10-500x slower compared to real-world impact events. For instance, they can only achieve 4N with a minimal duration of 50-500ms compared to the 20-400N forces generated within 1-5ms for baseball, ping-pong, drumming, and tennis. To overcome these limitations, we present AirCharge, a novel haptic device that accumulates air propulsion momentum to generate instantaneous, directional impact forces. By mounting compressed air jets on rotating swingarms, AirCharge can amplify impact force magnitude by more than 10x while matching real-world impulse duration of 3ms. To support high-frequency impacts, we explored and evaluated a series of device designs, culminating in a novel reciprocating dual-swingarm design that leverages a reversing bevel gearbox to eliminate gyro effects and to achieve impact feedback of up to 10Hz. User experience evaluation (n = 16) showed that AirCharge significantly enhanced realism and is preferred by participants compared to air jets without the charging mechanism.

We present AirRacket, perceptual modeling and design of ungrounded, directional force feedback for virtual racket sports. Using compressed air propulsion jets to provide directional impact forces, we iteratively designed for three popular sports that span a wide range of force magnitudes: ping-pong, badminton, and tennis. To address the limited force magnitude of ungrounded force feedback technologies, we conducted a perception study which discovered the novel illusion that users perceive larger impact force magnitudes with longer impact duration, by an average factor of 2.57x. Through a series of formative, perceptual, and user experience studies with a combined total of 72 unique participants, we explored several perceptual designs using force magnitude scaling and duration scaling methods to expand the dynamic range of perceived force magnitude. Our user experience evaluation showed that perceptual designs can significantly improve realism and preference vs. physics-based designs for ungrounded force feedback systems.

Haptic feedback significantly enhances virtual experiences. However, supporting haptics currently requires modifying the codebase, making it impractical to add haptics to popular, high-quality experiences such as best selling games, which are typically closed-source. We present HapticSeer, a multi-channel, black-box, platform-agnostic approach to detecting game events for real-time haptic feedback. The approach is based on two key insights: 1) all games have 3 types of data streams: video, audio, and controller I/O, that can be analyzed in real-time to detect game events, and 2) a small number of user interface design patterns are reused across most games, so that event detectors can be reused effectively. We developed an open-source HapticSeer framework and implemented several real-time event detectors for commercial PC and VR games. We validated system correctness and real-time performance, and discuss feedback from several haptics developers that used the HapticSeer framework to integrate research and commercial haptic devices.

Strength training improves overall health, well-being, physical appearance, and sports performance.There are four major factors that affect training efficacy in a training session: exercise type, number of repetitions, movement velocity, and workload. Prior research has used wearable sensors to detect exercise type, number of repetitions, and movement velocity while training. However, detecting workload remains constrained to instrumented exercise equipment, such as smart exercise machines or RFID-tagged free weights.This paper presents MuscleSense, an approach that estimates exercise workload by using wearable Surface Electromyography (sEMG) sensors and regression analysis. We evaluated the accuracy of several regression models and the effects of sensor placement through a 20-person user study. Results showed that MuscleSense achieved an accuracy of 0.68kg (root mean square error, RMSE) in sensing workload using both forearm and arm sensors and support vector regression (SVR).

IEEE 802.11 wireless network supports multiple link rates at the physical layer. Each link rate is associated with a certain required Signal-to-Interference-and-Noise Ratio (SINR) threshold for successfully decoding received packets. On transmission failures, the 802.11 DCF performs a binary exponential backoff mechanism to discourage channel access attempts, hoping to reduce congestion. When traditional link adaptation is applied, both rate reduction and binary backoff represent double penalties for this wireless link, which may cause overly conservative transmission attempts. On the other hand, once transmission succeeds, 802.11 DCF resets the backoff contention window to the minimum value to encourage channel access attempts. At the same time, traditional link adaptation may also decide to increase the data rate, which leads to overly aggressive transmission attempts. We observe this improper interaction of link rate and backoff mechanism that harms the 802.11 system performance, due to separate consideration of those two parameters. In this paper, we propose to jointly adapt the rate and backoff parameters. Specifically, an Enhanced Adaptation of link Rate and Contention window, abbreviated as EARC, is devised. EARC is a closed-loop (receiver-assisted) link rate adaptation protocol that jointly considers the backoff mechanism. With only one extra byte carried by the DATA packet, EARC incurs little controlling overhead despite its receiver-assisted nature. Moreover, since SINR information commonly utilized by receiver-assisted protocols is not precisely supported in real devices, we introduce a rate selection reference (RSR) table empirically derived by constantly monitoring the environmental energy level and reception behavior. The RSR table then guides the receiver to select the best sustainable rate for the transmitter. Simulation results demonstrate the RSR table is a practical option for making the rate decision, and the proposed EARC approach is effective in maintaining high system throughput, compared to other link adaptation algorithms.

IEEE 802.11 wireless network supports multiple link rates at the physical layer. Each link rate is associated with a certain required Signal-to-Interference-and-Noise Ratio (SINR) threshold for successfully decoding received packets. Suppose constant noise and no power adjustment strategy exists, apparently SINR is solely affected by the accumulated interference power level 1. The method of selecting an appropriate link rate for transmitting/retransmitting packets is generally known as the link adaptation mechanism. Traditional link adaptation approaches try to reduce the transmit rate (hence lower SINR threshold is required) on transmission failures (potentially due to the increased denominator I of SINR), whereas upgrade the transmit rate (hence higher SINR threshold is required) on successful transmissions (potentially due to the decreased denominator I of SINR). The accumulated interference power level I in some sense indicates the medium congestion status. In 802.11, on transmission failures, the DCF performs a binary exponential backoff mechanism to discourage channel access attempts, hoping to reduce congestion. When traditional link adaptation is applied, both rate reduction and binary backoff represent double penalties for this wireless link, which may cause overly conservative transmission attempts. On the other hand, once transmission succeeds, 802.11 DCF resets the backoff contention window to the minimum value to encourage channel access attempts. At the same time, traditional link adaptation may also decide to increase the data rate, which leads to overly aggressive transmission attempts. We observe this improper interaction of link rate and backoff mechanism that harms the 802.11 system performance, due to separate consideration of those two parameters. In this paper, rather than independently dealing with the two parameters, we propose to perform link adaptations by firstly considering if a proper backoff window has been reached. Specifically, if the medium congestion level I can be reduced by imposing a larger backoff window on transmissions, then there may be no need to decrease the link rate, given SINR can be sustained. Conversely, if there is extra interference that may be tolerated in I, a smaller backoff window can be used to encourage more transmission activities while keeping the required SINR. In particular, a joint Adaptation of link Rate and backoff Contention window, abbreviated as ARC, is devised. Our ARC protocol first estimates the optimal contention window (optCW) based on Calz's approximation methods. On transmission successes (failures), the current contention window size cwpshould be compared with optCW. If cwp> optCW (cwppis decreased (increased) to perform more aggressive (conservative) transmission attempts while leaving the link rate R unchanged. Otherwise, R is upgraded (reduced) to the next higher (lower) rate. One nice property of ARC is the ability to intelligently maintain link stability, avoiding unnecessary rate fluctuations. Simulation results show that the proposed ARC protocol outperforms several traditional link adaptation mechanisms. We also propose an analytic Markov chain model on ARC operations for performance validation.