Playing video games can enhance older adults’ well-being and social connections. However, most mainstream games rely on button-based controls that require fine motor skills, limiting accessibility. We present ElderPlay, a real-time game input translation system that enables older adults to play unmodified commercial games using intuitive, motion-based interaction. We first conducted a gesture elicitation study to derive user-defined gestures grounded in everyday experiences, which informed the design of a proof-of-concept system translating gestures into controller inputs. We then evaluated ElderPlay with two commercial Nintendo Switch games. Results show that gesture-based interaction improves enjoyment, perceived physical engagement, and performance. Rather than replacing controllers, our findings highlight the effectiveness of hybrid interaction, where gesture and controller inputs support different gameplay actions. We discuss implications for context-dependent and inclusive game interaction design.

Blind and visually impaired (BVI) people primarily rely on non-visual senses to interact with a physical environment. Doing so requires a high cognitive load to perceive and memorize the presence of a large set of objects, such as at home or in a learning setting. In this work, we explored opportunities to enable object-centric note-taking by using a 3D printing pen for interactive, personalized tactile annotations. We first identified the benefits and challenges of self-created tactile graphics in a formative diary study. Then, we developed TacNote, a system that enables BVI users to annotate, explore, and memorize critical information associated with everyday objects. Using TacNote, the users create tactile graphics with a 3D printing pen and attach them to the target objects. They capture and organize the physical labels by using TacNote’s camera-based mobile app. In addition, they can specify locations, ordering, and hierarchy via finger-pointing interaction and receive audio feedback. Our user study with ten BVI participants showed that TacNote effectively alleviated the memory burden, offering a promising solution for enhancing users’ access to information.

Body-controlled avatars provide a more intuitive method to real-time control virtual avatars but require larger environment space and more user effort. In contrast, hand-controlled avatars give more dexterous and fewer fatigue manipulations within a close-range space for avatar control but provide fewer sensory cues than the body-based method. This paper investigates the differences between the two manipulations and explores the possibility of a combination. We first performed a formative study to understand when and how users prefer manipulating hands and bodies to represent avatars’ actions in current popular video games. Based on the top video games survey, we decided to represent human avatars’ motions. Besides, we found that players used their bodies to represent avatar actions but changed to using hands when they were too unrealistic and exaggerated to mimic by bodies (e.g., flying in the sky, rolling over quickly). Hand gestures also provide an alternative to lower-body motions when players want to sit during gaming and do not want extensive effort to move their avatars. Hence, we focused on the design of hand gestures and upper-body postures. We present Puppeteer, an input prototype system that allows players directly control their avatars through intuitive hand gestures and upper-body postures. We selected 17 avatar actions discovered in the formative study and conducted a gesture elicitation study to invite 12 participants to design best representing hand gestures and upper-body postures for each action. Then we implemented a prototype system using the MediaPipe framework to detect keypoints and a self-trained model to recognize 17 hand gestures and 17 upper-body postures. Finally, three applications demonstrate the interactions enabled by Puppeteer.

Current haptic devices are usually designed to provide one type of force feedback; however, most VR scenarios require versatile force feedback, which may require the integration of different devices to provide various types of forces. In addition, besides the main haptic effects caused by the forces, multiple types of oscillation may also commonly accompany them, which are crucial for improving VR realism and immersion. Therefore, we simulate versatile force feedback by rendering the corresponding types of oscillation as the effects caused by those forces. We take inertia and impact forces as examples in this paper, and achieve versatility using the proposed device, OsciHead, on a head-mounted display (HMD), instead of integrating different devices. By controlling elastic bands' elasticity and stored power, OsciHead uses two rotatable oscillators on both sides of the HMD, in order to render various multilevel and multidimensional oscillation feedback in 2D translation and 2D rotation directions on a head. In an exploratory study, we explored different scenarios in which multiple types of oscillation could be simulated by OsciHead. We then observed oscillation level distinguishability in two just-noticeable difference (JND) studies, and evaluated the oscillation type recognition rates in a recognition study. Based on the results, we performed a VR study, which verified that the inertia and impact feedback simulated by OsciHead enhances realism and achieves versatility.

Force feedback from damped oscillation is a common effect in our daily lives, especially when shaking an elastic object, an object hanging or containing other stuff, or a container with liquid, e.g., casting with a fishing pole or wine-swirling. Such a force, affected by complex physical variations and collisions, is difficult to properly simulate using current force feedback methods. Therefore, we propose ElastOscillation on a virtual reality (VR) controller to provide 3D multilevel force feedback for damped oscillation to enhance VR experiences. ElastOscillation consists of a proxy, six elastic bands and DC motors. It leverages the motors to control the bands' elasticity to restrain the movement of the proxy, which is connected with the bands. Therefore, when users shake the ElastOscillation device, the proxy shakes or moves in corresponding ranges of movement. The users then perceive the force from oscillation at different levels. In addition, elastic force from the bands further reinforces the oscillation force feedback. We conducted a force perception study to understand users' distinguishability for perceiving oscillation forces in 1D and 2D movement, respectively. Based on the results, we performed a VR experience study to show that the force feedback provided by ElastOscillation enhances VR realism.