This paper introduces Motion Personalization, a new task that generates personalized motions aligned with text descriptions using several basic motions containing Persona. To support this novel task, we introduce a new large-scale motion dataset called PerMo (PersonaMotion), which captures the unique personas of multiple actors. We also propose a multi-modal finetuning method of a pretrained motion diffusion model called PersonaBooth. PersonaBooth addresses two main challenges: i) A significant distribution gap between the persona-focused PerMo dataset and the pretraining datasets, which lack persona-specific data, and ii) the difficulty of capturing a consistent persona from the motions vary in content (action type). To tackle the dataset distribution gap, we introduce a persona token to accept new persona features and perform multi-modal adaptation for both text and visuals during finetuning. To capture a consistent persona, we incorporate a contrastive learning technique to enhance intra-cohesion among samples with the same persona. Furthermore, we introduce a context-aware fusion mechanism to maximize the integration of persona cues from multiple input motions. PersonaBooth outperforms state-of-the-art motion style transfer methods, establishing a new benchmark for motion personalization.

We propose a novel 3D gaze redirection framework that leverages an explicit 3D eyeball structure. Existing gaze redirection methods are typically based on neural radiance fields, which employ implicit neural representations via volume rendering. Unlike these NeRF-based approaches, where the rotation and translation of 3D representations are not explicitly modeled, we introduce a dedicated 3D eyeball structure to represent the eyeballs with 3D Gaussian Splatting (3DGS). Our method generates photorealistic images that faithfully reproduce the desired gaze direction by explicitly rotating and translating the 3D eyeball structure. In addition, we propose an adaptive deformation module that enables the replication of subtle muscle movements around the eyes. Through experiments conducted on the ETH-XGaze dataset, we demonstrate that our framework is capable of generating diverse novel gaze images, achieving superior image quality and gaze estimation accuracy compared to previous state-of-the-art methods.

TapSix is a one-handed wearable keyboard that enables typing in situations where using a keyboard is not possible. It detects finger taps on six virtual keys on a tactile surface while users can type without paying visual attention to their fingers. Its unique palm-worn design provides the stable view of all five fingers for the low-cost camera sensor, even when there is unusual motion in the upper limb. The captured image is processed using the proposed algorithm that both robustly detects finger taps on keys using only geometric features and measures the distance between each finger and the tactile surface. A new letter-to-tap mapping that fully utilizes the six keys in light of learnability, anatomical comfort, and algorithm accuracy is also proposed. We demonstrate the utility of TapSix in a virtual reality environment and evaluate the algorithm’s accuracy, typing performance, and user acceptance by comparing it with three commercial virtual reality interfaces.

Recent advancements in deep learning have created numerous possibilities for real-world application, among which the recognition of human motions with sensors. We employed a convolutional neural network (CNN) to process workout motion data to solve the segmentation and recognition problem. We focused on deploying the network architecture in a mobile environment characterized by limited resources. Our experimental results were promising in terms of both segmentation and recognition. Furthermore, we analyzed the performance correlation between sampling the rate and recognition rate. This result indicated 55Hz to provide an appropriate amount of information for workout motion. Then we investigated the computational cost and memory space usage of the hyper-parameter selection. The experimental results implied that appropriate hyper-parameter selection could reduce the computational burden on the mobile environment. Subsequently, we suggest an efficiency score regarding the computational cost, memory usage, and the recognition rate. Our suggested efficiency score shows that our method could find hyper-parameters that offer minimum loss of accuracy and are computationally inexpensive with optimal utilization of memory space.

BMK stick, a Bluetooth mouse with a keyboard is a joystick-type input device based on an inertial measurement unit (IMU) sensor that recognizes human hand movements along with the combinations of five buttons and enables users to input text with one hand. It is designed as an input device for computers and virtual environments. The BMK stick has two modes, mouse mode (mode 0) and keyboard mode (mode 1), which can be set by pushing a button. In mode 0, the IMU sensor extracts the value of the user's hand motion with the 3D Euler angle. The extracted values are then converted into two-dimensional Cartesian coordinates through a built-in algorithm in the microcontroller unit (MCU). The converted value is mapped onto the computer screen to control the mouse pointer. In mode 1, text can be entered with with one hand using a five-button combination by using an AEIOU key-map provided by a commercial text-entry device, TAP. Two experiments were conducted to evaluate the performance of the BMK stick. In the first experiment, the mouse pointing performance was compared with three existing commercial ones. In the second experiment, the text-entry rate was evaluated in words per minute (WPM) compared with the two existing commercial one-handed keyboard input devices. The result shows that our proposed device in the first experiment performed better than the other commercial products. In the second experiment, even though some experimenters had some difficulties in manipulating our device, it produced more WPM than the commercial ones.

In this study, we addressed the challenging problem of 2D hand-pose tracking based on an RGB-only sequence by using a hand data generator. For training various deep networks on hand-pose tracking, we propose a synthetic hand generator based on an application. Our generator could be combined with a kinematic hand model to generalize well to unseen data. In addition, it is robust to occlusions and varying camera viewpoints and leads to anatomically smooth hand motions. Our generator also allows to set the range of each property and add objects (hand models and backgrounds) easily to the application. This greatly diversifies the architecture and improves performance of hand pose tracking. We evaluated our generator by comparing with other public hand datasets and propose a novel annotation technique for accurate 2D (3D) hand labeling and joint angles even in case of partial occlusions. We demonstrate that the dataset generated through our generator outperforms other public datasets with only challenging RGB.

We propose a light and fast hand gesture recognition method using geometric feature for a smartphone based robot and apply it to early childhood mathematics education. The feature of hand gesture is defined by the number of extrema in the plot for the distances between the center point of hand and the outer points of hand from active contour model or snakes. The region of interest (ROI) is continuously updated by Continuously Adaptive Mean Shift Algorithm (CamShift) algorithm, and the snake model is used to make the outer points sequential efficiently. A mathematics learning application for an Android OS smartphone based robot is developed using the hand gesture recognition algorithm. The experiment with Korean children (5-6 years of age) is conducted to evaluate if hand gesture based HRI could promote their mathematics learning. The result suggests that the idea of hand gesture based HRI for early childhood education is feasible and that children can learn mathematics by hand gesture based interaction with a robot.

We present a novel robotic task-practice system, i.e., adaptive and automatic presentation of tasks (ADAPT), which is designed to enhance the recovery of upper extremity functions in patients with stroke. We designed ADAPT in accordance with current training guidelines for stroke rehabilitation; ADAPT engages the patient intensively, actively, and adaptively in a variety of realistic functional tasks that require reaching and manipulation. A general-purpose robot simulates the dynamics of the functional tasks and presents these functional tasks to the patient. A novel tool-changing system enables ADAPT to automatically switch between the tools corresponding to the functional tasks. The control architecture of ADAPT is composed of three main components: a high-level task scheduler, a functional task model, and a low-level admittance controller. The high-level task scheduler adaptively selects the task to practice and sets the task difficulty based on the previous performance of the patients. The functional task model generates desired trajectories based on learned models of task dynamics. Tasks dynamics are modeled with receptive field weighted regression (RFWR), such that the feel of the task tools is accurately modeled, and the task difficulty can be easily adjusted. The low-level admittance controller,which is also learned with RFWR,implements the selected task trajectory for robot-patient interaction. The results of a preliminary experiment with a healthy subject demonstrate the successful operation of ADAPT.

We propose a new robotic task practice system designed to enhance the recovery of upper extremity functions in patients with stroke. Our system, ADAPT (ADaptive and Automatic Presentation of Tasks), which was designed in accordance to current training guidelines for stroke rehabilitation, engages the patient intensively, actively, and adaptively in a variety of realistic functional tasks that require reaching and manipulation. A general-purpose robot simulates the dynamics of the functional tasks. Based on the subject’s performance, a task scheduler adaptively selects a task and sets the task difficulty. The tool changer selects the tool corresponding to the selected task, a doorknob for instance. The low-level controller then implements the selected task with the desired difficulty on the robot during the robot-patient interaction. Our preliminary experimental results demonstrate the feasibility of our system.