Fine-grained hand interaction with Inertial Measurement Unit (IMU) and machine learning offers a low-cost and effective solution. However, the robustness and generalizability of machine learning models are highly dataset-dependent. Existing datasets for interaction design are typically constructed through extensive real user data collection, which limits interaction diversity and personalization. To address these challenges, we propose Rob2HanD, a novel data-generation tool which utilizes large language models (LLMs) to regulate the motion processes of the robotic arm and rapidly constructs IMU datasets. Rob2HanD demonstrates the capability to generate large and usable IMU interaction datasets under few-shot or zero-shot conditions, thereby enhancing the potential for diverse and personalized fine-grained hand interactions. Using a real human dataset, we evaluate machine learning models trained on Rob2HanD-generated data and validate the usability of Rob2HanD. In real-world applications, models trained on Rob2HanD-generated datasets demonstrate strong performance across a variety of customized interaction tasks.

Simultaneous Localization and Mapping (SLAM) is critical for real-time robotic applications, enabling precise localization and comprehensive scene reconstruction. Recent advances in 3D Gaussian Splatting (3DGS) enable high-quality view synthesis and rapid rendering, yet robust and consistent semantic scene representation remains under-explored. In this work, we introduce a dense semantic SLAM framework that integrates high-dimensional semantic features with an explicit 3D Gaussian-based scene representation to address these challenges. Our approach employs a lightweight projection layer that maps low-dimensional semantic features to high-dimensional embeddings, a coarse-to-fine and semantically informed camera tracking strategy that robustly estimates camera poses, a mapping module that incrementally refines the Gaussian map by simultaneously leveraging geometric and photometric cues alongside semantic information and a covisibility-based local bundle adjustment module for joint optimization of camera poses and Gaussian parameters. Extensive experiments on synthetic and real-world indoor datasets demonstrate that our framework achieves superior reconstruction quality, enhanced semantic segmentation accuracy, and competitive camera pose estimation.

Individuals with upper-body motor impairments, but having full lower-body functionality, can use their feet to interact with mobile devices, such as smartphones and tablets. However, they face significant challenges controlling mobile devices and entering text via touchscreen with their feet. Moreover, such interactions are typically limited to specific scenarios, such as sitting on a bed. To address this, we developed a wearable, pressure-sensor-based foot interaction system featuring a novel two-key text entry method. Two use case studies, with a total of 18 participants, demonstrated that our system enables these individuals to control devices and achieve acceptable text entry speeds in diverse scenarios.

Though many approaches have been proposed for Automated Program Repair (APR) and indeed achieved remarkable performance, they still have limitations in fixing bugs that require analyzing and reasoning about the logic of the buggy program. Recently, large language models (LLMs) instructed by prompt engineering have attracted much attention for their powerful ability to address many kinds of tasks including bug-fixing. However, the quality of the prompt will highly affect the ability of LLMs and manually constructing high-quality prompts is a costly endeavor. To address this limitation, we propose a self-directed LLM-based automated program repair, ThinkRepair, with two main phases: collection phase and fixing phase. The former phase automatically collects various chains of thoughts that constitute pre-fixed knowledge by instructing LLMs with the Chain-of-Thought (CoT) prompt. The latter phase targets fixing a bug by first selecting examples for few-shot learning and second automatically interacting with LLMs, optionally appending with feedback of testing information. Evaluations on two widely studied datasets (Defects4J and QuixBugs) by comparing ThinkRepair with 12 SOTA APRs indicate the priority of ThinkRepair in fixing bugs. Notably, ThinkRepair fixes 98 bugs and improves baselines by 27%∼344.4% on Defects4J V1.2. On Defects4J V2.0, ThinkRepair fixes 12∼65 more bugs than the SOTA APRs. Additionally, ThinkRepair also makes a considerable improvement on QuixBugs (31 for Java and 21 for Python at most).

Constrained image splicing detection and localization (CISDL) is a fundamental task of multimedia forensics, which detects splicing operation between two suspected images and localizes the spliced region on both images. Recent works regard it as a deep matching problem and have made significant progress. However, existing frameworks typically perform feature extraction and correlation matching as separate processes, which may hinder the model's ability to learn discriminative features for matching and can be susceptible to interference from ambiguous background pixels. In this work, we propose a multi-scale target-aware framework to couple feature extraction and correlation matching in a unified pipeline. In contrast to previous methods, we design a target-aware attention mechanism that jointly learns features and performs correlation matching between the probe and donor images. Our approach can effectively promote the collaborative learning of related patches, and perform mutual promotion of feature learning and correlation matching. Additionally, in order to handle scale transformations, we introduce a multi-scale projection method, which can be readily integrated into our target-aware framework that enables the attention process to be conducted between tokens containing information of varying scales. Our experiments demonstrate that our model, which uses a unified pipeline, outperforms state-of-the-art methods on several benchmark datasets and is robust against scale transformations.

For comprehensively participating in society, independent and safe mobility is an important skill for many daily activities. Spatial cognition is one of the most important human capabilities and addresses the acquisition, processing and utilization of knowledge about the spatial layout of environments. Humans predominantly use the visual sense for this and for blind and low vision people, the lack of spatial perception reduces their quality of life and their ability of independent living. In particular the spatial navigation in unknown environments imposes challenges, since there is no possibility to train navigation tasks in advance. Today, blind and visually impaired people still rely on traditional navigation aids such as a cane for micro-navigation, which - however - does not help for developing orientation at larger scale or for planning of routes. To overcome this problem, this paper introduces the concept of a virtual environment that allows experiencing unknown locations by real walking while still staying in a sage controlled environment. Since this virtual environment can be controlled in its complexity, it can be adjusted from an abstract training scenario to a real-life situation such as train stations or airports.

In daily life, a guide dog as a companion assists blind and visually impaired people (BVIP) to perform independent and safe journey. However, due to several reasons (e.g., cost and long-term training) only a few BVIP would own their guide dogs. To let much more BVIP would have accessible guidance services in unfamiliar public or private areas, we plan to develop an interactive guide dog robot prototype. In this paper, we present one of the most important components of a guidance robot for BVIP, an interactive haptic rein which consists of force-sensing sensors to control and balance the walking speed between BVIP and robots in a natural way, and vibrated actuators under fingers to acquire haptic information, e.g., turning left/right. The results of preliminary user studies indicated that the proposed haptic rein allow BVIP to control and communicate with a guidance robot via a natural haptic interaction.

In the paper we present our Range-IT prototype, which is a 3D depth camera based electronic travel aid (ETA) to assist visually impaired people in finding out detailed information of surrounding objects. In addition to detecting indoor obstacles and identifying several objects of interest (e.g., walls, open doors and stairs) up to 7 meters, the Range-IT system employs a multimodal audio-vibrotactile user interface to present this spatial information.

Blind and visually impaired people often use an "object-to-object" strategy to explore unknown spaces through physical contact via their canes or bodies. Camera-based mobile electronic travel aids (ETAs) not only offer a larger work range than a white cane while detecting obstacles, but also enable users to recognise objects of interest without physical contact. In this paper, we conducted a case study with seven visually impaired participants, to investigate how they use a depth-sensing camera-based ETA for exploring unknown spaces, and how they reconstruct cognitive mappings of surrounding objects. We found that camera-based mobile ETAs would assist visually impaired users to explore the surrounding environment effectively and without physical contact. Furthermore, our original study indicates ETA users change their strategies for exploring the surrounding environment from using an "object-to-object" approach to an "observation-point to observation-point" strategy. Participants improved their cognitive mappings of the surrounding environment compared to white cane users.

Supporting blind and visually impaired people with digital maps that can enhance spatial awareness and learning about the surrounding environment is a challenge. This study introduces an audio-tactile you-are-here map system that presents map elements and users' updated location on a mobile pin-matrix display for blind and visually impaired people. In addition to panning and zooming a map, in the system, a set of tactile map symbols consisting of raised and lowered pins have been proposed to present varying map elements. A field test with eight visually impaired subjects and eight blindfolded subjects who did not have experience with tactile maps and Braille was conducted. Subjects would locate surrounding streets easily after a short-time training, as well as nearby point-of-interests (POIs). The results of the evaluation indicated that the mean relative distance error was significantly lower with nearby POIs, which do not require panning while exploring, than far away POIs. Furthermore, it is important to improve the portability of the proposed prototype and develop a one-hand map exploration method.

Due to inaccurate GPS data and complex environment close to buildings, for blind people it is often hard to find entrances in unknown areas. In this paper, we propose an enhance method that combines crowd-sourcing technology and location-based services, in order to help blind pedestrians find entrances independently in unknown regions. Besides collecting environmental accessibility information collaboratively, within this method a reference point has to be created for each entrance. Users are guided to a reference point by current GPS-based navigation systems, and rely on the environmental accessibility information walking from the reference point to the target entrance finally. A pilot study with 10 subjects (5 blind and 5 blindfolded) indicated that, the proposed method would help blind people find entrances in unfamiliar areas, and we found the blind and blindfolded individuals performed differently due to their different levels of orientation and mobility skills. Overall, it is promising to integrate the proposed approach into existing navigation systems for blind and visually impaired people.

In this article, we propose and evaluate a novel audio-haptic map system to improve map accessibility for the visually impaired while reading city maps and making pre-journey routes. The system employs a touch-sensitive pin-matrix display and represents various map elements through a pre-defined set of tactile map symbols consisting of raised and lowered pins. The interactive user interface allows users to not only acquire auditory and Braille geographic information by touching the involved map elements, but also prepare pre-journey routes by exploring large-scale areas through panning and zooming. Furthermore, the positive results of the evaluation conducted with 10 blind users indicate that blind users are able to use the proposed map system to read street maps and make pre-journey routes in unknown areas, with acquired configurational knowledge and route knowledge. Two-thirds of participants prefer swell-paper maps with only two zoom levels.

Proximity-based interaction through a long cane is essential for the blind and the visually impaired. We designed and implemented an obstacle detector consisting of a 3D Time-of-Flight (TOF) camera and a planar tactile display to extend the interaction range and provide rich non-visual information about the environment. Users choose a better path after acquiring the spatial layout of obstacles than with a white cane alone. A user study with 6 blind people was analyzed and showed extra time is needed to ensure safe walking while reading the layout. Both hanging and ground-based obstacles were circumvented. Tactile mapping information has been designed for representation of precise spatial information around a blind user.