CHI ’25, Yokohama, Japan

Accurate and efficient segmentation of unknown objects in unstructured environments is essential for robotic manipulation. Unknown Object Instance Segmentation (UOIS), which aims to identify all objects in unknown categories and backgrounds, has become a key capability for various robotic tasks. However, existing methods struggle with over-segmentation and under-segmentation, leading to failures in manipulation tasks such as grasping. To address these challenges, we propose QuBER (Quadruple Boundary Error Refinement), a novel error-informed refinement approach for high-quality UOIS. QuBER first estimates quadruple boundary errors-true positive, true negative, false positive, and false negative pixels-at the instance boundaries of the initial segmentation. It then refines the segmentation using an error-guided fusion mechanism, effectively correcting both fine-grained and instance-level segmentation errors. Extensive evaluations on three public benchmarks demonstrate that QuBER outperforms state-of-the-art methods and consistently improves various UOIS methods while maintaining a fast inference time of less than 0.1 seconds. Furthermore, we show that QuBER improves the success rate of grasping target objects in cluttered environments. Code and supplementary materials are available at https://sites.google.com/view/uois-quber.

Grasp detection requires flexibility to handle objects of various shapes without relying on prior object knowledge, while also offering intuitive, user-guided control. In this paper, we introduce GraspSAM, an innovative extension of the Segment Anything Model (SAM) designed for prompt-driven and category-agnostic grasp detection. Unlike previous methods, which are often limited by small-scale training data, Grasp-SAM leverages SAM's large-scale training and prompt-based segmentation capabilities to efficiently support both target-object and category-agnostic grasping. By utilizing adapters, learnable token embeddings, and a lightweight modified decoder, GraspSAM requires minimal fine-tuning to integrate object segmentation and grasp prediction into a unified frame-work. Our model achieves state-of-the-art (SOTA) performance across multiple datasets, including Jacquard, Grasp-Anything, and Grasp-Anything++. Extensive experiments demonstrate GraspSAM's flexibility in handling different types of prompts (such as points, boxes, and language), highlighting its robustness and effectiveness in real-world robotic applications. Robot demonstrations, additional results, and code can be found at https://gistailab.github.io/GraspSAM/.

Reliable navigation of autonomous vessels critically depends on robust situational awareness, particularly object detection. For this, an accurate, 360-degree perception of the surrounding environment is essential. However, most existing datasets lack the comprehensive multi-view data required for this full environmental coverage. This absence of large-scale, multi-view image datasets specifically designed for maritime situational awareness on vessels presents a significant challenge. To address this, we introduce the Multi-View Maritime Vision (MV2) dataset, comprising 159,386 visible-light images captured from six distinct viewpoints around a vessel. MV2 provides a complete 360-degree omnidirectional perspective, offering critical support for maritime situational awareness applications. The dataset includes object bounding boxes, along with semantic, instance, and panoptic segmentation labels, and encompasses a wide range of environmental conditions, supporting diverse computer-vision tasks. Additionally, we benchmarked state-of-the-art object-detection and panoptic-segmentation models on MV2, demonstrating its contribution to advancing maritime autonomy research. The dataset is available at https://sites.google.com/view/multi-view-maritime-vision.

This study addresses the challenge of robust object detection in maritime environments, where dynamic conditions such as fog, brightness variations, and motion blur can degrade accuracy. We propose a novel framework, Joint Semantic Learning (JSL), which combines ocean scene segmentation and object detection to improve both performance and robustness. JSL incorporates the ocean scene segmentation module into the detection network during training and removes it during inference, ensuring no additional computational overhead. Through ocean scene segmentation, the feature extractor learns to understand the overall context of the image and extract detailed information about objects. Extensive experiments show that JSL, applied to various convolutional neural network-based detectors, achieves significant performance improvements on maritime datasets SMD and SeaShips. Notably, the proposed method shows substantial performance gains on the SMD-C and SeaShips-C datasets, which include adverse conditions, demonstrating the robustness of the proposed method. Furthermore, experiments comparing our method with existing state-of-the-art multi-task methods on the Cityscapes dataset validate its effectiveness in generalizing to urban environments. The efficient integration of spatial and semantic information of JSL ensures accurate and reliable object detection across diverse applications. Our code is available at: https://github.com/gistailab/JSL.

Test-time adaptation (TTA) aims to adapt a pre-trained model to a new test domain without access to source data after deployment. Existing approaches typically rely on self-training with pseudo-labels since ground-truth cannot be obtained from test data. Although the quality of pseudo labels is important for stable and accurate long-term adaptation, it has not been previously addressed. In this work, we propose DPLOT, a simple yet effective TTA framework that consists of two components: (1) domain-specific block selection and (2) pseudo-label generation using paired-view images. Specifically, we select blocks that involve domain-specific feature extraction and train these blocks by entropy minimization. After blocks are adjusted for current test domain, we generate pseudo-labels by averaging given test images and corresponding flipped counterparts. By simply using flip augmentation, we prevent a decrease in the quality of the pseudo-labels, which can be caused by the domain gap resulting from strong augmentation. Our experimental results demonstrate that DPLOT outperforms previous TTA methods in CIFAR10-C, CIFAR100-C, and ImageNet-C benchmarks, reducing error by up to 5.4%, 9.1%, and 2.9%, respectively. Also, we provide an extensive analysis to demonstrate effectiveness of our framework. Code is available at https://github.com/gist-ailab/domain-specific-block-selection-and-paired-view-pseudo-labeling-for-online-TTA.

Abstract Multi-agent trajectory prediction is crucial to autonomous driving and understanding the surrounding environment. Learning-based approaches for multi-agent trajectory prediction, such as primarily relying on graph neural networks, graph transformers, and hypergraph neural networks, have demonstrated outstanding performance on real-world datasets in recent years. However, the hypergraph transformer-based method for trajectory prediction is yet to be explored. Therefore, we present a M ultisc A le R elational T ransformer ( MART ) network for multi-agent trajectory prediction. MART is a hypergraph transformer architecture to consider individual and group behaviors in transformer machinery. The core module of MART is the encoder, which comprises a Pair-wise Relational Transformer (PRT) and a Hyper Relational Transformer (HRT). The encoder extends the capabilities of a relational transformer by introducing HRT, which integrates hyperedge features into the transformer mechanism, promoting attention weights to focus on group-wise relations. In addition, we propose an Adaptive Group Estimator (AGE) designed to infer complex group relations in real-world environments. Extensive experiments on three real-world datasets (NBA, SDD, and ETH-UCY) demonstrate that our method achieves state-of-the-art performance, enhancing ADE/FDE by 3.9%/11.8% on the NBA dataset. Code is available at https://github.com/gist-ailab/MART .

The study addresses the foundational and challenging task of peg-in-hole assembly in robotics, where misalignments caused by sensor inaccuracies and mechanical errors often result in insertion failures or jamming. This research introduces PolyFit, representing a paradigm shift by transitioning from a reinforcement learning approach to a supervised learning methodology. PolyFit is a Force/Torque (F/T)-based supervised learning framework designed for 5-DoF peg-in-hole assembly. It utilizes F/T data for accurate extrinsic pose estimation and adjusts the peg pose to rectify misalignments. Extensive training in a simulated environment involves a dataset encompassing a diverse range of peg-hole shapes, extrinsic poses, and their corresponding contact F/T readings. The study proposes a sim-to-real adaptation method for real-world application, using a sim-real paired dataset to enable effective generalization to complex and unseen polygon shapes. Real-world evaluations demonstrate substantial success rates of 96.7% and 91.3%, highlighting the robustness and adaptability of the proposed method. Videos of data generation and experiments are available online at https://sites.google.com/view/polyfit-peginhole.

Deep neural networks have demonstrated remarkable performance in various vision tasks, but their success heavily depends on the quality of the training data. Noisy labels are a critical issue in medical datasets and can significantly degrade model performance. Previous clean sample selection methods have not utilized the well pre-trained features of vision foundation models (VFMs) and assumed that training begins from scratch. In this paper, we propose CUFIT, a curriculum fine-tuning paradigm of VFMs for medical image classification under label noise. Our method is motivated by the fact that linear probing of VFMs is relatively unaffected by noisy samples, as it does not update the feature extractor of the VFM, thus robustly classifying the training samples. Subsequently, curriculum fine-tuning of two adapters is conducted, starting with clean sample selection from the linear probing phase. Our experimental results demonstrate that CUFIT outperforms previous methods across various medical image benchmarks. Specifically, our method surpasses previous baselines by 5.0\%, 2.1\%, 4.6\%, and 5.8\% at a 40\% noise rate on the HAM10000, APTOS-2019, BloodMnist, and OrgancMnist datasets, respectively. Furthermore, we provide extensive analyses to demonstrate the impact of our method on noisy label detection. For instance, our method shows higher label precision and recall compared to previous approaches. Our work highlights the potential of leveraging VFMs in medical image classification under challenging conditions of noisy labels.

Out-of-distribution (OOD) detection involves binary classification whether the given data is from outside the training data or not. Previous studies proposed outlier exposure (OE) that trains the model on an outlier dataset designed to represent potential future OOD data, thereby enhancing OOD detection performance. However, obtaining an outlier dataset representing all possible future OOD data can be challenging, and such dataset may be unavailable in some cases. This study proposes a novel approach to expose the model to jigsaw puzzles generated from training images as the outlier data. Specifically, the model is trained to have a low LogitNorm for given jigsaw puzzles. We argue that jigsaw puzzles can effectively represent future OOD data because they contain similar background information as the in-distribution data but with their semantic information destroyed. Our experimental results demonstrate that our approach outperforms previous competitive OOD detection methods and effectively detects semantically shifted OOD examples. Our code is available at https://github.com/gist-ailab/jigsaw-training-OOD.

Automatic sleep scoring is essential for the diagnosis and treatment of sleep disorders and enables longitudinal sleep tracking in home environments. Conventionally, learning-based automatic sleep scoring on single-channel electroencephalogram (EEG) is actively studied because obtaining multi-channel signals during sleep is difficult. However, learning representation from raw EEG signals is challenging owing to the following issues: (1) sleep-related EEG patterns occur on different temporal and frequency scales and 2) sleep stages share similar EEG patterns. To address these issues, we propose an automatic Sleep scoring framework that incorporates (1) a feature Pyramid and 2) supervised Contrastive learning, named SleePyCo. For the feature pyramid, we propose a backbone network named SleePyCo-backbone to consider multiple feature sequences on different temporal and frequency scales. Supervised contrastive learning allows the network to extract class discriminative features by minimizing the distance between intra-class features and simultaneously maximizing that between inter-class features. Comparative analyses on four public datasets demonstrate that SleePyCo consistently outperforms existing frameworks based on single-channel EEG. Extensive ablation experiments show that SleePyCo exhibited an enhanced overall performance, with significant improvements in discrimination between sleep stages, especially for N1 and rapid eye movement (REM). Source code is available at https://github.com/gist-ailab/SleePyCo.

Detecting out-of-distribution (OOD) inputs during the inference stage is crucial for deploying neural networks in the real world. Previous methods typically relied on the highly activated feature map outputted by the network. In this study, we revealed that the norm of the feature map obtained from a block other than the last block can serve as a better indicator for OOD detection. To leverage this insight, we propose a simple framework that comprises two metrics: FeatureNorm, which computes the norm of the feature map, and NormRatio, which calculates the ratio of FeatureNorm for ID and OOD samples to evaluate the OOD detection performance of each block. To identify the block that provides the largest difference between FeatureNorm of ID and FeatureNorm of OOD, we create jigsaw puzzles as pseudo OOD from ID training samples and compute NormRatio, selecting the block with the highest value. After identifying the suitable block, OOD detection using FeatureNorm outperforms other methods by reducing FPR95 by up to 52.77% on CIFAR10 benchmark and up to 48.53% on ImageNet benchmark. We demonstrate that our framework can generalize to various architectures and highlight the significance of block selection, which can also improve previous OOD detection methods. Our code is available at https://github.com/gistailab/block-selection-for-OOD-detection.

Neural networks (NN) have shown promising performance in point cloud segmentation (PCS). However, the measured points are too numerous to be used as model input at once. It results in a long inference time and high computational cost due to iterative sampling and inference. This study proposes Probability Propagation (PP) as a stochastic upsampling method. PP propagates the predicted probability of a sampled part of a point cloud into the other unpredicted points by considering proximity. By replacing the iterative inference of NN with PP, large point clouds can be dealt with quickly and efficiently. We investigated the effectiveness of PP using the ShapeNet benchmark on various settings: sampling methods (random, farthest point, and Poisson disk sampling) with sampling ratios (5%, 10%, 20%, 39%, and 78%) for NN and the stochastic mapping conditions (uniform, linear, cosine, Gaussian, and exponential distributions) for PP. Using NN with PP achieved higher performance and faster inference speed than when using NN alone. For the farthest point sampling method of 5% sampling ratio, NN+PP improved the instance mIoU by 2.457%p with 102 times faster speed compared to that when using NN alone. The result indicates that PP can significantly contribute to the improvement of performance and efficiency in PCS when used in edge AI systems.

Instance-aware segmentation of unseen objects is essential for a robotic system in an unstructured environment. Although previous works achieved encouraging results, they were limited to segmenting the only visible regions of unseen objects. For robotic manipulation in a cluttered scene, amodal perception is required to handle the occluded objects behind others. This paper addresses Unseen Object Amodal Instance Segmentation (UOAIS) to detect 1) visible masks, 2) amodal masks, and 3) occlusions on unseen object instances. For this, we propose a Hierarchical Occlusion Modeling (HOM) scheme designed to reason about the occlusion by assigning a hierarchy to a feature fusion and prediction order. We evaluated our method on three benchmarks (tabletop, indoors, and bin environments) and achieved state-of-the-art (SOTA) performance. Robot demos for picking up occluded objects, codes, and datasets are available at https://sites.google.com/view/uoais.

In this paper, we present a deep learning-based approach to detect and identify multiple fasteners from various camera poses. To distinguish fasteners of similar size and shape from each other, we propose a part identifier network and simple visual calibration method using a reference image. Though the camera poses changes, the model can infer the actual scale of detected parts by just capturing a reference object at once. Also, we present a synthetic data generation pipeline that adopts domain randomization and can automatically generate a training set for various fastener identification. In the experiment, we evaluated the real-world performance of the fully synthetically trained model and showed that it could be directly applied to real-world part identification. This indicates that our approach has the potential to accelerate the model retraining procedure for various part identification tasks since data acquisition requires almost no cost.

Segmentation of unseen industrial parts is essential for autonomous industrial systems. However, industrial components are texture-less, reflective, and often found in cluttered and unstructured environments with heavy occlusion, which makes it more challenging to deal with unseen objects. To tackle this problem, we present a synthetic data generation pipeline that randomizes textures via domain randomization to focus on the shape information. In addition, we propose an RGB-D Fusion Mask R-CNN with a confidence map estimator, which exploits reliable depth information in multiple feature levels. We transferred the trained model to real-world scenarios and evaluated its performance by making comparisons with baselines and ablation studies. We demonstrate that our methods, which use only synthetic data, could be effective solutions for unseen industrial components segmentation.

Augmented reality(AR) provides a unique viewing experience at museums where people understand abstract history through physical artifacts. Although AR usage in museum settings has been increasing, it is not well understood how AR viewing experience differs in different groups of visitors, which can be problematic considering that museums are places visited by diverse groups of people. In this study, we evaluate the differences in AR experiences according to the characteristics of the visitors. The results show the effect of AR usage in museum settings with visitors’ different age groups and motivations for visiting.

We propose a novel method for the demosaicing of event-based images that offers substantial performance improvement of far-distance gesture recognition based on deep Convolutional Neural Network. Unlike the conventional demosaicing technique using the spatial color interpolation of Bayer patterns, our new approach utilizes spatiotemporal correlation between pixel arrays, whereby timestamps of high-resolution pixels are efficiently generated in real-time from the event data. In this paper, we describe this new method and evaluate its performance with a hand motion recognition task.

Dynamic vision sensor (DVS) as a novel type of visual sensors can detect a moving object in a fast and cost effective way by outputting events on edges of the object. This paper proposes a body part localization method using structured output Deep Belief Network (s-DBN) to label the body parts in block of pixels in very fast fashion. Experiments show that our proposed algorithm achieves pixel accuracy 90.13% on body parts localization compared to Deep Belief Network (87.01%) and Random Forests (84.15%) under the same computational cost. For head/hand detection s-DBN has significant better accuracy of 99.3%/87.8% compared to DBN 98.7%/81.7% and RF 97.1%/47.1% under recall rate 99%/90%. Specifically, the process time on a 240×180 sized image is less than 1ms on Intel Core2 2.83GHZ CPU.

We propose a novel method for identifying and classifying motions that offers significantly reduced computational cost as compared to deep convolutional neural network systems with comparable performance. Our new approach is inspired by the information processing network architecture of biological visual processing systems, whereby spatial pyramid kernel features are efficiently extracted in real-time from temporally-differentiated image data. In this paper, we describe this new method and evaluate its performance with a hand motion gesture recognition task.

Fast and efficient motion estimation is essential for a number of applications including the gesture-based user interface (UI) for portable devices like smart phones. In this paper, we propose a highly efficient method that can estimate four degree of freedom (DOF) motional components of a moving object based on an event-based vision sensor, the dynamic vision sensor (DVS). The proposed method finds informative events occurred at edges and estimates their velocities for global motion analysis. We will also describe a novel method to correct the aperture problem in the motion estimation.

With an increase in the number of handicapped persons, there is a growing demand for human friendly interface as mobility aids. To meet this need, we have developed two interfaces with using shoulder and head motion for powered wheelchair control. To acquire proper wheelchair control instruction signal, workspaces of shoulder and head are analyzed by magnetic position sensor. Two interfaces are developed to meet four guidelines :human friendly design, easiness of wearability, intuitive drive function, and low cost. FSR (force sensing resistor) is used to measure changes in the shoulder and head motion. Interface's usefulness is verified by clinical experiment with six subjects who are spinal cord injured with C4 or C5.

This paper addresses a wearable master device for physically handicapped persons whose arms are disabled. Optical fiber curvature sensors are used to measure the human body motion. For the developed wearable master device, a calibration and mapping method of the sensors is proposed to extract 2-DOF human shoulder motions. An experiment shows that the wearable master device can be used for a 2-DOF input device effectively for handicapped persons. It has also shown that a subject can control a mobile robot with the wearable master device.

This paper addresses sensors and actuators for a wearable haptic master device for physically handicapped persons whose arms are disabled. In the first part of this paper, optical fiber curvature sensors are used to measure the human body motion. An efficient calibration and interpolation method of the sensors is then proposed to extract 2-DOF human shoulder motions. In the second part of this paper, solenoid vibrotactile actuators are used to feed back the contact force. Experimental results show that tactile-force substitution is helpful for force regulating tasks and that the frequency and magnitude of vibrotactile stimulus should be increased exponentially to achieve the linearly increased force sensing.