Grasp detection requires flexibility to handle objects of various shapes without relying on prior object knowledge, while also offering intuitive, user-guided control. In this paper, we introduce GraspSAM, an innovative extension of the Segment Anything Model (SAM) designed for prompt-driven and category-agnostic grasp detection. Unlike previous methods, which are often limited by small-scale training data, Grasp-SAM leverages SAM's large-scale training and prompt-based segmentation capabilities to efficiently support both target-object and category-agnostic grasping. By utilizing adapters, learnable token embeddings, and a lightweight modified decoder, GraspSAM requires minimal fine-tuning to integrate object segmentation and grasp prediction into a unified frame-work. Our model achieves state-of-the-art (SOTA) performance across multiple datasets, including Jacquard, Grasp-Anything, and Grasp-Anything++. Extensive experiments demonstrate GraspSAM's flexibility in handling different types of prompts (such as points, boxes, and language), highlighting its robustness and effectiveness in real-world robotic applications. Robot demonstrations, additional results, and code can be found at https://gistailab.github.io/GraspSAM/.

Test-time adaptation (TTA) aims to adapt a pre-trained model to a new test domain without access to source data after deployment. Existing approaches typically rely on self-training with pseudo-labels since ground-truth cannot be obtained from test data. Although the quality of pseudo labels is important for stable and accurate long-term adaptation, it has not been previously addressed. In this work, we propose DPLOT, a simple yet effective TTA framework that consists of two components: (1) domain-specific block selection and (2) pseudo-label generation using paired-view images. Specifically, we select blocks that involve domain-specific feature extraction and train these blocks by entropy minimization. After blocks are adjusted for current test domain, we generate pseudo-labels by averaging given test images and corresponding flipped counterparts. By simply using flip augmentation, we prevent a decrease in the quality of the pseudo-labels, which can be caused by the domain gap resulting from strong augmentation. Our experimental results demonstrate that DPLOT outperforms previous TTA methods in CIFAR10-C, CIFAR100-C, and ImageNet-C benchmarks, reducing error by up to 5.4%, 9.1%, and 2.9%, respectively. Also, we provide an extensive analysis to demonstrate effectiveness of our framework. Code is available at https://github.com/gist-ailab/domain-specific-block-selection-and-paired-view-pseudo-labeling-for-online-TTA.

The study addresses the foundational and challenging task of peg-in-hole assembly in robotics, where misalignments caused by sensor inaccuracies and mechanical errors often result in insertion failures or jamming. This research introduces PolyFit, representing a paradigm shift by transitioning from a reinforcement learning approach to a supervised learning methodology. PolyFit is a Force/Torque (F/T)-based supervised learning framework designed for 5-DoF peg-in-hole assembly. It utilizes F/T data for accurate extrinsic pose estimation and adjusts the peg pose to rectify misalignments. Extensive training in a simulated environment involves a dataset encompassing a diverse range of peg-hole shapes, extrinsic poses, and their corresponding contact F/T readings. The study proposes a sim-to-real adaptation method for real-world application, using a sim-real paired dataset to enable effective generalization to complex and unseen polygon shapes. Real-world evaluations demonstrate substantial success rates of 96.7% and 91.3%, highlighting the robustness and adaptability of the proposed method. Videos of data generation and experiments are available online at https://sites.google.com/view/polyfit-peginhole.

Neural networks (NN) have shown promising performance in point cloud segmentation (PCS). However, the measured points are too numerous to be used as model input at once. It results in a long inference time and high computational cost due to iterative sampling and inference. This study proposes Probability Propagation (PP) as a stochastic upsampling method. PP propagates the predicted probability of a sampled part of a point cloud into the other unpredicted points by considering proximity. By replacing the iterative inference of NN with PP, large point clouds can be dealt with quickly and efficiently. We investigated the effectiveness of PP using the ShapeNet benchmark on various settings: sampling methods (random, farthest point, and Poisson disk sampling) with sampling ratios (5%, 10%, 20%, 39%, and 78%) for NN and the stochastic mapping conditions (uniform, linear, cosine, Gaussian, and exponential distributions) for PP. Using NN with PP achieved higher performance and faster inference speed than when using NN alone. For the farthest point sampling method of 5% sampling ratio, NN+PP improved the instance mIoU by 2.457%p with 102 times faster speed compared to that when using NN alone. The result indicates that PP can significantly contribute to the improvement of performance and efficiency in PCS when used in edge AI systems.

Instance-aware segmentation of unseen objects is essential for a robotic system in an unstructured environment. Although previous works achieved encouraging results, they were limited to segmenting the only visible regions of unseen objects. For robotic manipulation in a cluttered scene, amodal perception is required to handle the occluded objects behind others. This paper addresses Unseen Object Amodal Instance Segmentation (UOAIS) to detect 1) visible masks, 2) amodal masks, and 3) occlusions on unseen object instances. For this, we propose a Hierarchical Occlusion Modeling (HOM) scheme designed to reason about the occlusion by assigning a hierarchy to a feature fusion and prediction order. We evaluated our method on three benchmarks (tabletop, indoors, and bin environments) and achieved state-of-the-art (SOTA) performance. Robot demos for picking up occluded objects, codes, and datasets are available at https://sites.google.com/view/uoais.

In this paper, we present a deep learning-based approach to detect and identify multiple fasteners from various camera poses. To distinguish fasteners of similar size and shape from each other, we propose a part identifier network and simple visual calibration method using a reference image. Though the camera poses changes, the model can infer the actual scale of detected parts by just capturing a reference object at once. Also, we present a synthetic data generation pipeline that adopts domain randomization and can automatically generate a training set for various fastener identification. In the experiment, we evaluated the real-world performance of the fully synthetically trained model and showed that it could be directly applied to real-world part identification. This indicates that our approach has the potential to accelerate the model retraining procedure for various part identification tasks since data acquisition requires almost no cost.