Segmenting unseen objects from images is a critical perception skill that a robot needs to acquire. In robot manipulation, it can facilitate a robot to grasp and manipulate unseen objects. Mean shift clustering is a widely used method for image segmentation tasks. However, the traditional mean shift clustering algorithm is not differentiable, making it difficult to integrate it into an end-to-end neural network training framework. In this work, we propose the Mean Shift Mask Transformer (MSMFormer), a new transformer architecture that simulates the von Mises-Fisher (vMF) mean shift clustering algorithm, allowing for the joint training and inference of both the feature extractor and the clustering. Its central component is a hypersphere attention mechanism, which updates object queries on a hypersphere. To illustrate the effectiveness of our method, we apply MSMFormer to unseen object instance segmentation. Our experiments show that MSMFormer achieves competitive performance compared to state-of-the-art methods for unseen object instance segmentation1.

Endoscopic Submucosal Dissection (ESD) is a minimally invasive procedure designed to remove precancerous and cancerous lesions from the gastrointestinal (GI) tract. Given the GI tract’s tortuous and narrow shape, along with the need for varied movements during dissection, this requires highly flexible and compact instruments, making flexible continuum robots suitable candidates. In this paper, we propose a novel two-segment continuum robot system named DESectBot, featuring a diameter of 5.5 mm and a total length of the active bending module of 48 mm, while the robot’s total length exceeds 1 m. We designed a novel joint combination structure called the spatial cross-curved disk skeleton for the robot, which addresses the mechanical coupling problem between flexible robot actuators. The DESectBot boasts six degrees of freedom, and its kinematic modeling has been derived and utilized in the closed-loop control of the DESectBot. The validation of the DESectBot was conducted through a two-stage test: first, the decoupling performance of the DESectBot was validated. The results show that when one active bending segment bends, the other segment remains almost uninfluenced, with a maximum variation of 1.15 degrees, demonstrating the robot’s effective decoupling capability. Secondly, the accuracy of DESectBot was validated through trajectory-following experiments. The results reveal that the average tracking error for both trajectories is less than 2 mm, and the maximum tracking error is below 2.5 mm. Taking marking, one of the ESD procedures with a 5mm tolerance, as an example, the DESectBot has the potential to be utilized for ESD procedure.

Statistical Relational Learning (SRL) models have attracted significant attention due to their ability to model complex data while handling uncertainty. However, most of these models have been restricted to discrete domains owing to the complexity of inference in continuous domains. In this work, we introduce Relational Neural Markov Random Fields (RN-MRFs) that allow handling of complex relational hybrid domains, i.e., those that include discrete and continuous quantities, and we propose a maximum pseudolikelihood estimation-based learning algorithm with importance sampling for training the neural potential parameters. The key advantage of our approach is that it makes minimal data distributional assumptions and can seamlessly embed human knowledge through potentials or relational rules. Our empirical evaluations across diverse domains, such as image processing and relational object mapping, demonstrate its practical utility.

Non-stationary data stream mining aims to classify large scale online instances that emerge continuously. The most apparent challenge compared with the offline learning manner is the issue of consecutive emergence of new categories, when tackling non-static categorical distribution. Non-stationary stream settings often appear in real-world applications, e.g., online classification in E-commerce systems that involves the incoming productions, or the summary of news topics on social networks (Twitter). Ideally, a learning model should be able to learn novel concepts from labeled data (in new tasks) and reduce the abrupt degradation of model performance on the old concept (also named catastrophic forgetting problem). In this work, we focus on improving the performance of the stream mining approach under the constrained resources, where both the memory resource of old data and labeled new instances are limited/scarce. We propose a simple yet efficient resource-constrained framework CLEAR to facilitate previous challenges during the one-pass stream mining. Specifically, CLEAR focuses on creating and calibrating the class representation (the prototype) in the embedding space. We first apply the contrastive-prototype learning on large amount of unlabeled data, and generate the discriminative prototype for each class in the embedding space. Next, for updating on new tasks/categories, we propose a drift estimation strategy to calibrate/compensate for the drift of each class representation, which could reduce the knowledge forgetting without storing any previous data. We perform experiments on public datasets (e.g., CUB200, CIFAR100) under stream setting, our approach is consistently and clearly better than many state-of-the-art methods, along with both the memory and annotation restriction.

Lifted inference algorithms exploit model symmetry to reduce computational cost in probabilistic inference. However, most existing lifted inference algorithms operate only over discrete domains or continuous domains with restricted potential functions. We investigate two approximate lifted variational approaches that apply to domains with general hybrid potentials, and are expressive enough to capture multi-modality. We demonstrate that the proposed variational methods are highly scalable and can exploit approximate model symmetries even in the presence of a large amount of continuous evidence, outperforming existing message-passing-based approaches in a variety of settings. Additionally, we present a sufficient condition for the Bethe variational approximation to yield a non-trivial estimate over the marginal polytope.

Lifted inference algorithms for first-order logic models, e.g., Markov logic networks (MLNs), have been of significant interest in recent years. Lifted inference methods exploit model symmetries in order to reduce the size of the model and, consequently, the computational cost of inference. In this work, we consider the problem of lifted inference in MLNs with continuous or both discrete and continuous groundings. Existing work on lifting with continuous groundings has mostly been limited to special classes of models, e.g., Gaussian models, for which variable elimination or message-passing updates can be computed exactly. Here, we develop approximate lifted inference schemes based on particle sampling. We demonstrate empirically that our approximate lifting schemes perform comparably to existing state-of-the-art for models for Gaussian MLNs, while having the flexibility to be applied to models with arbitrary potential functions.