Recent advances in large language models (LLMs) have demonstrated exceptional reasoning capabilities in natural language processing, sparking interest in applying LLMs to task planning problems in robotics. Most studies focused on task planning for clear natural language commands that specify target objects and their locations. However, for more user-friendly task execution, it is crucial for robots to autonomously plan and carry out tasks based on abstract natural language commands that may not explicitly mention target objects or locations, such as ‘Put the food ingredients in the same place.’ In this study, we propose an LLM-based autonomous task planning framework that generates task plans for abstract natural language commands. This framework consists of two phases: an environment recognition phase and a task planning phase. In the environment recognition phase, a large vision-language model generates a hierarchical scene graph that captures the relationships between objects and spaces in the environment surrounding a robot agent. During the task planning phase, an LLM uses the scene graph and the abstract user command to formulate a plan for the given task. We validate the effectiveness of the proposed framework in the AI2THOR simulation environment, demonstrating its superior performance in task execution when handling abstract commands.

We propose a novel motion-based tracker specifically designed for tracking multiple people in low frame rate scenarios. While previous studies have predominantly focused on scenarios with high frame rates (exceeding 10 frames per second), tracking in low frame rate conditions is significant for robotic platforms with limited computational resources. Our tracker optimizes the cost function, cascade structure and Kalman filter correction to better adapt to the characteristics of low frame rate environments. First, we enhance the cost function by incorporating stable variables through the introduction of height-based and displacement-based cost terms. Second, we prioritize handling occlusion among individuals during association, which reduces ambiguity in subsequent tracking processes. Third, we utilize the error-compensated detection to correct the Kalman filter, thereby improving tracking accuracy. Experimental results demonstrate that our proposed tracker, LoFSORT, outperforms other motion model-based trackers across various frame rate scenarios. Ablation studies further confirm that each component of our tracker enhances tracking performance in low frame rate scenarios.

Percutaneous coronary intervention (PCI) involves the insertion of a catheter or guidewire into a blood vessel of a patient, which poses a problem as a doctor is exposed to radiation during the procedure. The use of assistive robots has been proposed to address this issue. Furthermore, recent research is progressing toward complete autonomous navigation using deep reinforcement learning (DRL). Nevertheless, existing algorithms face limitations when operating in numerous unseen environments close to real PCI. This study proposes a robust DRL framework for image-based guidewire navigation to overcome the limitation. We introduce a subtasks strategy and domain randomization to improve robustness in various environments. The subtasks strategy consistently addresses complex global tasks by breaking them into subtasks designed using local maps, allowing them to be robustly solved by a single agent. Domain randomization is applied to handle real PCI issues, including variations in vessel geometry, guidewire deformation, and camera settings. By integrating the two novel methods, our DRL algorithm demonstrates superior performance compared to existing methods across various challenging simulation and phantom environments, validating its effectiveness in real-world scenarios. A video of our experiment is available at https://youtu.be/93Q88gESzOY.

This paper presents a novel method for state estimation of rigid body attitude system evolving on the manifold S3, which is crucial in robotics and drone applications. We introduce a particle filter with stable embedding that extends the system into Euclidean space while ensuring stability of the manifold. Our particle filter with stable embedding enables accurate state estimation by maintaining estimated state values in close proximity to the manifold, while requiring significantly fewer computational resources than the standard exponential-map-based method that keeps state estimates on the manifold. Furthermore, our method facilitates the application of usual techniques designed for particle filters in Euclidean spaces, to the manifold system, as is, without any modification. The accuracy and the efficiency of our particle filter are confirmed both by simulation and by real drone experiments.

Multi-object tracking (MOT) presents a crucial challenge in robotics. Due to limited resources embedded in robots, one time step per processing time for algorithms can be considerably large. This scenario necessitates the operation of MOT at a low frame rate. However, algorithms within the MOT research field have been constructed around datasets functioning at 10–30 frames per second (fps) which can be difficult to operate in the limited resources. In response to it, we introduce a new algorithm, called FocoTrack, which maintains tracking ability in four situations, one of which is when objects are overlapped by each other. Our algorithm exhibits remarkable performance without using any deep appearance descriptor, surpassing existing MOT methods which even use the deep appearance descriptor on a 2.5 fps dataset. We also demonstrate strong results with our algorithm on DanceTrack dataset at 20 fps and provide comprehensive insights through detailed analysis of our tracking model.

The optimization with orthogonality has been shown useful in training deep neural networks (DNNs). To impose orthogonality on DNNs, both computational efficiency and stability are important. However, existing methods utilizing Riemannian optimization or hard constraints can only ensure stability while those using soft constraints can only improve efficiency. In this paper, we propose a novel method, named Feedback Gradient Descent (FGD), to our knowledge, the first work showing high efficiency and stability simultaneously. FGD induces orthogonality based on the simple yet indispensable Euler discretization of a continuous-time dynamical system on the tangent bundle of the Stiefel manifold. In particular, inspired by a numerical integration method on manifolds called Feedback Integrators, we propose to instantiate it on the tangent bundle of the Stiefel manifold for the first time. In our extensive image classification experiments, FGD comprehensively outperforms the existing state-of-the-art methods in terms of accuracy, efficiency, and stability.

Percutaneous coronary intervention (PCI) is a frequently used surgical treatment for cardiovascular disease, one of the leading cause of death in the world. In traditional PCI, a doctor navigates a thin guidewire in a patient's vessel toward a target location by looking into live X-ray angiogram images of the patient. Recently, researchers are using reinforcement learning to automate this guidewire navigation process without attaching any sensor to the guidewire tip. These researchers use a real vessel phantom to train their behavior policy using reinforcement learning. Training a reinforcement learning algorithm on a real setup can give a good guidewire control on that setup, but it is under question whether the trained algorithm can be applied to other vessel structures. We can make various vessel phantoms and train the algorithm on the setups, but it can be really time and money consuming. In this paper, we devise a method for sim-to-real transfer of a guidewire navigation trained by reinforcement learning using only images. We pretrain our behavior policy using data collected by running an expert algorithm in the virtual environment. Then, we train the behavior policy by deep deterministic policy gradient (DDPG) in a virtual environment. With behavior cloning, our method learns to successfully navigate a guidewire in much shorter time than training DDPG from scratch without behavior cloning. After done with the training, we transfer the behavior policy trained in the virtual environment to the guidewire navigation in a real vessel phantom. Our trained behavior policy navigates the guidewire to destinations successfully in all test episodes and navigates faster than the expert algorithm. Experiment video is available at: https: //youtu.be/HCEbIhZsXqw

With the growing application of various robots in real life, the need for an automatic anomaly detection system for robots is necessary for safety. In this paper, we develop an anomaly detection method using a stacked LSTM that can be applied to any robot controlled by a feedback control. Our method does not need installation of additional sensors. Our method is model-free and unsupervised because it does not require the analytical model of the system and the training data does not require faulty operation conditions. We validate our method on real fixed-wing unmanned aerial vehicle flight data containing control surface failure scenarios. We demonstrate the superiority of the proposed algorithm over existing anomaly detection methods in the literature. Our code is available at https://github.com/superhumangod/Model-free-unsupervised-anomaly-detection.

This paper presents a vision-based modularized drone racing navigation system that uses a customized convolutional neural network (CNN) for the perception module to produce high-level navigation commands and then leverages a state-of-the-art planner and controller to generate low-level control commands, thus exploiting the advantages of both data- based and model-based approaches. Unlike the state-of-the-art method, which only takes the current camera image as the CNN input, we further add the latest three estimated drone states as part of the inputs. Our method outperforms the state-of-the-art method in various track layouts and offers two switchable navigation behaviors with a single trained network. The CNN-based perception module is trained to imitate an expert policy that automatically generates ground truth navigation commands based on the pre-computed global trajectories. Owing to the extensive randomization and our modified dataset aggregation (DAgger) policy during data collection, our navigation system, which is purely trained in simulation with synthetic textures, successfully operates in environments with randomly-chosen photo-realistic textures without further fine-tuning.

Forecasting the motion of surrounding obstacles (vehicles, bicycles, pedestrians and etc.) benefits the on-road motion planning for intelligent and autonomous vehicles. Complex scenes always yield great challenges in modeling the patterns of surrounding traffic. For example, one main challenge comes from the intractable interaction effects in a complex traffic system. In this paper, we propose a multi-layer architecture Interaction-aware Kalman Neural Networks (IaKNN) which involves an interaction layer for resolving high-dimensional traffic environmental observations as interaction-aware accelerations, a motion layer for transforming the accelerations to interaction-aware trajectories, and a filter layer for estimating future trajectories with a Kalman filter network. Attributed to the multiple traffic data sources, our end-to-end trainable approach technically fuses dynamic and interaction-aware trajectories boosting the prediction performance. Experiments on the NGSIM dataset demonstrate that IaKNN outperforms the state-of-the-art methods in terms of effectiveness for traffic trajectory prediction.

Lie group symmetry in a mechanical system can lead to a dimensional reduction in its dynamical equations. Typically, the symmetries that one exploits are intrinsic to the mechanical system at hand, e.g. invariance of the system's Lagrangian to some group of motions. In the present work we consider symmetries that arise from an extrinsic control task, rather than the intrinsic structure of configuration space, constraints, or system dynamics. We illustrate this technique with several examples. In the examples, the reduction enables us to design essentially global feedback controllers on the reduced systems.We apply task-induced symmetry and reduction to a recently developed 6 DOF kinematic model of steerable bevel-tip needles. The resulting controllers cause the needle tip to track a subspace of its configuration space. We envision that the methodology presented in this paper will form the basis for a new planning and control framework for needle steering.

This paper presents a global diffeomorphism from a visible set of rigid-body configurations, a subset of SE(3), to an image space. Using the diffeomorphism, we develop an image-based, essentially global, dynamic visual servoing algorithm that keeps features in the field of view and avoids self-occlusions. The approach is geometric in the sense that the visible set and its corresponding image are differentiable manifolds, and the diffeomorphism is global. The mapping to image space and the resulting Jacobian rely on a specific target geometry, a sphere with a known radius marked with an "arrow" feature point. The paper presents simulation experiments for a more typical visual target comprised of a collection of isolated feature points. In this setting, the diffeomorphism to image space is approximate, nevertheless, the simulations converge for a wide variety of target geometries and initial conditions.