Lifelong multiagent path finding (LMAPF) requires continuous task assignment and collision-free path planning for large robot fleets. However, existing LMAPF methods assume synchronous unit-time actions, whereas real robots execute movements asynchronously with nonuniform durations. This mismatch can lead to execution-time collisions and makes frequent replanning challenging in large-scale systems. We study LMAPF with asynchronous actions and present a framework that ensures safety under asynchronous execution and real-time scalability through coordinated path planning and action scheduling, complemented by a simple greedy task allocation. To ensure safety, the path planner eliminates cycle conflicts to prevent structural deadlocks, while a lightweight action scheduler enforces vertex precedence during execution, resolving following conflicts without explicit temporal modeling. To achieve real-time performance, the planner performs partial replanning by reusing the residual paths of nonidle agents, which are converted into a synchronous form through the scheduler’s path resynchronization, enabling consistent and efficient replanning. Experiments in warehouse environments with up to 1000 robots demonstrate that our framework more than doubles throughput compared with state-of-the-art LMAPF methods, while maintaining safe operations under asynchronous actions.

Natural language is the most intuitive means for humans to interact with robots, making task planning based on natural language commands a longstanding area of research. Large language models (LLMs) have significantly improved task planning by enhancing understanding of language and common sense. However, current methods still face several challenges: they lack a deep understanding of physical environments, their performance relies heavily on prompt examples, LLMs are oversized and not customized for specific tasks, and the planning costs remain high. To overcome these issues, we introduce the GNN-Transformer Task Planner (GTTP), designed to predict task-level actions by leveraging the semantic environment and incorporating historical state data. The GTTP architecture is scalable through the use of GNN layers, while transformer layers facilitate understanding task progression. In addition, our model uses a text encoder to embed environments, allowing it to be trained on simulated datasets and applied directly in real-world scenarios. We also propose an automated data generation method that includes semantic augmentation, planning verification, and instruction generation via LLM. This method enables the collection of 14k instruction-annotated tasks in the VirtualHome environment with minimal human effort. The model has been validated across diverse scenes containing up to 715 objects, achieving significantly higher success rates compared to baseline models. It has also been successfully deployed on a physical mobile manipulator, demonstrating its practical applicability and effectiveness.

Deep neural network (DNN)-based policy models, such as vision-language-action (VLA) models, excel at automating complex decision-making from multi-modal inputs. However, scaling these models greatly increases computational overhead, complicating deployment in resource-constrained settings like robot manipulation and autonomous driving. To address this, we propose Saliency-Aware Quantized Imitation Learning (SQIL), which combines quantization-aware training with a selective loss-weighting strategy for mission-critical states. By identifying these states via saliency scores and emphasizing them in the training loss, SQIL preserves decision fidelity under low-bit precision. We validate SQIL's generalization capability across extensive simulation benchmarks with environment variations, real-world tasks, and cross-domain tasks (self-driving, physics simulation), consistently recovering full-precision performance. Notably, a 4-bit weight-quantized VLA model for robotic manipulation achieves up to 2.5x speedup and 2.5x energy savings on an edge GPU with minimal accuracy loss. These results underline SQIL's potential for efficiently deploying large IL-based policy models on resource-limited devices.

With recent advancements in Large Language Models, task planning methods that interpret human commands have garnered significant attention. However, as home robots become more common, specifying every daily task could become impractical. This paper introduces a novel semantic map called the Task-Aware Semantic Map (TASMap), which enables robots to autonomously assign and propose necessary tasks in a scene without explicit human commands. The core innovation of this approach is the ability of TASMap to comprehend the context of objects within a scene and autonomously generate task proposals. This capability significantly advances autonomous robotic assistance, reducing the dependency on specific commands and enhancing interaction with environments. We present two key applications of TASMap: contextual task proposal and spatial task proposal. Our results, verified across 35 diverse and realistically disordered scenes, underscore the effectiveness of TASMap in both simulation and real-world environments.

In the field of robot manipulation, learnable task planners are gaining attention, especially for long-horizon tasks such as cooking. However, existing methods that predominantly rely on symbolic representations suffer from limitations in generalization capabilities, particularly in handling unseen objects. Given that objects may vary in real-world environments, this limitation may constrain their practical applicability. To address this issue, we propose a novel task-planning framework that leverages a pretrained large language model (LLM) for environmental interpretation. Our proposed framework extracts semantic features directly from textual data, enabling the planner to accommodate unfamiliar objects. We further incorporate a transformer-based encoder-decoder framework to understand environmental attributes derived from the language model and generate sequential predictions in line with object-oriented subgoals. To validate the effectiveness of our model, we utilize a dataset focused on cooking recipes. Going a step further, we propose a method that automatically generates object-oriented data from natural language description using recurrent LLM, enhancing the framework to manage previously unseen targets as well. Our framework shows an average success rate of 95% when validated with test sets that involve unseen objects. By providing the automatically generated dataset to the framework, we achieve a significant 27% increase in success rate on unknown target recipes. We also provide evidence of the real-world viability of our planner by successfully deploying it on a robot platform.

The main challenge in vision-and-language navigation (VLN) is how to understand natural-language instructions in an unseen environment. The main limitation of conventional VLN algorithms is that if an action is mistaken, the agent fails to follow the instructions or explores unnecessary regions, leading the agent to an irrecoverable path. To tackle this problem, we propose Meta-Explore, a hierarchical navigation method deploying an exploitation policy to correct misled recent actions. We show that an exploitation policy, which moves the agent toward a well-chosen local goal among unvisited but observable states, outperforms a method which moves the agent to a previously visited state. We also highlight the demand for imagining regretful explorations with semantically meaningful clues. The key to our approach is understanding the object placements around the agent in spectral-domain. Specifically, we present a novel visual representation, called scene object spectrum (SOS), which performs category-wise 2D Fourier transform of detected objects. Combining exploitation policy and SOS features, the agent can correct its path by choosing a promising local goal. We evaluate our method in three VLN benchmarks: R2R, SOON, and REVERIE. Meta-Explore outper-forms other baselines and shows significant generalization performance. In addition, local goal search using the proposed spectral-domain SOS features significantly improves the success rate by 17.1% and SPL by 20. 6% against the state-of-the-art method of the SOON benchmark. Project page: https://rllab-snu.github.io/projects/Meta-Explore/doc.html

A core capability of robots is to reason about multiple objects under uncertainty. Partially Observable Markov Decision Processes (POMDPs) provide a means of reasoning under uncertainty for sequential decision making, but are computationally intractable in large domains. In this paper, we propose Object-Oriented POMDPs (OO-POMDPs), which represent the state and observation spaces in terms of classes and objects. The structure afforded by OO-POMDPs support a factorization of the agent's belief into independent object distributions, which enables the size of the belief to scale linearly versus exponentially in the number of objects. We formulate a novel Multi-Object Search (MOS) task as an OO-POMDP for mobile robotics domains in which the agent must find the locations of multiple objects. Our solution exploits the structure of OO-POMDPs by featuring human language to selectively update the belief at task onset. Using this structure, we develop a new algorithm for efficiently solving OO-POMDPs: Object-Oriented Partially Observable Monte-Carlo Planning (OOPOMCP). We show that OO-POMCP with grounded language commands is sufficient for solving challenging MOS tasks both in simulation and on a physical mobile robot.

We propose a robust target tracking algorithm for a mobile robot under identity uncertainty, which arises in crowded environments. When a mobile robot has a sensor with a fan-shaped field of view and finite sensing region, the proposed algorithm aims to minimize the probability of losing a moving target. We predict the next position of a moving target in a crowded environment using a multiple-hypothesis prediction algorithm which combines the motion model and appearance model of the target. When the distribution of the target's next position follows a Gaussian mixture model, the proposed tracking algorithm can track a target with a guaranteed tracking success probability. If the tracking success probability is sufficiently good, the method minimizes the moving distance of the mobile robot. The performance of the method is extensively validated in simulation and experiments using a Pioneer robot with a Microsoft Kinect sensor.

This paper presents a robust target tracking algorithm for a mobile sensor with a fan-shaped field of view and finite sensing range. The goal of the mobile robot is to track a moving target such that the probability of losing the target is minimized. We assume that the distribution of the next position of a moving target can be estimated using a motion prediction algorithm. If the next position of a moving target has the Gaussian distribution, the proposed algorithm can guarantee the tracking success probability. In addition, the proposed method minimizes the moving distance of the mobile robot based on a bound on the tracking success probability. While the problem considered in this paper is a non-convex optimization problem, we derive analytical solutions which can be easily solved in real-time. The performance of the proposed method is evaluated extensively in simulation and validated in pedestrian following experiments using a Pioneer mobile robot with a Microsoft Kinect sensor.