In real-world field operations, aerial grasping systems face significant challenges in dynamic environments due to strong winds, shifting surfaces, and the need to handle heavy loads. Particularly when dealing with heavy objects, the powerful propellers of the drone can inadvertently blow the target object away as it approaches, making the task even more difficult. To address these challenges, we introduce SPI- BOT, a novel drone-tethered mobile gripper system designed for robust and stable autonomous target retrieval. SPIBOT operates via a tether, much like a spider, allowing the drone to maintain a safe distance from the target. To ensure both stable mobility and secure grasping capabilities, SPIBOT is equipped with six legs and sensors to estimate the robot's and mission's states. It is designed with a reduced volume and weight compared to other hexapod robots, allowing it to be easily stowed under the drone and reeled in as needed. Designed for the 2024 MBZIRC Maritime Grand Challenge, SPIBOT is built to retrieve a 1kg target object in the highly dynamic conditions of the moving deck of a ship. This system integrates a real-time action selection algorithm that dynamically adjusts the robot's actions based on proximity to the mission goal and environmental conditions, enabling rapid and robust mission execution. Experimental results across various terrains, including a pontoon on a lake, a grass field, and rubber mats on coastal sand, demonstrate SPIBOT's ability to efficiently and reliably retrieve targets. SPIBOT swiftly converges on the target and completes its mission, even when dealing with irregular initial states and noisy information introduced by the drone.

This paper proposes novel methods to enhance the performance of monocular 3D object detection models by lever-aging the generalized feature extraction capabilities of a vision foundation model. Unlike traditional CNN-based approaches, which often suffer from inaccurate depth estimation and rely on multi-stage object detection pipelines, this study employs a Vision Transformer (ViT)-based foundation model as the backbone, which excels at capturing global features for depth estimation. It integrates a detection transformer (DETR) archi-tecture to improve both depth estimation and object detection performance in a one-stage manner. Specifically, a hierarchical feature fusion block is introduced to extract richer visual features from the foundation model, further enhancing feature extraction capabilities. Depth estimation accuracy is further improved by incorporating a relative depth estimation model trained on large-scale data and fine-tuning it through transfer learning. Additionally, the use of queries in the transformer's decoder, which consider reference points and the dimensions of 2D bounding boxes, enhances recognition performance. The proposed model outperforms recent state-of-the-art methods, as demonstrated through quantitative and qualitative evaluations on the KITTI 3D benchmark and a custom dataset collected from high-elevation racing environments. Code is available at https://github.com/JihyeokKim/MonoDINO-DETR.

This paper presents the design and validation of our autonomous driving system developed for one-on-one high-speed racing at the Indy Autonomous Challenge (IAC) 2024. In this event, each team is required to pass the opponent car on an oval racetrack at very high speeds reaching more than 250 km/h. In order to meet the critical challenges of high-speed autonomous racing, we developed reliable perception, behavior planning for passing, and control algorithms stable in the high speed range. For perception, we constructed a multi-modal pipeline using cameras, LiDARs, and radars to achieve accurate and reliable real-time object detection. We also designed a new state machine-based algorithm for effective planning of overtaking maneuvers. As for the control, we improved the vehicle controller at higher speed ranges by building a more accurate engine torque map and fine-tuning the controller during the actual runs. Our newly designed racing system was proven effective, enabling our car to cruise stably at 250.1 km/h in solo lap and executing multiple successful overtaking maneuvers during the one-on-one races at the Las Vegas Motor Speedway (LVMS) for the Consumer Electronics Show (CES) 2024.

This paper introduces an innovative application of foundation models, enabling Unmanned Ground Vehicles (UGVs) equipped with an RGB-D camera to navigate to designated destinations based on human language instructions. Unlike learning-based methods, this approach does not require prior training but instead leverages existing foundation models, thus facilitating generalization to novel environments. Upon receiving human language instructions, these are transformed into a ‘cognitive route description’ using a large language model (LLM)-a detailed navigation route expressed in human language. The vehicle then decomposes this description into landmarks and navigation maneuvers. The vehicle also determines elevation costs and identifies navigability levels of different regions through a terrain segmentation model, GANav, trained on open datasets. Semantic elevation costs, which take both elevation and navigability levels into account, are estimated and provided to the Model Predictive Path Integral (MPPI) planner, responsible for local path planning. Concurrently, the vehicle searches for target landmarks using foundation models, including YOLO-World and EfficientViT-SAM. Ultimately, the vehicle executes the navigation commands to reach the designated destination, the final landmark. Our experiments demonstrate that this application successfully guides UGVs to their destinations following human language instructions in novel environments, such as unfamiliar terrain or urban settings.

We introduce MoNet, a novel functionally modular network for self-supervised and interpretable end-to-end learning. By leveraging its functional modularity with a latent-guided contrastive loss function, MoNet efficiently learns task-specific decision-making processes in latent space without requiring task-level supervision. Moreover, our method incorporates an online, post-hoc explainability approach that enhances the interpretability of end-to-end inferences without compromising sensorimotor control performance. In real-world indoor environments, MoNet demonstrates effective visual autonomous navigation, outperforming baseline models by 7% to 28% in task specificity analysis. We further explore the interpretability of our network through post-hoc analysis of perceptual saliency maps and latent decision vectors. This provides valuable insights into the incorporation of explainable artificial intelligence into robotic learning, encompassing both perceptual and behavioral perspectives. Supplementary materials are available at https://sites.google.com/view/monet-lgc.

Existing exploration algorithms mainly generate frontiers using random sampling or motion primitive methods within a specific sensor range or search space. However, frontiers generated within constrained spaces lead to back-and-forth maneuvers in large-scale environments, thereby diminishing exploration efficiency. To address this issue, we propose a method that utilizes a 3D dense map to generate Segmented Exploration Regions (SERs) and generate frontiers from a global-scale perspective. In particular, this paper presents a novel topological map generation approach that fully utilizes Line-of-Sight (LOS) features of LiDAR sensor points to enhance exploration efficiency inside large-scale subterranean environments. Our topological map contains the contributions of keyframes that generate each SER, enabling rapid exploration through a switch between local path planning and global path planning to each frontier. The proposed method achieved higher explored volume generation than the state-of-the-art algorithm in a large-scale simulation environment and demonstrated a 62% improvement in explored volume increment performance. For validation, we conducted field tests using UAVs in real subterranean environments, demonstrating the efficiency and speed of our method.

A pilot needs to manipulate various gadgets in the cockpit based on vast knowledge of rules and procedures while verbally communicating with air traffic controllers. While precision manipulation in the cockpit during the flight is already a difficult task, a far more difficult thing is how to make a robot learn all the knowledge needed to fly an airplane in accordance with all the rules and regulations. As a pioneering effort, this paper introduces LLM-PIBOT, which leverages the latest advances in Large Language Models (LLMs) to empower a humanoid pilot robot (PIBOT) to take the full authority of an airplane by understanding and executing complex procedures outlined in Pilot’s Operating Handbooks (POHs). Unlike traditional rule-based methods, LLM-PIBOT system infers suitable flight procedures, employs an embedding process to accurately identify relevant procedures within documents, and structures the text-extracted flight tasks into tuples using our carefully crafted prompts. This approach enables PIBOT to adapt to the given POHs, generating and executing task plans in real-time in response to commands and situations. Experimental results show that LLM-PIBOT can comprehend and follow the complex procedures specified in the manuals and to fly the airplane on a full-scale simulator using the generated flight plans.

This paper focuses on the acquisition of mapless navigation skills within unknown environments. We introduce the Skill Q-Network (SQN), a novel reinforcement learning method featuring an adaptive skill ensemble mechanism. Unlike existing methods, our model concurrently learns a high-level skill decision process alongside multiple low-level navigation skills, all without the need for prior knowledge. Leveraging a tailored reward function for mapless navigation, the SQN is capable of learning adaptive maneuvers that incorporate both exploration and goal-directed skills, enabling effective navigation in new environments. Our experiments demonstrate that our SQN can effectively navigate complex environments, exhibiting a 40% higher performance compared to baseline models. Without explicit guidance, SQN discovers how to combine low-level skill policies, showcasing both goal-directed navigations to reach destinations and exploration maneuvers to escape from local minimum regions in challenging scenarios. Remarkably, our adaptive skill ensemble method enables zero-shot transfer to out-of-distribution domains, characterized by unseen observations from non-convex obstacles or uneven, subterranean-like environments. The project page is available at https://sites.google.com/view/skill-q-net.

In this paper, we present an innovative trajectory prediction algorithm that is specifically crafted for high-speed autonomous racing, with a focus on the mutual influence between vehicles. This algorithm was developed in the context of the Hyundai Autonomous Challenge 2023, a pioneering event that featured the world’s first competitive racing scenario involving three autonomous vehicles simultaneously navigating a road course race track. Stable overtaking in 1:N races requires accurate prediction of the trajectories of surrounding vehicles, taking into account their inter-vehicle dynamics. To meet this challenge, our approach leverages the Model Predictive Path Integral technique, which not only considers information from neighboring vehicles but also incorporates prior knowledge of the race track. Furthermore, we have augmented our algorithm with maneuver intention estimation-based trajectory prediction, an approach that leverages a vehicle’s historical trajectory data to forecast its future path. By integrating these two methodologies, our algorithm adeptly anticipates the motion of other vehicles under a variety of conditions on the race track. The efficacy of our proposed solution has been substantiated through extensive simulation and real-world testing, demonstrating its capability to deliver real-time performance in high-speed environments, with a processing time as low as 20 milliseconds.

Recently, numerous studies have investigated cooperative traffic systems using the communication among vehicle-to-everything (V2X). Unfortunately, when multiple autonomous vehicles are deployed while exposed to communication failure, there might be a conflict of ideal conditions between various autonomous vehicles leading to adversarial situation on the roads. In South Korea, virtual and real-world urban autonomous multi-vehicle races were held in March and November of 2021, respectively. During the competition, multiple vehicles were involved simultaneously, which required maneuvers such as overtaking low-speed vehicles, negotiating intersections, and obeying traffic laws. In this study, we introduce a fully autonomous driving software stack to deploy a competitive driving model, which enabled us to win the urban autonomous multi-vehicle races. We evaluate module-based systems such as navigation, perception, and planning in real and virtual environments. Additionally, an analysis of traffic is performed after collecting multiple vehicle position data over communication to gain additional insight into a multi-agent autonomous driving scenario. Finally, we propose a method for analyzing traffic in order to compare the spatial distribution of multiple autonomous vehicles. We study the similarity distribution between each team’s driving log data to determine the impact of competitive autonomous driving on the traffic environment. Our fully autonomous software architecture, proven successful in winning urban autonomous multi-vehicle races in South Korea, is ready for deployment on urban robot taxis. Our traffic analysis addresses multi-agent scenarios and resolves competitive conflicts among robot taxi companies, crucial for smart city integration and optimizing autonomous vehicle performance in complex urban settings.

In this paper, we present a LiDAR Inertial Odometry (LIO) algorithm utilizing adaptive keyframe generation which achieves fast and accurate state estimation for aerial and ground robots. It is known that keyframe generation significantly affects the performance of Simultaneous Localization and Mapping (SLAM) algorithms. Unlike existing SLAM algorithms that generate keyframes based on fixed conditions, we propose to use adaptive keyframe generation conditions considering characteristics of surrounding environment using real-time LiDAR scans. When a keyframe is generated, the keyframe and the corresponding LiDAR measurements are stored in our novel data structure designed for efficient sub- map generation. The scan to sub-map matching module then uses the Generalized Iterative Closest Point (GICP) algorithm to adjust estimated states at a global scale, producing more accurate and globally consistent state estimation results even in large-scale underground environments. Experimental results from diverse types of underground environments show that the proposed method outperforms the existing state-of-the-art LIO algorithms in various metrics such as computational speed, CPU usage, and accuracy.

This paper presents a localization system robust against unreliable measurements and a resilient navigation system recovering from localization failures for Indy autonomous challenge (IAC). The IAC is a competition with full-scale autonomous race cars that drive at speeds up to 300 kph. Owing to high-speed and heavy vibration in the car, a GPS/INS system is prone to degrade causing critical localization errors, which leads to catastrophic accidents.In order to address this issue, we propose a robust localization system that probabilistically evaluates the credibility of multi-modal measurements. At a correction step of the Kalman filter, a degradation identification method with a novel hyper-parameter derived from Bayesian decision theory is introduced to choose the most credible measurement values in real-time. Since the racing condition is so harsh that even our robust localization method can fail for a short period of time, we present a resilient navigation system that enables the race car to continue to follow the race track in the event of a localization failure. Our system uses direct perception information in planning and execution until the completion of localization recovery.The proposed localization system is first validated in a simulation with real measurement data contaminated by large artificial noises. The experimental validation during an actual race is also presented. The last part of our paper shows the results from the real-world tests where our system recovers from failures and prevents accidents in real-time, which proves the resilience of the proposed navigation system.

This paper addresses a computationally efficient approach to localization and mapping in an indoor parking garage in the context of simultaneous localization and mapping. A parameterized map-building approach is introduced and implemented to represent the surrounding structures using a small number of geometric parameters. These parameters are obtained from horizontally and vertically ordered 3D LIDAR measurements and incorporated into an online filter to simultaneously estimate the map parameters and localize the vehicle. This approach enables the high-precision navigation and memory-efficient map representation of an environment with man-made structures with no need of global positioning system or external position fixes. Driving experiments were performed in indoor parking garages to verify and demonstrate the performance of the proposed localization and mapping approach.

Autonomous driving in an urban environment is one of the most actively studied topics. To date, many studies on autonomous driving can be classified into two main approaches: 1.local perception-based approach 2.global path tracking-based approach. However, each approach has its own limitations for fully autonomous driving. In the case of perception-based approach, it is impossible to autonomously drive to the global destination because it only runs locally within the sensor range. On the other hand, the path tracking-based approach relies heavily on accurate navigation. For accurate navigation, there are many studies using expensive equipment or extremely precise and detailed maps, but they have not been resolved yet, and they are also not practical. In this paper, we address the problem of autonomous driving in an urban environment through the proposed hybrid control architecture. Proposed control architecture is designed to be complementary of local perception-based and global path tracking-based approaches. Especially, end-to-end deep learning based autonomous driving, which mimics human driving, is applied as a local perceptionbased approach. In addition, path tracking-based autonomous driving is performed in an environment where directional information to the destination is required, such as intersections. At the same time, our hybrid control architecture effectively compensates for the navigation error using ICP matching between the perception-based driven trajectory and the global path to the destination without any highly detailed prior map or expensive equipment. The performance of the proposed architecture on a full-scale autonomous vehicle is verified through experiments in the urban environment.

In this paper, we derive an algorithm for enabling a single robotic unmanned aerial vehicle to herd a flock of birds away from a designated volume of space, such as the air space around an airport. The herding algorithm, referred to as the m-waypoint algorithm, is designed using a dynamic model of bird flocking based on Reynolds' rules. We derive bounds on its performance using a combination of reduced-order modeling of the flock's motion, heuristics, and rigorous analysis. A unique contribution of the paper is the experimental demonstration of several facets of the herding algorithm on flocks of live birds reacting to a robotic pursuer. The experiments allow us to estimate several parameters of the flocking model, and especially the interaction between the pursuer and the flock. The herding algorithm is also demonstrated using numerical simulations.

Autonomous driving technology has made significant advances in recent years. In order for self-driving cars to become practical, they are required to operate safely and reliably even under adverse driving conditions. However, most current autonomous driving cars have only been shown to be operational under amiable weather conditions, i.e., on sunny days on dry roads. In order to enable autonomous cars to handle adverse driving conditions such as rain and wet roads, the algorithm must be able to detect roads within a tolerable margin of error using sensors such as cameras and laser scanners. In this paper, we propose a sensor fusion algorithms that is able to operate under a variety of weather conditions, including rain. Our algorithm was validated when a strong shower occurred during the 2014 Hyundai Motor Company's Autonomous Car Competition. In this paper, we present the competition results that were collected on the same course on both sunny and rainy days. Based on the comparison, we propose the future directions to improve the autonomous driving capability under adverse environmental conditions.

Unmanned aerial vehicles (UAVs) are now very popular for many applications such as surveillance and transport and they are typically constructed starting from the design process. However, it is very time consuming and require all new airworthiness process. In this paper, we aim to provide a novel framework to automate an existing aircraft for unmanned operations using a humanoid robot, which is capable of manipulating the yoke, levers, switches and pedals to fly the airplane while operating various components such as lights and landing gears just as a human pilot would do. In this manner, the airplane can be converted for automated mode in a very short time without losing airworthiness. For testing, due to the complication of operating the robot in a real airplane, it is validated using a flight simulator, which provides the flight data over the network. The simulator is installed on a motion platform, which makes the robot move around due to the airplane's motion for realism. The proposed method is successfully validated in a series of flight simulations with various scenarios to show the feasibility of humanoid robot operation of an aircraft under various conditions.

Recently, there have been significant advances in self-driving cars, which will play key roles in future intelligent transportation systems. In order for these cars to be successfully deployed on real roads, they must be able to autonomously drive along collision-free paths while obeying traffic laws. In contrast to many existing approaches that use prebuilt maps of roads and traffic signals, we propose algorithms and systems using Unified Map built with various onboard sensors to detect obstacles, other cars, traffic signs, and pedestrians. The proposed map contains not only the information on real obstacles nearby but also traffic signs and pedestrians as virtual obstacles. Using this map, the path planner can efficiently find paths free from collisions while obeying traffic laws. The proposed algorithms were implemented on a commercial vehicle and successfully validated in various environments, including the 2012 Hyundai Autonomous Ground Vehicle Competition.

This paper presents modified A* and Lifelong Planning A* algorithms to facilitate more accurate path finding than existing methods, including the Minkowski sum for differential wheeled vehicles with shape constraints. We use a graphical method to check for obstructions without adding the outline of vehicles to obstacles. The method applies a procedure that enables vehicles to have forward movement with the smallest rotation possible, including their turning directions. Furthermore, we show that vehicles can pass through narrow passages because we accurately check for interference against obstacles using the graphical method. Consequently, we demonstrate via a series of simulations that our method can quickly replan a collision-free path while accurately taking into account the shape of vehicles.

We present a recursive path-planning method that efficiently generates a path by using reduced states of the search space and taking into account the kinematics, shape, and turning space of a car-like vehicle. Our method is based on a kinematics-aware node expansion method that checks for collisions based on the shape and turning space of a vehicle. We present two heuristics that simultaneously consider the kinematics of a vehicle with and without obstacles. In particular, for challenging environments containing complex obstacles and even narrow passages, we recursively identify intermediate goals and nodes that allow the vehicle to compute a path to its destination. We show the benefits of our method through simulations and experimental results by using an autonomous ground vehicle. Furthermore, we show that our method can efficiently generate a collision-free path for vehicles in complex environments with passageways.

Humanoid robots can be a highly desirable substitute for humans when it performs various tasks using tools and equipment designed for humans. One of such possible applications is controlling a vehicle. A humanoid robot can sit in the pilot's seat and command the vehicle using the control columns or steering wheels, pedals, switches, levers, and dials. In this paper, we propose a framework of automating a vehicle, an airplane in particular, with a humanoid. In order to perform various tasks of flying an unmodified airplane, the robot needs to perform three levels of tasks - recognition, decision, and action. The robot should collect information of the vehicle by using its own sensors and from various instruments in the cockpit, in addition to possible data link, a privilege of a robot. The robot then decides how to operate the flight control equipment in order to follow a given flight plan. Finally, it directly manipulates the control input equipment by computing the kinematic variables in the presence of various constraints from the surroundings. In order to validate the proposed framework, a piloting robot system is developed using a small low-cost humanoid and a flight simulation equipment designed for humans. The robot showed adequate performance to fly the airplane from cold start to landing to a stop on the runway.

This paper describes an integrated navigation sensor module, including a camera, a laser scanner, and an inertial sensor, for unmanned aerial vehicles (UAVs) to fly both indoors and outdoors. The camera and the gimbaled laser sensor work in a complementary manner to extract feature points from the environment around the vehicle. The features are processed using an online extended Kalman filter (EKF) in simultaneous localization and mapping (SLAM) algorithm to estimate the navigational states of the vehicle. In this paper, a new method is proposed for calibrating a camera and a gimbaled laser sensor. This calibration method uses a simple visual marker to calibrate the camera and the laser scanner with each other. We also propose a real-time navigation algorithm based on the EKF SLAM algorithm, which is suitable for our camera-laser sensor package. The algorithm merges image features with laser range data for state estimation. Finally, these sensors and algorithms are implemented on our octo-rotor UAV platform and the result shows that our onboard navigation module can provide a real-time three-dimensional navigation solution without any assumptions or prior information on the surroundings.

This paper presents recent advances in multiagent sensing and operation in dynamic environments. Technology trends point towards a fusion of wireless sensor networks with robotic swarms of mobile robots. In this paper, we discuss the coordination and collaboration between networked robotic systems, featuring algorithms for cooperative operations such as unmanned aerial vehicles (UAVs) swarming. We have developed cooperative actions of groups of agents such as probabilistic pursuit-evasion game for search and rescue operations, protection of resources, and security applications. We have demonstrated a hierarchical system architecture which provides wide-range sensing capabilities to unmanned vehicles through spatially deployed wireless sensor networks, highlighting the potential collaboration between wireless sensor networks and unmanned vehicles. This paper also includes a short review of our current research efforts in heterogeneous sensor networks, which is being evolved into mobile sensor networks with swarm mobility. In a very essential way, this represents the fusion of mobility of ensembles with the network embedded systems, the robotic sensor web.

In this paper, we present a vision-based terrain mapping and analysis system, and a model predictive control (MPC)-based flight control system, for autonomous landing of a helicopter-based unmanned aerial vehicle (UAV) in unknown terrain. The vision system is centered around Geyer et al.'s recursive multi-frame planar parallax algorithm (2006), which accurately estimates 3D structure using geo-referenced images from a single camera, as well as a modular and efficient mapping and terrain analysis module. The vision system determines the best trajectory to cover large areas of terrain or to perform closer inspection of potential landing sites, and the flight control system guides the vehicle through the requested flight pattern by tracking the reference trajectory as computed by a real-time MPC-based optimization. This trajectory layer, which uses a constrained system model, provides an abstraction between the vision system and the vehicle. Both vision and flight control results are given from flight tests with an electric UAV.

This paper presents a flight management system (FMS) implemented as on-board intelligence for rotorcraft-based unmanned aerial vehicles (RUAV's), in order to gradually refine given abstract mission commands into real-time control signals for each vehicle. A strategy planner uses the probabilistic decision making algorithms to determine suboptimal action at each time step. A graphical interface on ground station enables human intervention. We derive nonlinear dynamics model upon which we design a tracking control layer using nonlinear model predictive control and integrate with a trajectory generator for logistical action planning. The proposed structure has been implemented on Berkeley RUAVs and validated in probabilistic pursuit-evasion games to show the possibility of intelligent flying robots.

We consider the problem of having a team of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) pursue a second team of evaders while concurrently building a map in an unknown environment. We cast the problem in a probabilistic game theoretical framework, and consider two computationally feasible greedy pursuit policies: local-mar and global-max. To implement this scenario on real UAVs and UGVs, we propose a distributed hierarchical hybrid system architecture which emphasizes the autonomy of each agent, yet allows for coordinated team efforts. We describe the implementation of the architecture on a fleet of UAVs and UGVs, detailing components such as high-level pursuit policy computation, map building and interagent communication, and low-level navigation, sensing, and control. We present both simulation and experimental results of real pursuit-evasion games involving our fleet of UAVs and UGVs, and evaluate the pursuit policies relating expected capture times to the speed and intelligence of the evaders and the sensing capabilities of the pursuers.