We introduce RMP-YOLO, a unified framework designed to provide robust motion predictions even with incomplete input data. Our key insight stems from the observation that complete and reliable historical trajectory data plays a pivotal role in ensuring accurate motion prediction. Therefore, we propose a new paradigm that prioritizes the reconstruction of intact historical trajectories before feeding them into the prediction modules. Our approach introduces a novel scene tokenization module to enhance the extraction and fusion of spatial and temporal features. Following this, our proposed recovery module reconstructs agents' incomplete historical trajectories by leveraging local map topology and interactions with nearby agents. The reconstructed, clean historical data is then integrated into the downstream prediction modules. Our framework is able to effectively handle missing data of varying lengths and remains robust against observation noise while maintaining high prediction accuracy. Furthermore, our recovery module is compatible with existing prediction models, ensuring seamless integration. Extensive experiments validate the effectiveness of our approach, and deployment in real-world autonomous vehicles confirms its practical utility. In the 2024 Waymo Motion Prediction Competition, our method, RMP-YOLO, achieves state-of-the-art performance, securing third place. Our code is open-source at https://github.com/ggosjw/RMP-YOLO.

Imitation learning has emerged as an effective paradigm for training visuo-motor policies in robotic manipulation. In real-world scenarios, visuo-motor policies are required to be effective, sample-efficient, and capable of adapting to dynamic environments. A key factor influencing these capabilities is the quality of visual representations. Conventional approaches that learn a vision encoder and policy network from scratch often result in suboptimal representations, as the training process tends to prioritize policy optimization over rich semantic feature extraction. Alternatively, while pre-trained large vision models offer strong general-purpose features, they often fail to capture the fine-grained, task-specific information required for effective manipulation. To capture rich and informative visual features, we propose TOC-DP, a novel framework that integrates SlotAttention to facilitate object-centric representation learning. Task-specific segmentation priors are incorporated as an inductive bias to enhance the task-awareness and object-awareness of the learned visual features. The extracted representations are subsequently refined to encode action-aware information during downstream policy learning. Extensive experiments on the Meta-World benchmark and real-world tasks demonstrate that TOC-DP achieves a 30% improvement in success rate over baseline methods during deployment for a variety of scenarios.

This paper presents a novel control framework that integrates reservoir computing (RC) with Tube model predictive control (Tube-MPC) for robust path following in quadrotor autonomous underwater vehicles (QAUVs) under sudden fault conditions. The proposed RC-Tube-MPC leverages the dynamic modeling capabilities of RC to efficiently approximate complex nonlinear behaviors, while Tube correction ensures robust performance despite model uncertainties and external disturbances. Comparative simulations demonstrate that RC-Tube-MPC outperforms alternative approaches in terms of path following accuracy and computational efficiency. Additionally, the influence of training data length on learning performance is analyzed, revealing that the proposed method maintains superior performance across various data regimes. Notably, in severe fault scenarios, such as a fault factor of 0.3, RC-Tube-MPC uniquely restores convergence to the reference path. These results underscore the potential of the integrated RC-Tube-MPC approach for real-time control applications in dynamic, fault-prone underwater environments.

As the field progresses toward Artificial General Intelligence (AGI), there is a pressing need for more comprehensive and insightful evaluation frameworks that go beyond aggregate performance metrics. This paper introduces a unified rating system that jointly models the difficulty of individual test cases and the competency of AI models (or humans) across vision, language, and action domains. Unlike existing metrics that focus solely on models, our approach allows for fine-grained, difficulty-aware evaluations through competitive interactions between models and tasks, capturing both the long-tail distribution of real-world challenges and the competency gap between current models and full task mastery. We validate the generalizability and robustness of our system through extensive experiments on multiple established datasets and models across distinct AGI domains. The resulting rating distributions offer novel perspectives and interpretable insights into task difficulty, model progression, and the outstanding challenges that remain on the path to achieving full AGI task mastery. We have made our code and results publicly available at https://ss47816.github.io/AGI-Elo/.

Learning an effective manipulation policy with high efficiency in robotics continues to be a significant challenge. In this paper, we propose E-GAIL, which aims to learn manipulation policies efficiently from a limited set of demonstrations with negative corruption and long-term rewards under the framework of GAIL. Specifically, we propose two techniques: 1) Utilizing both short-term and long-term observations to offer additional rewards for training, accelerating convergence. 2) Incorporating negative actions into generated trajectories for corruption to improve data effectiveness and increase success rates. E-GAIL achieves a 25% improvement in success rates across multiple manipulation tasks, requiring 70% fewer episodes for policy convergence, highlighting its efficiency with limited demonstrations. Our video is available at https://youtu.be/bIDfOjYcY54 .

Collision-free motion planning for redundant robot manipulators in complex environments is yet to be explored. Although recent advancements at the intersection of deep reinforcement learning (DRL) and robotics have highlighted its potential to handle versatile robotic tasks, current DRL-based collision-free motion planners for manipulators are highly costly, hindering their deployment and application. This is due to an overreliance on the minimum distance between the manipulator and obstacles, inadequate exploration and decision-making by DRL, and inefficient data acquisition and utilization. In this article, we propose URPlanner, a universal paradigm for collision-free robotic motion planning based on DRL. URPlanner offers several advantages over existing approaches: it is platform-agnostic, cost-effective in both training and deployment, and applicable to arbitrary manipulators without solving inverse kinematics. To achieve this, we first develop a parameterized task space and a universal obstacle avoidance reward that is independent of minimum distance. Second, we introduce an augmented policy exploration and evaluation algorithm that can be applied to various DRL algorithms to enhance their performance. Third, we propose an expert data diffusion strategy for efficient policy learning, which can produce a large-scale trajectory dataset from only a few expert demonstrations. Finally, the superiority of the proposed methods is comprehensively verified through experiments.

In the realm of robotic grasping, achieving accurate and reliable interactions with the environment is a pivotal challenge. Traditional methods of grasp planning methods utilizing partial point clouds derived from depth image often suffer from reduced scene understanding due to occlusion, ultimately impeding their grasping accuracy. Furthermore, scene reconstruction methods have primarily relied upon static techniques, which are susceptible to environment change during manipulation process limits their efficacy in real-time grasping tasks. To address these limitations, this paper introduces a novel two-stage pipeline for dynamic scene reconstruction. In the first stage, our approach takes scene scanning as input to register each target object with mesh reconstruction and novel object pose tracking. In the second stage, pose tracking is still performed to provide object poses in real-time, enabling our approach to transform the reconstructed object point clouds back into the scene. Unlike conventional methodologies, which rely on static scene snapshots, our method continuously captures the evolving scene geometry, resulting in a comprehensive and up-to-date point cloud representation. By circumventing the constraints posed by occlusion, our method enhances the overall grasp planning process and empowers state-of-the-art 6-DoF robotic grasping algorithms to exhibit markedly improved accuracy.

This paper presents a novel approach for affordance-informed robotic manipulation by introducing 3D keypoints to enhance the understanding of object parts’ functionality. The proposed approach provides direct information about what the potential use of objects is, as well as guidance on where and how a manipulator should engage, whereas conventional methods treat affordance detection as a semantic segmentation task, focusing solely on answering the what question. To address this gap, we propose a Fusion-based Affordance Keypoint Network (FAKP-Net) by introducing 3D keypoint quadruplet that harnesses the synergistic potential of RGB and Depth image to provide information on execution position, direction, and extent. Benchmark testing demonstrates that FAKP-Net outperforms existing models by significant margins in affordance segmentation task and keypoint detection task. Real-world experiments also showcase the reliability of our method in accomplishing manipulation tasks with previously unseen objects. Our source code and video demo will be public.

Robotic manipulation is a grand domain that primarily involves the use of robotic arms to interact with objects in the environment. While proposed methods have achieved advancements in grasping objects, they rely heavily on extensive training data that presents a significant challenge due to the labor-intensive process of human annotation. To address the issue, we propose GraspContrast, a self-supervised contrastive learning framework leveraging unlabeled RGB-D images to enhance point-wise feature representations for 6-DoF grasp detection. Our method designs a dual-branch network architecture to learn transformations that embed positive point pairs nearby, while pushing negative point pairs far apart. Specifically, we discuss a false negative elimination strategy to explicitly detect and remove the false negative samples that undesirably repel the point instances from the geometrically similar samples. Our method exhibits consistent improvements over existing learning-based grasp detection methods on both the GraspNet-1B benchmark and physical UR10e platform. These significant performance gains demonstrate the effectiveness of our proposed framework.

For humans, dense bin packing heavily relies on force perception. However, current robotic packing studies only focus on the visual input or adopt auxiliary push-to-place actions to eliminate gaps, suffering from high time expenditure and poor robustness. To address such limitations, we first introduce a novel external force estimation method based on the generalized momentum observer, which can avoid the influence of joint acceleration noises and achieve real-time high-precision monitoring. Second, to obtain compliant interaction and fine robustness, an adaptive variable impedance policy is developed to track dynamic motion and desired force, and compensate for uncertainties. Meanwhile, we perform dissipativity analysis and a virtual energy supply function is augmented to the system for optimization, providing a solid foundation for stability. Third, we propose an efficient packing methodology with three sub- tasks by considering the distinct interaction and constraint states in different areas. Our packing strategies eliminate the need for subsequent auxiliary actions and are proven to enhance efficiency. We perform quantitative evaluations to verify our external force estimation method, conduct comparison studies with current packing methods, and investigate the contribution of our dissipativity-based adaptive controller. The superior results not only prove the robustness and efficiency of our pipeline, but also pave the way for practical applications of packing.

Adverse weather has raised a big challenge for autonomous vehicles. Unreliable measurements due to sensor degradation could seriously affect the performance of autonomous driving tasks, such as perception and localization. In this work, we study sensor degradation in rainy weather and present a novel method that evaluates the uncertainty for each laser measurement from a 3D LiDAR. With uncertainty estimation, downstream tasks that rely on LiDAR input (e.g., perception or localization) can increase their reliability by adjusting their reliance on laser measurements with varying fidelity. Alternatively, uncertainty estimation can be used for sensor performance evaluation. Our proposed method, SmartRainNet, uses an attention-based Mixture Density Network to model the dependence between neighboring laser measurements and then calculate the probability density for each laser measurement as an uncertainty score. We evaluate SmartRainNet on synthetic and naturalistic sensor degradation datasets and provide qualitative and quantitative results to demonstrate the effectiveness of our method in evaluating uncertainty. Finally, we demonstrate three practical applications of uncertainty estimation to address autonomous driving challenges in rainy weather.

Trajectory planning plays a crucial role in autonomous driving systems, as it is tasked to generate feasible trajectories under highly dynamic scenarios within the time constraint. This paper proposes a novel two-stage coarse-to-fine framework for efficient sampling-based trajectory planning. The proposed method is designed to iteratively generate new trajectory samples focused on the low-cost regions in the sampling space. Two trajectory exploration algorithms are well-designed for efficient search in discretized coarse global space and continuous fine local space, respectively. Experimental results on the first-of-its-kind planning benchmark tool CommonRoad show that our method significantly outperforms the baseline methods both in optimality and computational efficiency. Overall, our approach offers a promising solution for efficient and effective trajectory planning in more autonomous vehicle applications.

Autonomous driving in rain remains challenging. Rain causes sensor performance degradation that can affect sensor measurement quality. During the rain, lasers may suffer from energy loss due to raindrop absorption. As a result, some laser measurements reflected from obstacles may not be recognized by the LiDAR sensor, thus raising potential risks for autonomous vehicles. This work investigates a novel task that aims to detect those missing measurements. Our solution uses a two-stage learning method to generate an anomaly score for each missing measurement, representing the likelihood of being caused by rain. We evaluate our method with real-world data and demonstrate its effectiveness in identifying anomalous missing measurements through qualitative and quantitative experiments.

Autonomous driving in the rain remains a challenge. One main problem is performance degradation caused by rain. This work introduces a new dataset to study this problem. Our dataset is collected from a full-scale vehicle equipped with a 3D LiDAR sensor and multiple forward-facing cameras under various rainy conditions. In addition, rainfall intensity is recorded in real-time from a rain sensor. The combination of sensor and rainfall intensity measurement is designed for studying algorithm performance under different levels of rainfall. In this work, in addition to presenting dataset creation details, we also introduce three degradation evaluation tasks with baseline results, including rainfall intensity estimation, LiDAR degradation estimation, and 2D object detection evaluation. This dataset, development kit, and baseline codes will be made available at https://smart-rain-dataset.github.io/

Sensor degradation is one of the major challenges for autonomous driving. During the rain, the interference from raindrops can negatively influence LiDAR measurements. For example, valid measurements could be reduced during the rain, and some measurements may become noisy. Unreliable measurements can lead to potential safety issues if autonomous driving systems are unaware of these changes. In this work, we will release a naturalistic driving dataset to advance the research in studying LiDAR degradation. Our dataset consists of 3D LiDAR scans collected by a data collection vehicle under various rainy conditions. Besides these raw scans, we also release LiDAR scan pairs (each pair consists of one scan from rainy weather and one scan from clear weather at the same location). These LiDAR pairs are developed to help researchers identify LiDAR degradation. Finally, we will release a toolbox integrated with mapping, localization, and scan synthesis functions used to create this dataset. This toolbox can facilitate dataset creation for studying degradation in other harsh weather conditions. More information can be found at https://smart-rain-dataset.github.io/.

Category-level object pose estimation involves estimating the 6D pose and the 3D metric size of objects from predetermined categories. While recent approaches take categorical shape prior information as reference to improve pose estimation accuracy, the single-stage network design and training manner lead to sub-optimal performance since there are two distinct tasks in the pipeline. In this paper, the advantage of two-stage pipeline over single-stage design is discussed. To this end, we propose a two-stage deformation-and-registration pipeline called DR-Pose, which consists of completion-aided deformation stage and scaled registration stage. The first stage uses a point cloud completion method to generate unseen parts of target object, guiding subsequent deformation on the shape prior. In the second stage, a novel registration network is designed to extract pose-sensitive features and predict the representation of object partial point cloud in canonical space based on the deformation results from the first stage. DR-Pose produces superior results to the state-of-the-art shape prior-based methods on both CAMERA25 and REAL275 benchmarks. Codes are available at https://github.com/Zray26/DR-Pose.git.

The central idea of contrastive learning is to discriminate between different instances and force different views from the same instance to share the same representation. To avoid trivial solutions, augmentation plays an important role in generating different views, among which random cropping is shown to be effective for the model to learn a generalized and robust representation. Commonly used random crop operation keeps the distribution of the difference between two views unchanged along the training process. In this work, we show that adaptively controlling the disparity between two augmented views along the training process enhances the quality of the learned representations. Specifically, we present a parametric cubic cropping operation, ParamCrop, for video contrastive learning, which automatically crops a 3D cubic by differentiable 3D affine transformations. ParamCrop is trained simultaneously with the video backbone using an adversarial objective, so that it learns to increase the contrastive loss and thus gradually reduces the shared contents between two cropped views. Experiments show that this adaptive and gradual increase in the disparity yielded by ParamCrop is beneficial to learning a strong and generalized representation for downstream tasks, which is shown to be effective on multiple contrastive learning frameworks and video backbones.

In recent years, following the advances in autonomous driving technology, low-speed autonomous service vehicles such as delivery, patrolling, and road-cleaning vehicles have started to emerge. As a promising future cleaning solution, autonomous cleaning vehicles are expected to address the workforce shortage many countries face in the near future. This paper describes the detailed design of an autonomous research platform for cleaning operations in mixed indoor & outdoor environments. An electric manual vacuum sweeper is retrofitted into an autonomous sweeper equipped with the Drive-by-Wire (DBW) system, computer, sensors, and actuators essential for autonomous driving. A complete autonomous driving software stack is also developed upon this hardware setup to enable the vehicle to navigate itself safely in various challenging operating environments. The system has been extensively tested in different environments on the National University of Singapore (NUS) campus, including private roads, car parks, warehouses, and public plaza areas.

Multi-Object Tracking (MOT) is an important task in computer vision. Most research works on MOT focus on tracking multiple objects of a single category, such as humans. However, many applications, such as wildlife surveillance and autonomous driving, require multi-object trackers to track multiple categories of objects. In this work, the extension of state-of-the-art one-shot MOT methods for multi-class tracking is shown to be non-trivial, yielding deteriorated tracking performance due to the complexity of balancing the jointly learned detection and association tasks. To tackle this limitation, a simple improvement to FairMOT is proposed, which replaces the detection head with the decoupled head from YOLOX. The simplified multi-class detection framework yields improved detection and tracking results on the BDD100K MOT validation dataset, yielding an mHOTA of 25.3% compared to 19.7%.

Concrete infrastructures provide the means to connect cities and transport people and goods. They require regular inspection to assess their current conditions. Aerial work platforms and underbridge platforms or scaffolding are the common equipment used for inspection of elevated infrastructure. These methods often cost more to operate and maintain, are time-consuming, and raise risks for the inspector. One interesting field of research for UAVs that can be used for infrastructure inspection is aerial perching. A perching UAV can be loaded with an inspection apparatus foregoing the need for costly equipment and risks involved in the inspection. Many have presented aerial perching for various applications and not as much for applications related to concrete infrastructure inspection. This study investigates a perching UAV that can perform perching on both smooth and rough concrete surfaces. This paper presents an unmanned aerial system that utilizes a suction-based perching mechanism with a pneumatic supply tethered from the ground. The proposed perching mechanism provides a reliable and high payload capacity needed for non-destructive testing of the infrastructure. The paper introduces the concept, presents the design and proof of concept, and validates the idea through actual bridge experiments.

Point clouds are often sparse and incomplete, which imposes difficulties for real-world applications. Existing shape completion methods tend to generate rough shapes without fine-grained details. Considering this, we introduce a two-branch network for shape completion. The first branch is a cascaded shape completion sub-network to synthesize complete objects, where we propose to use the partial input together with the coarse output to preserve the object details during the dense point reconstruction. The second branch is an auto-encoder to reconstruct the original partial input. The two branches share a same feature extractor to learn an accurate global feature for shape completion. Furthermore, we propose two strategies to enable the training of our network when ground truth data are not available. This is to mitigate the dependence of existing approaches on large amounts of ground truth training data that are often difficult to obtain in real-world applications. Additionally, our proposed strategies are also able to improve the reconstruction quality for fully supervised learning. We verify our approach in self-supervised, semi-supervised and fully supervised settings with superior performances. Quantitative and qualitative results on different datasets demonstrate that our method achieves more realistic outputs than state-of-the-art approaches on the point cloud completion task.

Motions are reflected in videos as the movement of pixels, and actions are essentially patterns of inconsistent motions between the foreground and the background. To well distinguish the actions, especially those with complicated spatio-temporal interactions, correctly locating the prominent motion areas is of crucial importance. However, most motion information in existing videos are difficult to label and training a model with good motion representations with supervision will thus require a large amount of human labour for annotation. In this paper, we address this problem by self-supervised learning. Specifically, we propose to learn Motion from Static Images (MoSI). The model learns to encode motion information by classifying pseudo motions generated by MoSI. We furthermore introduce a static mask in pseudo motions to create local motion patterns, which forces the model to additionally locate notable motion areas for the correct classification. We demonstrate that MoSI can discover regions with large motion even without fine-tuning on the downstream datasets. As a result, the learned motion representations boost the performance of tasks requiring understanding of complex scenes and motions, i.e., action recognition. Extensive experiments show the consistent and transferable improvements achieved by MoSI. Codes will be soon released.

Deep learning technique has yielded significant improvements in point cloud completion with the aim of completing missing object shapes from partial inputs. However, most existing methods fail to recover realistic structures due to over-smoothing of fine-grained details. In this paper, we develop a voxel-based network for point cloud completion by leveraging edge generation (VE-PCN). We first embed point clouds into regular voxel grids, and then generate complete objects with the help of the hallucinated shape edges. This decoupled architecture together with a multi-scale grid feature learning is able to generate more realistic on-surface details. We evaluate our model on the publicly available completion datasets and show that it outperforms existing state-of-the-art approaches quantitatively and qualitatively. Our source code is available at https://github.com/xiaogangw/VE-PCN.

This paper addresses the safe path planning problem for autonomous mobility with multi-modal perception uncertainties. Specifically, we assume that different sensor inputs lead to different Gaussian process regulated perception uncertainties (named as multi-modal perception uncertainties). We implement a Bayesian inference algorithm, which merges the multi-modal GP-regulated uncertainties into a unified one and translates the unified uncertainty into a dynamic risk map. With the safe path planner taking the risk map as input, we are able to plan a safe path for the autonomous vehicle to follow. Experimental results on an autonomous golf cart testbed validate the applicability and efficiency of the proposed algorithm.

Navigation through dynamic pedestrian environments in a socially compliant manner is still a challenging task for autonomous vehicles. Classical methods usually lead to unnatural vehicle behaviours for pedestrian navigation due to the difficulty in modeling social conventions mathematically. This paper presents an end-to-end path planning system that achieves autonomous navigation in dynamic environments through imitation learning. The proposed system is based on a fully convolutional neural network that maps the raw sensory data into a confidence map for path extraction. Additionally, a classification network is introduced to reduce the unnecessary re-plannings and ensures that the vehicle goes back to the global path when re-planning is not needed. The imitation learning based path planner is implemented on an autonomous wheelchair and tested in a new real-world dynamic pedestrian environment. Experimental results show that the proposed system is able to generate paths for different driving tasks, such as pedestrian following, static and dynamic obstacles avoidance, etc. In comparison to the state-of-the-art method, our system is superior in terms of generating human-like trajectories.

Autonomous driving in rainy conditions remains a big challenge. One of the issues is sensor degradation. LiDAR is commonly used in autonomous driving systems to perceive and understand surrounding environments. However, LiDAR performance can be degraded by rain, thereby influencing other system performance (e.g., perception or localization). Therefore, knowing how much degradation exists in current LiDAR measurements is necessary. Most existing methods can only measure LiDAR degradation in controlled environments (e.g., a chamber with simulated rain); how to quantify LiDAR degradation in dynamic environments while the autonomous vehicle is moving is still a difficult problem. In this work, we propose a novel approach to address this problem using an anomaly detection method. Our method has been evaluated on simulated and real-world data. Experimental results demonstrate the effectiveness of our method to capture LiDAR degradation and yield reasonable degradation estimations. Our experimental data and codes are accessible from http://rain.smart.mit.edu/smartrain/.

Autonomous vehicles on city roads and especially in pedestrian environments require agility to navigate narrow passages and turn in tight spaces, leading to the need for a real-time, robust and adaptable controller. In this paper, we present orientation and context aware controllers for autonomous vehicles that can closely track the reference path wit alh respect to the current state of the vehicle, environmental properties, and the desired target orientation at the desired target location. Our proposed controllers are derived from the widely used pure pursuit controller. We validate our proposed controllers with respect to the baseline pure pursuit controller in simulation and on a full-size autonomous vehicle in a pedestrian environment. Our experimental results suggest significant improvements in adaptability and tracking performance compared to the pure pursuit controller.

This paper studies the group multi-object tracking (MOT) problem in dynamic pedestrian environments, with intended application to safe navigation for autonomous vehicles. We complete a full autonomous vehicle navigation pipeline from object detection, tracking, grouping, to risk map generation and safe path planning. Our main contribution is to instantiate a group multi-object tracking algorithm, which provides the crucial grouped activity information, i.e. group position, group velocity, group size, to the risk map generator, and therewith produce a stable and robust risk map for the downstream safe path planner. Experimental results with real world data show the socially acceptable, robust and stable performance of the proposed algorithm over its individual MOT counterpart.

Point clouds are often sparse and incomplete. Existing shape completion methods are incapable of generating details of objects or learning the complex point distributions. To this end, we propose a cascaded refinement network together with a coarse-to-fine strategy to synthesize the detailed object shapes. Considering the local details of partial input with the global shape information together, we can preserve the existing details in the incomplete point set and generate the missing parts with high fidelity. We also design a patch discriminator that guarantees every local area has the same pattern with the ground truth to learn the complicated point distribution. Quantitative and qualitative experiments on different datasets show that our method achieves superior results compared to existing state-of-the-art approaches on the 3D point cloud completion task. Our source code is available at https://github.com/xiaogangw/cascaded-point-completion.git.

Accurate 6D object pose estimation is fundamental to robotic manipulation and grasping. Previous methods follow a local optimization approach which minimizes the distance between closest point pairs to handle the rotation ambiguity of symmetric objects. In this work, we propose a novel discrete- continuous formulation for rotation regression to resolve this local-optimum problem. We uniformly sample rotation anchors in SO(3), and predict a constrained deviation from each anchor to the target, as well as uncertainty scores for selecting the best prediction. Additionally, the object location is detected by aggregating point-wise vectors pointing to the 3D center. Experiments on two benchmarks: LINEMOD and YCB-Video, show that the proposed method outperforms state-of-the-art approaches. Our code is available at https://github.com/mentian/object-posenet.

This paper investigates the safe path planning problem for an autonomous vehicle operating in unstructured, cluttered environments. While some objects may be accurately with canonical perception algorithms, other objects and clutter may be harder to track. We present an approach that combines two methods of risk assessment: for objects with reliable tracking, we use a Gaussian Process (GP) regulated risk map to describe the risk map information; for unknown objects that we fail to accurately track, we compute a Dynamic Risk Density (DRD) from the overall occupancy and velocity field from LiDAR scan snapshots. Several methods are proposed for combining the GP risk map and DRD, and the resultant hybrid risk map is used for the proposed safe path planning algorithm. Experimental results on an autonomous buggy show that the hybrid risk map is able to yield a safe path planner to navigate the autonomous testbed within the cluttered environments.

Many indoor spaces have constantly changing layouts and may not be mapped by an autonomous vehicle, yet maps such as floor plans or evacuation maps of these places are common. We propose a method for an autonomous robot to localize itself on such maps with inconsistent scale using Stochastic Gradient Descent (SGD) with scan matching using a 2D LiDAR. We also introduce a new scale state in 2D localization to manage the possible inconsistent scale of the input map. Experiments are conducted in an indoor corridor using three different input maps - a point cloud, a floor plan, and a hand-drawn map. The SGD localization algorithm is bench-marked to Adaptive Monte Carlo Localization (AMCL). In a point cloud mapped environment, our algorithm achieves 0.264m and 5.26° average position and heading error respectively. On the hand-drawn map, our SGD localization algorithm remains robust while AMCL fails. The role of the scale state in our SGD localization algorithm is demonstrated in poorly scaled maps.

In view of the difficulty in reconstructing object details in point cloud completion, we propose a shape prior learning method for object completion. The shape priors include geometric information in both complete and the partial point clouds. We design a feature alignment strategy to learn the shape prior from complete points, and a coarse to fine strategy to incorporate partial prior in the fine stage. To learn the complete objects prior, we first train a point cloud auto-encoder to extract the latent embeddings from complete points. Then we learn a mapping to transfer the point features from partial points to that of the complete points by optimizing feature alignment losses. The feature alignment losses consist of a L2 distance and an adversarial loss obtained by Maximum Mean Discrepancy Generative Adversarial Network (MMD-GAN). The L2 distance optimizes the partial features towards the complete ones in the feature space, and MMD-GAN decreases the statistical distance of two point features in a Reproducing Kernel Hilbert Space. We achieve state-of-the-art performances on the point cloud completion task. Our code is available at https://github.com/xiaogangw/point-cloud-completion-shape-prior.

In recent years, self-supervised methods for monocular depth estimation has rapidly become an significant branch of depth estimation task, especially for autonomous driving applications. Despite the high overall precision achieved, current methods still suffer from a) imprecise object-level depth inference and b) uncertain scale factor. The former problem would cause texture copy or provide inaccurate object boundary, and the latter would require current methods to have an additional sensor like LiDAR to provide depth ground-truth or stereo camera as additional training inputs, which makes them difficult to implement. In this work, we propose to address these two problems together by introducing DNet. Our contributions are twofold: a) a novel dense connected prediction (DCP) layer is proposed to provide better object-level depth estimation and b) specifically for autonomous driving scenarios, dense geometrical constrains (DGC) is introduced so that precise scale factor can be recovered without additional cost for autonomous vehicles. Extensive experiments have been conducted and, both DCP layer and DGC module are proved to be effectively solving the aforementioned problems respectively. Thanks to DCP layer, object boundary can now be better distinguished in the depth map and the depth is more continues on object level. It is also demonstrated that the performance of using DGC to perform scale recovery is comparable to that using ground-truth information, when the camera height is given and the ground point takes up more than 1.03% of the pixels. Code is available at https://github.com/TJ-IPLab/DNet.

Large-scale point cloud generated from 3D sensors is more accurate than its image-based counterpart. However, it is seldom used in visual pose estimation due to the difficulty in obtaining 2D-3D image to point cloud correspondences. In this paper, we propose the 2D3D-MatchNet - an end-to-end deep network architecture to jointly learn the descriptors for 2D and 3D keypoint from image and point cloud, respectively. As a result, we are able to directly match and establish 2D-3D correspondences from the query image and 3D point cloud reference map for visual pose estimation. We create our Oxford 2D-3D Patches dataset from the Oxford Robotcar dataset with the ground truth camera poses and 2D-3D image to point cloud correspondences for training and testing the deep network. Experimental results verify the feasibility of our approach.

Government data identifies driver behaviour errors as a factor in 94% of car crashes, and autonomous vehicles (AVs), which avoids risky driver behaviours completely, are expected to reduce the number of road crashes significantly. Thus, one of the central focuses of developing AVs is to ensure safety during navigation. However, in reality, AV safety has been far below its expectation, and so far, no government has allowed for complete autonomous driving without human supervision. This paper proposes a dynamic safe path planning algorithm for AVs with Gaussian process regulated risk map. By reasonably assuming that the output of the object detection and tracking module follows a multi-variate Gaussian distribution, we put forward a safe path planning paradigm with Gaussian process regulated risk map, ensuring safety with high confidence. Both simulation results and in-vehicle tests demonstrate the effectiveness of the proposed algorithm.

In this paper, we focus on a rain removal task from a single image of the urban street scene for autonomous driving in rain. We develop a Convolutional Neural Network which takes a rainy image as input, and directly recovers a clean image in the presence of rain streaks, atmospheric veiling effect (haze, fog, mist) caused by distant rain streak accumulation. We propose a synthetic dataset containing images of urban street scenes with different rain intensities, orientations and haziness levels for training and evaluation. We evaluate our method quantitatively and qualitatively on the synthetic data. Experiments show that our model outperforms state-of-the-art methods. We also test our method qualitatively on the real-world data. Our model is fast and it takes 0.05s for an image of 1024 × 512. Our model can be seamlessly integrated with existing image-based high-level perception algorithms for autonomous driving in rain. Experiment results show that our deraining method improves semantic segmentation and object detection largely for autonomous driving in rain.

Image-based localization plays an important role in today's autonomous driving technologies. However, in large scale outdoor environments, challenging conditions, e.g., lighting changes or different weather, heavily affect image appearance and quality. As a key component of feature-based visual localization, image feature detection and matching deteriorate severely and cause worse localization performance. In this paper, we propose a novel method for robust image feature matching under drastically changing outdoor environments. In contrast to existing approaches which try to learn robust feature descriptors, we train a deep network that outputs the low-rank representations of the images where the undesired variations on the images are removed, and perform feature extraction and matching on the learned low-rank space. We demonstrate that our learned low-rank images largely improve the performance of image feature matching under varying conditions over a long period of time.

Autonomous driving requires 3D perception of vehicles and other objects in the in environment. Much of the current methods support 2D vehicle detection. This paper proposes a flexible pipeline to adopt any 2D detection network and fuse it with a 3D point cloud to generate 3D information with minimum changes of the 2D detection networks. To identify the 3D box, an effective model fitting algorithm is developed based on generalised car models and score maps. A two-stage convolutional neural network (CNN) is proposed to refine the detected 3D box. This pipeline is tested on the KITTI dataset using two different 2D detection networks. The 3D detection results based on these two networks are similar, demonstrating the flexibility of the proposed pipeline. The results rank second among the 3D detection algorithms, indicating its competencies in 3D detection.

In this paper, we consider the problem of data association in feature cloud matching. While Joint Compatibility (JC) test is a widely adopted technique for searching the global optimal data association, it becomes less restrictive as more features are well matched. The early well-matched features contribute little to total matching cost while the gating threshold increases in the chi-square test, which allows the acceptance of bad feature pairings in the last step. In this paper, we propose the Conditional Compatibility (CC) test, which is not only more restrictive than JC test, but also probabilistically sound. The proposed test of a new feature pairing is based on the conditional probability distribution of feature locations given the early pairings. CC test can be added into any JC test based search algorithm, such as Joint Compatibility Branch and Bound (JCBB), Incremental Posterior Joint Compatibility (IPJC) and FastJCBB, without increasing much computational complexity. The more restrictive criterion of accepting a feature pairing, not only helps to reject bad associations, but also bounds the search space, which substantially improves the search efficiency. The real matching experiments justify that our algorithm produces better feature cloud matching results in a more efficient manner.

Environment understanding, object detection and recognition are crucial skills for robots operating in the real world. In this paper, we propose a Convolutional Neural Network with multi-task objectives: object detection and scene classification in one unified architecture. The proposed network reasons globally about an image to understand the scene, hypothesize object locations, and encodes global scene features with regional object features to improve object recognition. We evaluate our network on the standard SUN RGBD dataset. Experiments show that our approach outperforms state-of-the-arts. Network predictions are further transformed into continuous robot beliefs to ensure temporal coherence and extended to 3D space for robotics applications. We embed the whole framework in Robot Operating System, and evaluate its performance on a real robot for semantic mapping and grasp detection.

We present a real-time vehicle detection and tracking system to accomplish the complex task of driving behavior analysis in urban environments. We propose a robust fusion system that combines a monocular camera and a 2D Lidar. This system takes advantage of three key components: robust vehicle detection using deep learning techniques, high precision range estimation from Lidar, and road context from the prior map knowledge. The camera and Lidar sensor fusion, data association and track management are all performed in the global map coordinate system by taking into account the sensors' characteristics. Lastly, behavior reasoning is performed by examining the tracked vehicle states in the lane coordinate system in which the road context is encoded. We validated our approach by tracking a leading vehicle while it performed usual urban driving behaviors such as lane keeping, stop-and-go at intersections, lane changing, overtaking and turning. The leading vehicle was tracked consistently throughout the 2.3 km route and its behavior was classified reliably.

We focus on the task of amodal 3D object detection, which is to predict object locations, dimensions, poses and categories in the real world. We introduce a 3D Convolutional Neural Network that takes a volumetric representation of an indoor scene as input and predicts 3D object bounding boxes, object categories, and orientations. Unlike prior state-of-the-arts, our approach does not depend on region proposal techniques to hypothesize object locations. We treat detection and recognition as one regression problem in a single network. Our elegant model is extremely fast and all predictions are reasoned from the global context of a point cloud in a continuous pipeline. We evaluate our approach on two standard datasets: the NYUv2 RGBD dataset and the SUN RGBD dataset. Experiments show that our approach is faster than start-of-the-art 3D detectors by several orders of magnitude towards real-time amodal 3D object detection.

This paper introduces a map-based localization method aiming to increase robustness in rainy conditions. This method utilizes two types of features: ground reflectivity features and vertical features extracted from 3D LIDAR scans and builds vehicle pose belief with two filters: a histogram filter and a particle filter. The posterior distributions from the two filters are integrated to estimate vehicle poses. This method exploits advantages of both features and filters, compensating respective weakness to deal with complex urban environments. Testing was performed in the fair and rainy weather. Road test results prove robustness and reliability of the proposed method.

While 3D LiDAR has become a common practice for more and more autonomous driving systems, precise 3D mapping and robust localization is of great importance. However, current 3D map is always noisy and unreliable due to the existence of moving objects, leading to worse localization. In this paper, we propose a general vehicle-free point cloud mapping framework for better on-vehicle localization. For each laser scan, vehicle points are detected, tracked and then removed. Simultaneously, 3D map is reconstructed by registering each vehicle-free laser scan to global coordinate based on GPS/INS data. Instead of direct 3D object detection from point cloud, we first detect vehicles from RGB images using the proposed YVDN. In case of false or missing detection, which may result in the existence of vehicles in the map, we propose the K-Frames forward-backward object tracking algorithm to link detection from neighborhood images. Laser scan points falling into the detected bounding boxes are then removed. We conduct our experiments on the Oxford RobotCar Dataset and show the qualitative results to validate the feasibility of our vehicle-free 3D mapping system. Besides, our vehicle-free mapping system can be generalized to any autonomous driving system equipped with LiDAR, camera and/or GPS.

We present the design and implementation of a flexible force-vision-based interface; allowing local operators to visually specify a path constraint to a remote robot manipulator in an on-line fashion during the teleoperation. Using bilateral and unilateral configurations, we compare our system to direct teleoperation through user studies. Three performance metrics (smoothness, error and execution time) and a subjective evaluation (NASA TLX) were used to quantify user performance. The trials show that our system outperforms direct teleoperation and reduces cognitive load. Our findings show that the performance of a unilateral teleop configuration with visual-force constraints surpass a bilateral teleop configuration in terms of displacement error and variance, as well as allowing users to complete tasks faster and with a smoother trajectory.

Technologies in autonomous vehicles have seen dramatic advances in recent years; however, it still lacks of robust perception systems for car detection. With the recent development in deep learning research, in this paper, we propose a LIDAR and vision fusion system for car detection through the deep learning framework. It consists of three major parts. The first part generates seed proposals for potential car locations in the image by taking LIDAR point cloud into account. The second part refines the location of the proposal boxes by exploring multi-layer information in the proposal network and the last part carries out the final detection task through a detection network which shares part of the layers with the proposal network. The evaluation shows that the proposed framework is able to generate high quality proposal boxes more efficiently (77.6% average recall) and detect the car at the state of the art accuracy (89.4% average precision). With further optimization of the framework structure, it has great potentials to be implemented onto the autonomous vehicle.

This paper introduces a new concept of building soft pneumatic structures by assembling modular units of fabric-based rotary actuators (FRAs) and beams. Upon pressurization, the inner folds of FRA would expand, which causes the FRA module to unfold, generating angular displacement. Hence, FRAs would enable mobility function of the structure and its range of motion. The modular nature of the actuator units enables customized configuration of pneumatic structures, which can be modified and scaled by selecting the appropriate modules. FRAs are also designed to be bladder-less, that is, they are made without an additional layer of inner bladder. Thus, a simple fabrication process can be used to prepare the actuators. In this paper, we studied how the performance of the FRA modules changes with their dimensions and demonstrated how a soft gripper can be constructed using these modules. The kinematic response of the actuator segments of the gripper was analyzed and a pressure control algorithm was developed to regulate the pneumatic pressure of the actuator. The modular based soft robotic gripper alone weighs about 140g. Yet, based on the grip tests, it is able to lift heavier objects (up to 2.4kg), achieving a high payload-to-weight ratio of about 1714%, which is higher than the value reported by previously developed soft pneumatic grippers using elastomeric materials. Lastly, we also demonstrated that the gripper is capable of performing two essential grasping modes, which are power grasping and fingertip grasping, for objects of various shapes.

Silicone-based soft pneumatic actuator (SPA) is one of the key interests in soft robotic research. Currently, most of the SPAs bear a similar one-dimensional rod-like shape, regardless of their design and fabrication. There are few prototypes of SPAs with two-dimensional initial shapes, however, they are basically the linear combination of several one-dimensional SPAs. This paper presents a new class of silicone-based SPA named soft robotic pad (SRP). The SRP is shaped into a soft pad with a single flat air chamber and can generate different kinds of motions according to the internal constraint matrix within the SRP. In this paper, the design and fabrication of our SRPs are elaborated in detail, followed by the preliminary characterization of the different SRP prototypes. Although the current design is not optimized, our SRP prototypes prove the feasibility of the design and fabrication of a 2D shape-morphing SPA. With subsequent optimization, the SRP can work robustly in various real-world applications as strong flat artificial muscles for human joint rehabilitation, or as fins in marine robots.

We present the development of a full-scale “parallel autonomy” research platform including software and hardware. In the parallel autonomy paradigm, the control of the vehicle is shared; the human is still in control of the vehicle, but the autonomy system is always running in the background to prevent accidents. Our holistic approach includes: (1) a drive-by-wire conversion method only based on reverse engineering mounting of relatively inexpensive sensors onto the vehicle implementation of a localization and mapping system, (4) obstacle detection and (5) a shared controller as well as (6) integration with an advanced autonomy simulation system (Drake) for rapid development and testing. The system can operate in three modes: (a) manual driving, (b) full autonomy, where the system is in complete control of the vehicle and (c) parallel autonomy, where the shared controller is implemented. We present results from extensive testing of a full-scale vehicle on closed tracks that demonstrate these capabilities.

In this paper, we describe methods of conveying perception information and motion intention from self driving vehicles to the surrounding environment. One method is by equipping autonomous vehicles with Light-Emitting Diode (LED) strips to convey perception information; typical pedestrian-driver acknowledgement is replaced by visual feedback via lights which change color to signal the presence of obstacles in the surrounding environment. Another method is by broadcasting audio cues of the vehicle's motion intention to the environment. The performance of the autonomous vehicles as social robots is improved by building trust and engagement with interacting pedestrians. The software and hardware systems are detailed, and a video demonstrates the working system in real application. Further extension of the work for multi-class mobility in human environments is discussed.

In this work, we address the problem of robust data association for feature cloud matching. For matching two feature clouds observed at two different poses, we discover that the covariance matrix of the measurement prediction error can be written as the sum of a low rank matrix and a block diagonal matrix, if we assume that the features are observed independently at each pose. This special structure of the covariance matrix allows us to compute its inverse analytically and efficiently. Together with a good bookkeeping strategy, the complexity of the Joint Compatibility (JC) test is reduced to O(1). Contrary to the approximated JC test, ours is both exact and fast. Based on the efficient JC test algorithm and a branch and bound search procedure, we devise an algorithm, called Fast Joint Compatibility Branch and Bound (FastJCBB), to quickly obtain robust data association. The FastJCBB algorithm is essentially modified from the conventional Joint Compatibility Branch and Bound (JCBB) algorithm and both of these algorithms are able to produce exactly the same data association results. However, with the substantial improvement in the efficiency of JC tests, our FastJCBB algorithm is much faster than the conventional JCBB, especially when matching two large feature clouds. It is reported that our FastJCBB algorithm is more than 740 times faster than the conventional JCBB in carrying out one million JC tests when matching two clouds with about 100 features each. Since both FastJCBB and JCBB share the same branch and bound procedure in exploring the interpretation tree, the search complexity remains exponential. Our main contribution is the significant improvement in the efficiency of exploring each node of the interpretation tree.

As autonomous vehicles operating on the urban roads, being conscious of the road context is a crucial prerequisite to safely negotiate with the other vehicles. This paper proposes a probabilistic approach to infer the road context from the vehicle behaviors. Specifically, the consistencies of the randomly-observed vehicle states are extracted first, thereafter the road context is inferred in a probabilistic manner by coupling these consistencies. The feasibility of the proposed road context inference approach has been validated by the case study of an urban road that includes roundabout and T-junction. The experiments demonstrate that the inferred road context can be successfully applied for the autonomous vehicles in various aspects.

We detail the design of autonomous golf cars which were used in public trials in Singapore's Chinese and Japanese Gardens, for the purpose of raising public awareness and gaining user acceptance of autonomous vehicles. The golf cars were designed to be robust, reliable, and safe, while operating under prolonged durations. Considerations that went in to the overall system design included the fact that any member of the public had to not only be able to easily use the system, but to also not have the option to use the system in an unintended manner. This paper details the hardware and software components of the golf cars with these considerations, and also how the booking system and mission planner facilitated users to book for a golf car from any of ten stations within the gardens. We show that the vehicles performed robustly throughout the prolonged operations with a small localization variance, and that users were very receptive from the user survey results.

As autonomous vehicles begin venturing on the urban road, rational decision making is essential for driving safety and efficiency. This paper presents a situation-aware decision making algorithm for autonomous driving on urban road. Specifically, an urban road situation model is proposed first for proper environment representation, thereafter the situation-aware decision making problem is modeled as a Partially Observable Markov Decision Process (POMDP) and solved in an online manner. The proposed algorithm has been extensively evaluated, which is general enough for autonomous driving in various urban road scenarios, including leader following, collision avoidance and traffic negotiation at both T-junction and roundabout.

Vehicle-to-vehicle (V2V) communication enables intention sharing among neighboring vehicles and thereby vehicles' motion can be coordinated to incorporate collision (or conflict) avoidance. In this paper, we propose a general framework to distribute the computational burden for coordinating multiple vehicles' stop-or-go motion. We formulate the multi-vehicle motion coordination problem as a total stopping time minimization problem under the constraint of mutual collision avoidance. The minimal stopping time solution, if such exists, is found using the A* search algorithm in the coordination diagram. The solution is executed efficiently by placing temporary virtual obstacles on the desired path. The communication latency is analyzed for practical applications. The simulations show the correctness and efficacy of our algorithm. An on-road experiment involving two autonomous vehicles, each equipped with V2V communication devices, was performed to demonstrate how deadlocks are successfully avoided.

In this paper, we present a multivehicle cooperative driving system architecture using cooperative perception along with experimental validation. For this goal, we first propose a multimodal cooperative perception system that provides see-through, lifted-seat, satellite and all-around views to drivers. Using the extended range information from the system, we then realize cooperative driving by a see-through forward collision warning, overtaking/lane-changing assistance, and automated hidden obstacle avoidance. We demonstrate the capabilities and features of our system through real-world experiments using four vehicles on the road.

This paper provides sufficient conditions for stability of switched linear systems under dwell-time switching. Piece-wise quadratic functions are utilized to characterize the Lyapunov functions and bilinear matrix inequalities conditions are derived for stability of switched systems. By increasing the number of quadratic functions, a sequence of upper bounds of the minimum dwell time is obtained. Numerical examples suggest that if the number of quadratic functions is sufficiently large, the sequence may converge to the minimum dwell-time.

This paper considers the problem of motion planning for linear systems subject to Gaussian motion noise and proposes a risk-aware planning algorithm: CC-RRT*-D. The proposed CC-RRT*-D employs the chance-constraint approximation and leverages the asymptotically optimal property of RRT* framework to compute risk-aware and asymptotically optimal trajectories. By explicitly considering the state dependence for prior state estimate, the over-conservative problem of chance-constraint approximation can be provably solved. Computational experiment results show that CC-RRT*-D is efficient and robust compared with related algorithms. The real-time experiment on an autonomous vehicle shows that our proposed algorithm is applicable to real-time obstacle avoidance.

This paper proposes a semantic mapping method based on pedestrian activity in the urban road environment. Pedestrian activity patterns are learned from pedestrian tracks collected by a mobile platform. With the learned knowledge of pedestrian activity, semantic mapping is performed using Bayesian classification techniques. The proposed method is tested in real experiments, and shows promising results in recognizing four activity-related semantic properties of the urban road environment: pedestrian path, entrance/exit, pedestrian crossing and sidewalk.

This paper proposes a spatio-temporal motion feature detection and tracking method using range sensors working on a moving platform. The proposed spatio-temporal motion features are similar to optical flow but are extended on a moving platform with fusion of odometry and show much better classification accuracy with consideration of different uncertainties. In the proposal, the ego motion is compensated by odometry sensors and the laser scan points are accumulated and represented as space-time point clouds, from which the velocities and moving directions can be extracted. Based on these spatio-temporal features, a supervised learning technique is applied to classify the points as static or moving and Kalman filters are implemented to track the moving objects. A real experiment is performed during day and night on an autonomous vehicle platform and shows promising results in a crowded and dynamic environment.

This paper presents a precise localization algorithm for vehicles in 3D urban environment with only one 2D LIDAR and odometry information. A novel idea of synthetic 2D LIDAR is proposed to solve the localization problem on a virtual 2D plane. A Monte Carlo Localization scheme is adopted for vehicle position estimation, based on synthetic LIDAR measurements and odometry information. The accuracy and robustness of the proposed algorithm are demonstrated by performing real time localization in a 1.5 km driving test around the NUS campus area.

In this paper, we report a fully automated detailed mapping of a challenging urban environment using single LIDAR. To improve scan matching, extended correlative scan matcher is proposed. Also, a Monte Carlo loop closure detection is implemented to perform place recognition efficiently. Automatic recovery of the pose graph map in the presence of false place recognition is realized through a heuristic based loop closure rejection. This mapping framework is evaluated through experiments on the real world dataset obtained from NUS campus environment.

In this paper, we attempt to develop a reusable framework of cooperative perception for vehicle control on the road that can extend perception range beyond line-of-sight and beyond field-of-view. For this goal, the following problems are addressed: map merging, vehicle identification, sensor multi-modality, impact of communications, and impact on path planning. We provide experimental results using a self-driving vehicle and manned vehicles equipped with the cooperative perception systems that we propose and implement.

This paper presents a method of road detection and mapping using accumulated 3D data from 2D scans. The idea of 3D rolling window is introduced, and its probabilistic characteristics are studied. A cascaded road detection process is developed with region-growing and classification methods. A probabilistic framework is utilized for road mapping purposes with the detection results. The performance of detection and mapping algorithm is evaluated through experiments.

One of the most prominent features on an urban road is the curb, which defines the boundary of a road surface. An intersection is a junction of two or more roads, appearing where no curb exists. The combination of curb and intersection features and their idiosyncrasies carry significant information about the urban road network that can be exploited to improve a vehicle's localization. This paper introduces a Monte Carlo Localization (MCL) method using the curb-intersection features on urban roads. We propose a novel idea of “Virtual LIDAR” to get the measurement models for these features. Under the MCL framework, above road observation is fused with odometry information, which is able to yield precise localization. We implement the system using a single tilted 2D LIDAR on our autonomous test bed and show robust performance in the presence of occlusion from other vehicles and pedestrians.

We present an autonomous vehicle providing mobility-on-demand service in a crowded urban environment. The focus in developing the vehicle has been to attain autonomous driving with minimal sensing and low cost, off-the-shelf sensors to ensure the system's economic viability. The autonomous vehicle has successfully completed over 50 km handling numerous mobility requests during the course of multiple demonstrations. The video provides an overview of our approach, with special comments on our localization and perception modules showcasing one such request being serviced.

We investigate in this paper the problem of misalignment-robust facial expression recognition. To the best of our knowledge, this problem has not been formally addressed in the literature. Most existing facial expression recognition methods, however, can only work well when face images are well-aligned. In many real world applications such as human robot interaction and visual surveillance, it is still very challenging to obtain well-aligned face images for expression recognition due to currently imperfect vision techniques, especially under uncontrolled conditions. Motivated by the fact that interclass facial images with small differences are more easily mis-classified than those with large differences, we propose a biased linear discriminant analysis (BLDA) method by imposing large penalties on interclass samples with small differences and small penalties on those samples with large differences simultaneously, such that more discriminative features can be extracted for recognition. Moreover, we generate more virtually misaligned facial expression samples and assign different weights to them according to their occurrence probabilities in the testing phase to learn a weighted BLDA (WBLDA) feature space to extract misalignment-robust discriminative features for recognition. Experimental results on two widely used face databases are presented to show the efficacy of the proposed method.

This paper investigates the problem of cross-dataset facial expression recognition. To the best of our knowledge, this problem has not been formally addressed in the literature. Conventional facial expression recognition methods assume expression images in the training and testing sets are collected under the same condition such that they are independent and identically distributed. In many real applications, this assumption may not hold as the testing data are usually collected online and generally more uncontrollable than the training data, and hence, they are likely different from the training data. This problem is referred to as cross-dataset facial expression recognition in this paper as the training and testing data are considered to be collected from different datasets due to different acquisition conditions. To address this, we propose a new transfer subspace learning approach to learn a feature space which transfers the knowledge gained from the training set to the target (testing) data to improve the recognition performance under cross-dataset scenarios. Experimental results for facial expression recognition tasks on different datasets are presented to demonstrate the efficacy of the proposed approach.

This paper presents an integrated stochastic approach to tracking multiple live cells that interact with the surrounding gel matrix. Cells migrate in a stochastic manner, forming a functional structure. Tracking the trajectory of each migrating cell and its interactions with the gel and other cells provide useful insights into how a vascular structure is formed as a collection of migratory cells. In micro-fluidic 3-D angiogenic sprouting experiments, two types of images are obtained at discrete time steps using confocal microscopy: a) three-dimensional fluorescent images of stained cell nuclei and b) two dimensional visible light images of the gel matrix. These two sources of images provide supplementary information as the outline of the conduit or lumen formed in the matrix by the migrating cells can be seen in the images of the gel. A Bayesian filtering framework is developed which augments both the cell and conduit parameters to the same state vector, allowing mathematically consistent simultaneous observation updates from both channels. Issues encountered include the high dimensional state vector and non-Gaussian cell state updates. Results demonstrate that our method based on Rao-Blackwellized particle filtering treats these issues effectively.

This paper exploits optimal position of the mobile sensor to improve the target tracking performance of wireless sensor networks and simultaneously localize both of the static sensor nodes and mobile sensor nodes when tracking the human motion. In our approach, mobile sensors collaborate with static sensors and move optimally to achieve the required detection performance. The accuracy of final tracking result is then improved as the measurements of mobile sensors have higher signal-to-noise ratios after the movement. Specifically, we can simultaneously localize the mobile sensor and static sensors position when localizing the human's position based on augmented extended Kaiman filters (EKF). In the algorithm, we develop a sensor movement optimization algorithm that achieves near-optimal system tracking performance. We also presented an sensor nodes management scheme in order to deduce the computation complexity when localizing the static sensor nodes. The effectiveness of our approach is validated by extensive simulations.

Theoretically, the operational space control framework can be regarded to be the most advanced control framework for redundant robots. However, in practice, the control performance of this framework is significantly degraded in the presence of model uncertainties and discretizing effects. Using the singular perturbation theory, this paper shows that the same model uncertainties can create different effects on the task space and joint space control performance. From the analysis, a multi-rate operational space control was proposed to minimize the effects of model uncertainties on the control performance and while maintaining the advantages of the original operational space framework. In this paper, we present a stability analysis of the multi-rate operational space control framework using the Lyapunov's direct method.

We propose a local Voronoi decomposition (LVD) algorithm which is able to perform a robust and online task allocation for multiple agents based purely on local information. Because only local information is required in determining each agent's Voronoi region, each agent can then make its decision in a distributive fashion based on its allocated Voronoi region. These Voronoi regions eliminates the occurrence of agents executing instantaneous overlapping tasks. As our method does not require a pre-processing of the map, it is also able to work well in a dynamically changing map with changing number of agents. We will show our proof of concept in the problem of exploration in an unknown environment. In our experimental evaluation, we show that our method significantly outperforms the competing algorithms: Ants algorithm and the Brick&Mortar algorithm. Our results also show that our method is near the theoretical best solution.

In this paper, we introduce a new multi-rate hybrid motion and force control using operational space framework. Conventional implementation of operational space control (OSC) does not perform well in the presence of model uncertainties. In order to overcome this issue, a two layered hierarchical controller structure is proposed in this paper. In the outer control loop, the control torque from the OSC is applied to the joint space dynamic model of the robot. By integrating the output of this system, reference joint velocities for the next control loop can be obtained. At the inner control loop, the conventional computed-torque control scheme is used to ensure the robot can track the above desired joint velocities. To illustrate the effectiveness of the proposed controller, intensive experiments have been conducted on the Mitsubishi PA10 7-DOF manipulator. Experimental results indicate that there is a significant improvement in comparison to the conventional implementation and the proposed approach is shown to have performed favorably in comparison to the commercial force controller from PushCorp.

We research into using Mobile Sensor Networks to harvest physical quantities that emanate from sources and are distributed in space in hazardous environments. Examples are temperature, toxic emissions and pollutions. Mobile sensors are more advantageous than static sensors in many ways such as easy re-deployment and environmental friendliness. However, they are usually deployed at low node densities with equally spaced nodes. As a result, the reconstructed distribution maps are highly distorted. Our approach attacks the problem from the source, by mobilizing the sensors to harvest data with high information content cooperatively and intelligently. As a result, we achieve 3.6 times in error reduction relative to an equally distributed grid. Moreover, in comparison with the Random Waypoint and Broyden-Fletcher-Goldfarb-Shanno methods, we achieve at least 60% more reduction in energy usage by consuming 70% lesser energy. Finally, our approach has a resource utilization efficiency of 50 times that of static sensors.

Due to the extensive utilization in engineering designs, various vibration controller designs have been investigated to meet the design specifications. However, not all of them exactly meet the design requirements. In this paper, mechatronic design quotient (MDQ) approach and genetic algorithm are coupled together to perform this evaluation and optimization task. MDQ is presented to formulate an evaluation function of passive vibration controller design for flexible beam structures, and GA is then used to optimize this function so as to achieve a design solution with the highest MDQ value. Experimental results from one damper and two dampers are presented and compared. It showed that the linear dampers design with the proposed method can achieve the desired performance.

A parallel force and motion control algorithm using observed velocity is presented in this paper. Experimental results show improved control performance in both force and motion subspaces compared with the same controller using filtered velocity.

Most of the target tracking algorithms proposed for wireless sensor networks (WSNs) so far have been relying on sensors of single modality. To integrate multiple sensing modalities (e.g., by using the proximity sensors and ranging sensors together) to improve the tracking performance, the tracking algorithm shall be capable to deal with non-linear and non-Gaussian nature of the tracking problem. In this paper, we present a particle filter algorithm for multi-modality target tracking in WSNs. Simulation results show that the proposed algorithm can provide a good balance among sensor costs and tracking accuracy.

With our proposed decomposition into layers, a generic framework leading to the reuse of previously produced software and the extraction of useful portions can be achieved. The presented transition function based formalism can be applied to specifying programming frameworks for robot controllers executing very diverse tasks. Formalization introduces rigor into the discussion of the structure of embodied agent controllers. The paper deals with systems consisting of multiple agents executing jobs by: influencing the environment through effectors, gathering information from the environment through sensors and communicating with the other agents through communication channels. A paradigm shift is proposed: from building systems executing jobs, to agents acquiring resources (effectors and receptors) so that they can execute the jobs assigned to them. A supervisory controller coordinates the adequate sequencing of jobs and resolves contentions.

Target tracking among obstacles is an interesting class of motion planning problems that combine the usual motion constraints with robot sensors' visibility constraints. In this paper, we introduce the notion of vantage time and use it to formulate a risk function that evaluates the robot's advantage in maintaining the visibility constraint against the target. Local minimization of the risk function leads to a greedy tracking strategy. We also use simple velocity prediction on the target to further improve tracking performance. We compared our new strategy with earlier work in extensive simulation experiments and obtained much improved results

An operational space dynamic model of a redundantly actuated wheeled mobile robot taking into account the wheel-ground interaction is derived based on vehicle dynamics. The conditions for the robot to avoid slip are derived for both torque and motion. Two torque distribution schemes based on different criteria are proposed. The non-slip condition for motion is utilized to plan slip avoidance paths. The null space joint torques are utilized to fulfil the non-slip condition for torque and avoid actuator torque limits violation. Simulations results are presented to demonstrate the effects of wheel-ground interaction and performance of the proposed torque distribution schemes

This paper describes a distributed layered architecture for resource-constrained multirobot cooperation, which is utilized in autonomic mobile sensor network coverage. In the upper layer, a dynamic task allocation scheme self-organizes the robot coalitions to track efficiently across regions. It uses concepts of ant behavior to self-regulate the regional distributions of robots in proportion to that of the moving targets to be tracked in a nonstationary environment. As a result, the adverse effects of task interference between robots are minimized and network coverage is improved. In the lower task execution layer, the robots use self-organizing neural networks to coordinate their target tracking within a region. Both layers employ self-organization techniques, which exhibit autonomic properties such as self-configuring, self-optimizing, self-healing, and self-protecting. Quantitative comparisons with other tracking strategies such as static sensor placements, potential fields, and auction-based negotiation show that our layered approach can provide better coverage, greater robustness to sensor failures, and greater flexibility to respond to environmental changes

Traditional reinforcement learning algorithms learn based on discrete/finite states and actions, thus limit the learned behaviors to discrete/finite space. To address this problem, this paper introduces a distributed reinforcement learning controller that integrates reinforcement learning with behavior based control networks. This learning controller can enable the robot to generate appropriate control policy which combines different elementary behaviors. In addition, to address the problems in concurrent learning, a distributed learning control algorithm is proposed to coordinate concurrent learning processes. The distributed reinforcement learning controller and learning control algorithm are applied to multi-robot tracking of multiple moving targets. The efficacy is demonstrated by simulations.

Kinematic isotropy of omnidirectional mobile robots with powered caster wheels is discussed and utilized in optimizing the design parameters of the robots. The analysis is done to cover the most general cases, by observing the mathematical expression of the equations of motion rather than by numerical evaluation of specific parameters or actuation schemes. Optimal design guidelines are provided. Singularity analysis is given to support the proposed design guidelines. Simulation results are presented to verify the conclusions drawn.

In this paper, we present a robust observer-based controller (ROC) for robot manipulators to achieve robust velocity estimation and better operational space tracking performance. Without link velocity measurements, the overall ROC system can achieve a semi-global asymptotical stability result for the position and velocity tracking errors, and position and velocity estimation errors. Experimental results using PUMA 560 indicate that the proposed ROC is able to obtain more accurate and less ripple velocity estimation than that obtained from an observer-controller, hence higher tracking performance can be achieved.

Robotics research and development span over five decades but is still not pervasive in our daily lives. This paper attempts to explain why and reviews compliant motion which is a crucial capability required to make robots pervasive. Robotic tasks are analyzed and algorithms for control of compliant motion of manipulators are presented.

Self-organizing feature maps such as extended Kohonen maps (EKMs) have been very successful at learning sensorimotor control for mobile robot tasks. This letter presents a new ensemble approach, cooperative EKMs with indirect mapping, to achieve complex robot motion. An indirect-mapping EKM self-organizes to map from the sensory input space to the motor control space indirectly via a control parameter space. Quantitative evaluation reveals that indirect mapping can provide finer, smoother, and more efficient motion control than does direct mapping by operating in a continuous, rather than discrete, motor control space. It is also shown to outperform basis function neural networks. Furthermore, training its control parameters with recursive least squares enables faster convergence and better performance compared to gradient descent. The cooperation and competition of multiple self-organized EKMs allow a nonholonomic mobile robot to negotiate unforeseen, concave, closely spaced, and dynamic obstacles. Qualitative and quantitative comparisons with neural network ensembles employing weighted sum reveal that our method can achieve more sophisticated motion tasks even though the weighted-sum ensemble approach also operates in continuous motor control space.

A system supporting an interventional radiology (IR) procedure is presented in this paper. A 3D geometric modeling method has been developed to construct a 3D vascular model containing complete connection information of cerebral vasculature for IR planning and intra-operative guidance. The validation result proved the fidelity of the reconstructed model. A magnetic tracking system is employed in this project to trace the tip of the IR device, whose model can be displayed onto a 3D digital model in real time. Magnetic tracking of IR device plus the 3D model can potentially help the radiologist more easily manipulate IR device through vasculature. A method following automatic guidance of IR device (such as catheter and guidewire) from the entry to target is proposed, which can reduce the radiation dose and thus benefit both patients and staff.

This paper describes a reactive, distributed layered architecture for cooperation of multiple resource-bounded robots, which is utilized in mobile sensor network coverage. In the upper layer, a dynamic task allocation scheme self-organizes the robot coalitions to track efficiently in separate regions. It uses the concepts of ant behavior to self-regulate the regional distributions of robots in proportion to that of the targets to be tracked in the changing environment. As a result, the adverse effects of task interference between robots are minimized and sensor network coverage is improved. In the lower layer, the robots use self-organizing neural networks to coordinate their target tracking within a region. Quantitative comparisons with other tracking strategies such as static sensor placements, potential fields, and auction-based negotiation show that our approach can provide better coverage and greater flexibility in responding to environmental changes.

An operational space controller that employs a velocity observer and a friction adaptation law to achieve higher tracking accuracy is presented. Without velocity measurements, the overall observer-controller system can achieve a semi-global asymptotic stability for the position and velocity tracking errors, and position and velocity estimation errors. The estimated friction coefficients can also approach the actual coefficients asymptotically. Experimental results indicate that the proposed adaptive observer-controller is able to achieve higher tracking accuracy than the observer-controller without friction compensation.

Despite the many significant advances made in robot motion research, few works have focused on the tight integration of high-level deliberative planning with reactive control at the lowest level. In particular, the real-time performance of existing integrated planning and control architectures is still not optimal because the reactive control capabilities have not been fully realized. This paper aims to enhance the low-level reactive capabilities of integrated planning and control with Cooperative Extended Kohonen Maps for handling complex, unpredictable environments so that the work-load of the high-level planner can be consequently eased. The enhancements include fine, smooth motion control, execution of more complex motion tasks such as overcoming unforeseen concave obstacles and traversing between closely spaced obstacles, and asynchronous execution of behaviors.

This paper presents an operational space output feedback controller for non-redundant robot manipulators to achieve trajectory tracking without velocity measurements. The overall system can achieve a semi-global exponential stability (SGES) result for the position, orientation and velocity tracking errors as well as velocity observation errors. Experimental results of the proposed controller indicate good position and orientation tracking performance under parametric uncertainty and payload variations.

The Operational Space Formulation provides a framework for the analysis and control of manipulator systems with respect to the behavior of their end-effectors. Its application to aircraft canopy polishing is shown using a mobile manipulator. The mobile manipulator end-effector maintains a desired force normal to the canopy surface of unknown geometry in doing a compliant polishing motion, while, at the same time, its mobile base moves around the shop floor, effectively increasing the mobile manipulator's workspace. The mobile manipulator consists of a PUMA 560 mounted on top of a Nomad XR4000. Implementation issues are discussed and simultaneous motion and force regulation results are shown.

Despite the many significant advances made in robotics research, few works have focused on the tight integration of task planning and motion control. Most integration works involve the task planner providing discrete commands to the low-level controller, which performs kinematics and control computations to command the motor and joint actuators. This paper presents a framework of the integrated planning and control for mobile robot navigation. Unlike existing integrated approaches, it produces a sequence of checkpoints instead of a complete path at the planning level. At the motion control level, a neural network is trained to perform motor control that moves the robot from one checkpoint to the next. This method allows for a tight integration between high-level planning and low-level control, which permits real-time performance and easy modification of motion path while the robot is enroute to the goal position.

A singularity handling method is proposed in this paper. It is done by introducing virtual redundant joints into the Jacobian matrix to maintain the rank of the Jacobian matrix when singularity occurs. These additional joints do not exist physically. Therefore, although mathematically stable, the manipulator still cannot perform tasks in the degenerate direction(s). This method is comparatively straight forward to implement and it does not have a singular subspace defined within which a special and different control algorithm is performed, thus it avoids the problem associated with discontinuous control or switching of control. The method was tested on simulation and implemented in real-time on the PUMA 560 robot.

Different autofocusing methods exist for many cameras today. While not ignoring commercially available methods requiring specialized hardware, this paper focuses mainly on pixel based autofocusing algorithms as applied to CCD camera systems. Different measures of image sharpness are compared. For each of these, different algorithms for searching the best lens setting are assessed in terms of performance as well as their applicability to various situations. In addition, several other factors potentially affecting camera focusing are also discussed. Based on the information obtained, this research attempts to formulate a robust autofocusing algorithm.

Robot programming is a very tedious task that involves defining the relevant robot positions and configurations, and writing the robot program to execute the task. In this paper, we present an application of the control of the dynamic behavior of robotic manipulators to achieve walk-through programming of a powered arm. The algorithm is based on impedance control with zero stiffness to allow the robot to be moved by the human hand during walk-through teaching. We demonstrate the application using a welding robot we have customized and improved on for use in shipyards. We describe the architecture, our methodologies, the practical issues and present the results of our performance evaluation.

Presents a navigation algorithm, which integrates a virtual obstacle concept with a potential-field-based method to manoeuvre cylindrical mobile robots in unknown or unstructured environments. This study focuses on the real-time feature of the navigation algorithm for fast moving mobile robots. We mainly consider the potential-field method in conjunction with virtual obstacle concept as the basis of our navigation algorithm. Simulation and experiments of our algorithm shows good performance and ability to overcome the local minimum problem associated with potential field methods.

A neural network controller for constrained robot manipulators is presented. A feedforward neural network is used to adaptively compensate for the uncertainties in the robot dynamics. Training signals are proposed for the feedforward neural network controller. The neural network weights are tuned online, with no online learning phase required. It is shown that the controller is able to deal with the uncertainties of the robot which include modelled undertainties (dynamic parameter uncertainties, etc.) as well as unmodelled uncertainties (frictions, etc.). The suggested controller is simple in structure and can be implemented easily. The controller has the proportional-integral (PI) type force feedback control structure with a low proportional force feedback gain. Detailed experimental results show the effectiveness of the proposed controller.

Output redefinition is proposed for a flexible-link robot so that the transfer function of the system is minimum phase and this facilitates the design of trajectory tracking controllers. We show that the zero-dynamics is exponentially stable with the newly defined output function. Asymptotic tracking of step, linear and second order polynomial trajectories are achieved using controllers designed based on this new technique and the link vibrations are damped out significantly. The unique feature of the controller design technique is that the poles of the zero-dynamics can be placed at any desired locations in the left half of the s-plane. This enables us to suppress undesirable link vibrations well while the robot tip tracks a prescribed tip-trajectory.

Controller design for multi-link flexible robots taking into consideration control input constraints is presented. The control law is obtained as a solution that minimizes a certain performance index. A result on quadratic programming is used to synthesize the controller and the controller can be implemented online. The feedback controller guarantees that the control input constraints are not violated while joint-angle regulation and vibration damping are achieved. Local asymptotic stability of the closed-loop system is shown. Simulation results of a two-link flexible robot are presented in order to demonstrate the feasibility and good performance of the controller.

Command shaping is an important method to reduce vibration in flexible link robots. This paper presents a very simple rest-to-rest motion control command which eliminates multiple mode residual vibration in a flexible link robot in finite time. The command is constructed by solving linear equations. The finite time duration in which the desired motion of the joint angle is achieved along with elimination of the residual vibration can be arbitrarily specified, which offers the option for trade-offs among various considerations. The necessary conditions for using the command as a reference input for the joint angle in a closed-loop configuration (using a PD controller) are also discussed.

Constrained motion tasks involve interaction forces between the tool and the work-piece and demand certain amounts of compliance at either the tool or at the work-piece. We present a scheme to actively control the necessary compliance of the tool attached to the robot. This scheme allows the operator to specify the center of compliance and the three lateral and three angular compliances. The scheme uses a wrist-mounted, 6 axis force-torque sensor to measure the interaction forces. The necessary compliance is achieved by modifying the motion of the manipulator. This is achieved by programming a force-deflection relationship at the tool tip. This paper discusses the details of the algorithm, development of the control architecture for implementing the proposed scheme on an industrial robot, and real-time experimental results.

We explore the possibility of achieving passive compliance through the structure of the manipulator itself. The emphasis is on passive compliance because a minimum of passive compliance to prevent jamming will always be required even when active stiffness control is employed. Particular attention is given to the large class of robots with nonbackdrivable actuators, where the actuator must be commanded to move, and in which actuator forces or torques are not easily interpreted as end-effector forces and torques. We present a novel framework for specifying the desired end-effector compliance for several tasks in terms of stiffness matrices. We explore whether the desired stiffness matrix of a manipulator can be achieved by using the natural or designed stiffness of the manipulator limbs themselves. Several techniques for adjusting the manipulator stiffness matrix are proposed. Achieving this variable passive compliance allows the attainment of high stiffnesses for fast and accurate movements and low stiffness values for force control. Furthermore, achieving nondiagonal stiffness properties wherein there are force and motion coupling in different directions is shown to be useful to prevent jamming and contact induced vibrations.>

The concept of multiple interchangeable wrist mechanisms is introduced as a cost-effective, practical method to improve the flexibility of a robot system without increasing the complexity of the wrist design. The interchangeable wrist concept is a natural extension of interchangeable tooling. The feasibility of the proposed concept for real-time applications is demonstrated by an analytical framework that incorporates the wrist-exchange procedure in the robot control hierarchy for the case of manipulators with interchangeable spherical and nonspherical wrists. An algorithmic approach that utilizes the individual kinematic properties of the arm and the wrists is then developed to generate an efficient solution to the kinematic control problem of the coupled system. The accuracy and computational efficiency of the proposed algorithm are shown through a numerical example for the PUMA 560 robot.>

A modular architecture for general-purpose inverse robot kinematics is developed. The authors synthesize kinematic modules for the robot arm and wrist and develop computational blocks to describe their respective functions. They then present an analytical framework that defines the inverse kinematic problem in terms of the proper coordination of the kinematic modules to accomplish the desired robot task. In this general-purpose framework, the inverse kinematics problem is always solvable in the feasible regions of the robot workspace, irrespective of whether the solution is analytically tractable. The modular architecture is based upon a nonlinear equation solver for which the Banach fixed-point theorem provides the theoretical basis. The proposed framework allows for the mathematical definition of the region in the robot workspace where convergence to the correct solution is guaranteed. It is insensitive to the initial estimates and provides for the computation of multiple solutions.>