Navigating a nonholonomic robot in a cluttered, unknown environment requires accurate perception and precise motion control for real-time collision avoidance. This article presents neural proximal alternating-minimization network (NeuPAN): a real-time, highly accurate, map-free, easy-to-deploy, and environment-invariant robot motion planner. Leveraging a tightly coupled perception-to-control framework, NeuPAN has two key innovations compared to existing approaches: first, it directly maps raw point cloud data to a latent distance feature space for collision-free motion generation, avoiding error propagation from the perception to control pipeline; second, it is interpretable from an end-to-end model-based learning perspective. The crux of NeuPAN is solving an end-to-end mathematical model with numerous point-level constraints using a plug-and-play proximal alternating-minimization network, incorporating neurons in the loop. This allows NeuPAN to generate real-time, physically interpretable motions. It seamlessly integrates data and knowledge engines, and its network parameters can be fine-tuned via back propagation. We evaluate NeuPAN on a ground mobile robot, a wheel-legged robot, and an autonomous vehicle, in extensive simulated and real-world environments. Results demonstrate that NeuPAN outperforms existing baselines in terms of accuracy, efficiency, robustness, and generalization capabilities across various environments, including the cluttered sandbox, office, corridor, and parking lot. We show that NeuPAN works well in unknown and unstructured environments with arbitrarily shaped objects, transforming impassable paths into passable ones.

Compared with real-time multi-object tracking (MOT), offline multi-object tracking (OMOT) has the advantages to perform 2D-3D detection fusion, erroneous link correction, and full track optimization but has to deal with the challenges from bounding box misalignment and track evaluation, editing, and refinement. This paper proposes “BiTrack”, a 3D OMOT framework that includes modules of 2D-3D detection fusion, initial trajectory generation, and bidirectional trajectory re-optimization to achieve optimal tracking results from camera-LiDAR data. The novelty of this paper includes threefold: (1) development of a point-level object registration technique that employs a density-based similarity metric to achieve accurate fusion of 2D-3D detection results; (2) development of a set of data association and track management skills that utilizes a vertex-based similarity metric as well as false alarm rejection and track recovery mechanisms to generate reliable bidirectional object trajectories; (3) development of a trajectory re-optimization scheme that re-organizes track fragments of different fidelities in a greedy fashion, as well as refines each trajectory with completion and smoothing techniques. The experiment results on the KITTI dataset demonstrate that BiTrack achieves the state-of-the-art performance for 3D OMOT tasks in terms of accuracy and efficiency.

Robust and accurate perception and prediction of the driving scenarios are crucial for autonomous driving vehicles (ADV). State-of-the-art ADV frameworks have evolved from conventional modular design to an end-to-end (E2E) pipeline that enables joint feature learning and optimization. However, the evaluation of uncertainties in the intermediate features propagated between perception and prediction units is missing in current E2E pipelines. Consequently, adverse and extreme environment factors may incur highly untrustworthy features that ultimately result in degraded perception and prediction. In this work, we propose a novel uncertainty-aware E2E visual perception and prediction framework that utilized Bird's Eye View (BEV) representations. A feature distribution estimation network is introduced to explicitly quantify the uncertainties in the intermediate BEV features extracted from the images. To better exploit temporal information and generate more robust features for scene prediction, an uncertainty-aware transformer is designed to utilize the guidance of the quantified feature uncertainty via the attention mechanism. In addition, an evidential decoder generates accurate future instance segmentations along with the associated uncertainties. Comprehensive experiments conducted on real-world dataset validate the superiority of our proposed framework over conventional pipelines. Codes are available at: https://github.com/Huang121381/UAPnP.

Novel view image synthesis for large-scale outdoor traffic scenes presents significant challenges, including inaccurate depth measurements, moving objects, wide-angle rendering requirements, and the increased demand for memory and computational resources. In this paper, we propose an adaptive pipeline that constructs high-fidelity 3D surfel models and synthesizes realistic novel views in real time. Our contributions are threefold: 1) developing depth-refinement and moving-object-removal techniques to robustly reconstruct surfel-based scene geometry, while minimizing computational overhead; 2) developing a self-adaptive rendering mechanism which adjusts surfel geometry for large-scale scenes within constrained memory; 3) developing a hyper-parameter tuning approach for optimal surfel construction and rendering performance. An optional GAN-based inpainting module fills missing backgrounds (e.g., sky). Experiments on the KITTI dataset and CARLA simulator show that our method achieves image quality comparable to SOTA NeRF and 3D Gaussian Splatting techniques with significantly improved computational efficiency. This makes our approach particularly well-suited for large-scale traffic scenarios. Our simulation datasets with ground-truth data and source code are available at https://github.com/Billy1203/SurfelMapping.

Most autonomous driving (AD) datasets incur substantial costs for collection and labeling, inevitably yielding a plethora of low-quality and redundant data instances, thereby compromising performance and efficiency. Many applications in AD systems necessitate high-quality training datasets using both existing datasets and newly collected data. In this paper, we propose a traffic scene joint active learning (TSceneJAL) framework that can efficiently sample the balanced, diverse, and complex traffic scenes from both labeled and unlabeled data. The novelty of this framework is threefold: 1) a scene sampling scheme based on a category entropy, to identify scenes containing multiple object classes, thus mitigating class imbalance for the active learner; 2) a similarity sampling scheme, estimated through the directed graph representation and a marginalize kernel algorithm, to pick sparse and diverse scenes; 3) an uncertainty sampling scheme, predicted by a mixture density network, to select instances with the most unclear or complex regression outcomes for the learner. Finally, the integration of these three schemes in a joint selection strategy yields an optimal and valuable subdataset. Experiments on the KITTI, Lyft, nuScenes and SUScape datasets demonstrate that our approach outperforms existing state-of-the-art methods on 3D object detection tasks with up to 12% improvements.

Navigating a nonholonomic robot in a cluttered, unknown environment requires accurate perception and precise motion control for real-time collision avoidance. This article presents neural proximal alternating-minimization network (NeuPAN): a real-time, highly accurate, map-free, easy-to-deploy, and environment-invariant robot motion planner. Leveraging a tightly coupled perception-to-control framework, NeuPAN has two key innovations compared to existing approaches: first, it directly maps raw point cloud data to a latent distance feature space for collision-free motion generation, avoiding error propagation from the perception to control pipeline; second, it is interpretable from an end-to-end model-based learning perspective. The crux of NeuPAN is solving an end-to-end mathematical model with numerous point-level constraints using a plug-and-play proximal alternating-minimization network, incorporating neurons in the loop. This allows NeuPAN to generate real-time, physically interpretable motions. It seamlessly integrates data and knowledge engines, and its network parameters can be fine-tuned via backpropagation. We evaluate NeuPAN on a ground mobile robot, a wheel-legged robot, and an autonomous vehicle, in extensive simulated and real-world environments. Results demonstrate that NeuPAN outperforms existing baselines in terms of accuracy, efficiency, robustness, and generalization capabilities across various environments, including the cluttered sandbox, office, corridor, and parking lot. We show that NeuPAN works well in unknown and unstructured environments with arbitrarily shaped objects, transforming impassable paths into passable ones.

Simulation data can be accurately labeled and have been expected to improve the performance of data-driven algorithms, including object detection. However, due to the various domain inconsistencies from simulation to reality (sim-to-real), cross-domain object detection algorithms usually suffer from dramatic performance drops. While numerous unsupervised domain adaptation (UDA) methods have been developed to address cross-domain tasks between real-world datasets, progress in sim-to-real remains limited. This paper presents a novel Complex-to-Simple (CTS) framework to transfer models from labeled simulation (source) to unlabeled reality (target) domains. Based on a two-stage detector, the novelty of this work is threefold: 1) developing fixed-size anchor heads and RoI augmentation to address size bias and feature diversity between two domains, thereby improving the quality of pseudo-label; 2) developing a novel corner-format representation of aleatoric uncertainty (AU) for the bounding box, to uniformly quantify pseudo-label quality; 3) developing a noise-aware mean teacher domain adaptation method based on AU, as well as object-level and frame-level sampling strategies, to migrate the impact of noisy labels. Experimental results demonstrate that our proposed approach significantly enhances the sim-to-real domain adaptation capability of 3D object detection models, outperforming state-of-the-art cross-domain algorithms, which are usually developed for real-to-real UDA tasks.

Human pose estimation (HPE), particularly multiperson pose estimation (MPPE), has been applied in many domains, such as human–machine systems. However, the current MPPE methods generally run on powerful GPU systems and take a lot of computational costs. Real-time MPPE on mobile devices with low-performance computing is a challenging task. In this article, we propose a lightweight neural network, DIR-BHRNet, for real-time MPPE on smartphones. In DIR-BHRNet, we design a novel lightweight convolutional module, dense inverted residual (DIR), to improve accuracy by adding a depthwise convolution and a shortcut connection into the well-known inverted residual, and a novel efficient neural network structure, balanced HRNet (BHRNet), to reduce computational costs by reconfiguring the proper number of convolutional blocks on each branch. We evaluate DIR-BHRNet on the well-known COCO and CrowdPose datasets. The results show that DIR-BHRNet outperforms the state-of-the-art methods in terms of accuracy with a real-time computational cost. Finally, we implement the DIR-BHRNet on the current mainstream Android smartphones, which perform more than 10 FPS. The free-used executable file (Android 10), source code, and a video description of this work are publicly available on the page1to facilitate the development of real-time MPPE on smartphones.

The active scene flow estimation technology aims at higher quality of scene flow data through actively changing the ego-vehicle’s trajectory as well as sensor observation position. The major challenges for active 3D point cloud scene flow estimation for autonomous driving (AD) include mis-registrations of point cloud frames, inaccurate scene prediction, and inefficient scene similarity evaluation. This paper presents a framework of active scene flow estimation for AD applications, which consists of three main modules: robust scene flow estimation, reliable reachable area detection, and efficient observation position decision. The novelty of this work is threefold: 1) developing a robust bi-direction attention-based mechanism neural network scene flow estimation method; 2) developing a reliable reachable area detection strategy through hidden points removing (HPR) based scene prediction and a legality checking scheme between the ego-vehicle and the road as well as the predicted scene; 3) developing an efficient observation position decision strategy through building a scene similarity measure, which can help evaluate the differences between two frames of point clouds from different views. The proposed method is trained and verified using a CARLA simulator based dataset, the FlyingThings3D and KiTTI datasets, which have the accurate scene flow ground-truth. The experiment results demonstrate the superior estimation performance and generalization capacity of our method for various AD scenes as well as different system configurations, compared with other state-of-the-art (SOTA) methods.

Human pose estimation (HPE) has attracted a significant amount of attention from the computer vision community in the past decades. Moreover, HPE has been applied to various domains, such as human–computer interaction, sports analysis, and human tracking via images and videos. Recently, deep learning-based approaches have shown state-of-the-art performance in HPE-based applications. Although deep learning-based approaches have achieved remarkable performance in HPE, a comprehensive review of deep learning-based HPE methods remains lacking in literature. In this article, we provide an up-to-date and in-depth overview of the deep learning approaches in vision-based HPE. We summarize these methods of 2-D and 3-D HPE, and their applications, discuss the challenges and the research trends through bibliometrics, and provide insightful recommendations for future research. This article provides a meaningful overview as introductory material for beginners to deep learning-based HPE, as well as supplementary material for advanced researchers.

Providing safety guarantees for Autonomous Vehicle (AV) systems with machine-learning based controllers remains a challenging issue. In this work, we propose Simplex-Drive, a framework that can achieve runtime safety assurance for machine-learning enabled controllers of AVs. The proposed Simplex-Drive consists of an unverified Deep Reinforcement Learning (DRL)-based advanced controller (AC) that achieves desirable performance in complex scenarios, a Velocity-Obstacle (VO) based baseline safe controller (BC) with provably safety guarantees, and a verified mode management unit that monitors the operation status and switches the control authority between AC and BC based on safety-related conditions. We provide a formal correctness proof of Simplex-Drive and conduct a lane-changing case study in dense traffic scenarios. The simulation experiment results demonstrate that Simplex-Drive can always ensure the operation safety without sacrificing control performance, even if the DRL policy may lead to deviations from the safe status.

2D&3D object detection always suffers from a dramatic performance drop when transferring the model trained in the source domain to the target domain due to various domain shifts. In this paper, we propose a Joint Self-Training (JST) framework to improve 2D image and 3D point cloud detectors with aligned outputs simultaneously during the transferring. The proposed framework contains three novelties to overcome object biases and unstable self-training processes: 1) an anchor scaling scheme is developed to efficiently eliminate the object size biases without any modification on point clouds; 2) a 2D&3D bounding box alignment method is proposed to generate high-quality pseudo labels for the self-training process; 3) a model smoothing based training strategy is developed to reduce the training oscillation properly. Experiment results show that the proposed approach improves the performance of 2D and 3D detectors in the target domain simultaneously; especially the superior accuracy of 3D detection can be achieved on benchmark datasets over the state-of-the-art methods.

To ensure real-time response to passengers, existing solutions to the vehicle dispatch problem typically optimize dispatch policies using small batch windows and ignore the spatial-temporal dynamics over the long-term horizon. In this paper, we focus on improving the long-term performance of ride-sharing services and propose a deep reinforcement learning based approach for the ride-sharing dispatch problem. In particular, this work includes: (1) an offline policy evaluation (OPE) based method to learn a value function that indicates the expected reward of a vehicle reaching a particular state; (2) an online learning procedure to update the offline trained value function to capture the real-time dynamics during the operation; (3) an efficient online dispatch method that optimizes the matching policy by considering both past and future influences. Extensive simulations are conducted based on New York City taxi data, and show that the proposed solution further increases the service rate compared to the state-of-the-art farsighted ride-sharing dispatch approach.

The major challenges of collision avoidance for robot navigation in crowded scenes lie in accurate environment modeling, fast perceptions, and trustworthy motion planning policies. This paper presents a novel adaptive environment model based collision avoidance reinforcement learning (i.e., AEMCARL) framework for an unmanned robot to achieve collision-free motions in challenging navigation scenarios. The novelty of this work is threefold: (1) developing a hierarchical network of gated-recurrent-unit (GRU) for environment modeling; (2) developing an adaptive perception mechanism with an attention module; (3) developing an adaptive reward function for the reinforcement learning (RL) framework to jointly train the environment model, perception function and motion planning policy. The proposed method is tested with the Gym-Gazebo simulator and a group of robots (Husky and Turtlebot) under various crowded scenes. Both simulation and experimental results have demonstrated the superior performance of the proposed method over baseline methods.

This paper considers active SLAM problem for 3D deformable environments where the trajectory of the robot is planned to optimize the SLAM results. A planning strategy combining an efficient global planner with an accurate local planner is proposed to solve the problem. Simulation results under different scenarios have shown that the proposed active SLAM algorithm provides a good balance between accuracy and efficiency as compared to the local planner and the global planner. The MATLAB code of this first active SLAM algorithm for 3D deformable environments is made publicly available4.

Edge federated learning (FL) is an emerging paradigm that trains a global parametric model from distributed datasets based on wireless communications. This paper proposes a unit-modulus over-the-air computation (UMAirComp) framework to facilitate efficient edge federated learning, which simultaneously uploads local model parameters and updates global model parameters via analog beamforming. The proposed framework avoids sophisticated baseband signal processing, leading to low communication delays and implementation costs. Training loss bounds of UMAirComp FL systems are derived and two low-complexity large-scale optimization algorithms, termed penalty alternating minimization (PAM) and accelerated gradient projection (AGP), are proposed to minimize the nonconvex nonsmooth loss bound. Simulation results show that the proposed UMAirComp framework with PAM algorithm achieves a smaller mean square error of model parameters’ estimation, training loss, and test error compared with other benchmark schemes. Moreover, the proposed UMAirComp framework with AGP algorithm achieves satisfactory performance while reduces the computational complexity by orders of magnitude compared with existing optimization algorithms. Finally, we demonstrate the implementation of UMAirComp in a vehicle-to-everything autonomous driving simulation platform. It is found that autonomous driving tasks are more sensitive to model parameter errors than other tasks since the neural networks for autonomous driving contain sparser model parameters.

Wireless federated learning (FL) is an emerging machine learning paradigm that trains a global parametric model from distributed datasets via wireless communications. This paper proposes a unit-modulus wireless FL (UMWFL) framework, which simultaneously uploads local model parameters and computes global model parameters via optimized phase shifting. The proposed framework avoids sophisticated baseband signal processing, leading to both low communication delays and implementation costs. A training loss bound is derived and a penalty alternating minimization (PAM) algorithm is proposed to minimize the nonconvex nonsmooth loss bound. Experimental results in the Car Learning to Act (CARLA) platform show that the proposed UMWFL framework with PAM algorithm achieves smaller training losses and testing errors than those of the benchmark scheme.

Shared autonomous mobility-on-demand systems hold great promise for improving the efficiency of urban transportation, but are challenging to implement due to the huge scheduling search space and highly dynamic nature of requests. This paper presents a novel optimal schedule pool (OSP) assignment approach to optimally dispatch high-capacity ride-sharing vehicles in real time, including: (1) an incremental search algorithm that can efficiently compute the exact lowest-cost schedule of a ride-sharing trip with a reduced search space; (2) an iterative online re-optimization strategy to dynamically alter the assignment policy for new incoming requests, in order to maximize the service rate. Experimental results based on New York City taxi data show that our proposed approach outperforms the state-of-the-art in terms of service rate and system scalability.

Right invariant extended Kalman filter (RIEKF) based simultaneous localization and mapping (SLAM) proposed recently has shown to be able to produce more consistent SLAM estimates as compared with traditional EKF based SLAM methods, including some improved EKF SLAM methods such as observability constrained-EKF (OC-EKF) SLAM. Latest results have demonstrated that its performance is very close to optimization based SLAM algorithms such as iSAM. In this paper, we propose to use RIEKF SLAM algorithm in active SLAM where both the predicted SLAM results for choosing control actions and the actual estimated SLAM results applying the selected control actions are computed using RIEKF algorithms. The advantages over traditional EKF based active SLAM are the more accurate and consistent predicted uncertainty estimates which result in robustness of the active SLAM algorithm. The advantages over optimization based active SLAM is the reduced computational cost. Simulation results are presented to validate the advantages of the proposed algorithm3.

The technology of dynamic map fusion among networked vehicles has been developed to enlarge sensing ranges and improve sensing accuracies for individual vehicles. This paper proposes a federated learning (FL) based dynamic map fusion framework to achieve high map quality despite unknown numbers of objects in fields of view (FoVs), various sensing and model uncertainties, and missing data labels for online learning. The novelty of this work is threefold: (1) developing a three-stage fusion scheme to predict the number of objects effectively and to fuse multiple local maps with fidelity scores; (2) developing an FL algorithm which fine-tunes feature models (i.e., representation learning networks for feature extraction) distributively by aggregating model parameters; (3) developing a knowledge distillation method to generate FL training labels when data labels are unavailable. The proposed framework is implemented in the Car Learning to Act (CARLA) simulation platform. Extensive experimental results are provided to verify the superior performance and robustness of the developed map fusion and FL schemes.

In colonoscopy procedures, it is important to rebuild and visualize the colonic surface to minimize the missing regions and reinspect for abnormalities. Due to the fast camera motion and deformation of the colon in standard forward-viewing colonoscopies, traditional simultaneous localization and mapping (SLAM) systems work poorly for 3D reconstruction of colon surfaces and are prone to severe drift. Thus in this paper, a preoperative colon model segmented from CT scans is used together with the colonoscopic images to achieve the 3D colon reconstruction. The proposed framework includes dense depth estimation from monocular colonoscopic images using a deep neural network (DNN), visual odometry (VO) based camera motion estimation and an embedded deformation (ED) graph based non-rigid registration algorithm for deforming 3D scans to the segmented colon model. A realistic simulator is used to generate different simulation datasets with ground truth. Simulation results demonstrate the good performance of the proposed 3D colonic surface reconstruction method in terms of accuracy and robustness. In-vivo experiments are also conducted and the results show the practicality of the proposed framework for providing useful shape and texture information in colonoscopy applications.

Multi-object tracking (MOT) with camera-LiDAR fusion demands accurate results of object detection, affinity computation and data association in real time. This paper presents an efficient multi-modal MOT framework with online joint detection and tracking schemes and robust data association for autonomous driving applications. The novelty of this work includes: (1) development of an end-to-end deep neural network for joint object detection and correlation using 2D and 3D measurements; (2) development of a robust affinity computation module to compute occlusion-aware appearance and motion affinities in 3D space; (3) development of a comprehensive data association module for joint optimization among detection confidences, affinities and start-end probabilities. The experiment results on the KITTI tracking benchmark demonstrate the superior performance of the proposed method in terms of both tracking accuracy and processing speed.

On-demand ride-sharing is a promising way to improve mobility efficiency and reliability. The quality of passenger experience and the profit achieved by these platforms are strongly affected by the vehicle dispatch policy. However, existing ride-sharing research seldom considers travel time uncertainty, which leads to inaccurate dispatch allocations. This paper proposes a framework for dynamic vehicle dispatch that leverages stochastic travel time models to improve the performance of a fleet of shared vehicles. The novelty of this work includes: (1) a stochastic on-demand ride-sharing scheme to maximize the service rate (percentage of requests served) and reliability (probability of on-time arrival); (2) a technique based on approximate stochastic shortest path algorithms to compute the reliability for a ride-sharing trip; (3) a method to maximize the profit when a penalty for late arrivals is introduced. Based on New York City taxi data, it is shown that by considering travel time uncertainty, ride-sharing service achieves higher service rate, reliability and profit.

Full body scanning plays an important role in automated industrial manufacture and inspection. It requires the fusion of multi-view point cloud data and consumes large computational resources when the geometry of corresponding point clouds are unknown. Structured Light Illumination (SLI) is one of the most promising indoor 3D imaging techniques, but also has the same weakness for fusing the multi-view point clouds. This work proposes a mobile SLI system to alleviate this problem for full body 3D scanning. We derive the geometric relation between the phase and the motion of the mobile system, and develop an optimization approach to estimate the pose by comparing the phase image pair before and after the motion. Further more, a graph-based Simultaneous Localization And Mapping (SLAM) framework is built to improve the global accuracy of the pose estimation. By using the 2D phase comparison, the proposed method is more accurate than the normal 2D image feature point matching, and has lower computational complexity than the 3D point registration. The proposed system is experimented in both 3D Simulator and real environment. The yielded full body scan results demonstrated its higher accuracy and more efficiency than the current methods.

While machine-type communication (MTC) devices generate massive data, they often cannot process this data due to limited energy and computation power. To this end, edge intelligence has been proposed, which collects distributed data and performs machine learning at the edge. However, this paradigm needs to maximize the learning performance instead of the communication throughput, for which the celebrated water-filling and max-min fairness algorithms become inefficient since they allocate resources merely according to the quality of wireless channels. This paper proposes a learning centric power allocation (LCPA) method, which allocates radio resources based on an empirical classification error model. To get insights into LCPA, an asymptotic optimal solution is derived. The solution shows that the transmit powers are inversely proportional to the channel gain, and scale exponentially with the learning parameters. Experimental results show that the proposed LCPA algorithm significantly outperforms other power allocation algorithms.

The challenges of multi-robot navigation in dynamic environments lie in uncertainties in obstacle complexities, partially observation of robots, and policy implementation from simulations to the real world. This paper presents a cooperative approach to address the multi-robot navigation problem (MRNP) under dynamic environments using a deep reinforcement learning (DRL) framework, which can help multiple robots jointly achieve optimal paths despite a certain degree of obstacle complexities. The novelty of this work includes threefold: (1) developing a cooperative architecture that robots can exchange information with each other to select the optimal target locations; (2) developing a DRL based framework which can learn a navigation policy to generate the optimal paths for multiple robots; (3) developing a training mechanism based on dynamics randomization which can make the policy generalized and achieve the maximum performance in the real world. The method is tested with Gazebo simulations and 4 differential drive robots. Both simulation and experiment results validate the superior performance of the proposed method in terms of success rate and travel time when compared with the other state-of-art technologies.

The challenges of developing low-cost, large-scale multi-robot navigation systems include noisy measurements, a large number of robots, and computing efficiency for collision avoidance. This paper presents a distributed motion planning framework for a large number of robots to navigate with robust collision avoidance using low-cost range only measurements. The novelty of this work is threefold. (1) Developing a distributed collision-free navigation system for a large-scale robot group in which each robot performs motion planning based on the noisy range measurements of neighboring robots; (2) Developing a set of algorithms for each robot to accurately estimate the relative positions and orientations based on the range measurements and relative velocities; (3) Developing a velocity obstacle (VO) based motion planning algorithm for each robot which can take into account of the estimation uncertainties in relative positions and orientations. The proposed approach is tested with various numbers of differential-driven robots in the Gazebo simulator and real-world experiments. Both simulation and experiment results validate the superior performance of the proposed approach compared to other state-of-art technologies.

The major challenges of developing 3D point cloud annotation systems for autonomous driving datasets include convenient user-data interfaces, efficient operations on geometric data units, and scalable annotation tools. This paper presents a Portable pOint-cloud Interactive aNnotation plaTform System (i.e. SUSTech POINTS), which contains a set of user-friendly interfaces and efficient annotation tools to help achieve high-quality data annotations with high efficiency. The novelty of this work is threefold: (1) developing a set of visualization modules for fast annotation error localization and convenient annotator-data interactions; (2) developing a set of interactive tools for annotators labeling 3D point clouds and 2D images in high speed; (3) developing an annotation transfer method to label the same objects in different data frames. The developed POINTS system is tested with public datasets such as KITTI and a private dataset (SUSTech SCAPES). The experimental results show that the developed platform can help improve the annotation accuracy and efficiency compared with using other open-source annotation platforms.

Massive multiple-input multiple-output communications can achieve high-level security by concentrating radio frequency signals towards the legitimate users. However, this system is vulnerable in a Rician fading environment if the eavesdropper positions itself such that its channel is highly “similar” to the channel of a legitimate user. To address this problem, this paper proposes an angle aware user cooperation (AAUC) scheme, which avoids direct transmission to the attacked user and relies on other users for cooperative relaying. The proposed scheme only requires the eavesdropper’s angle information, and adopts an angular secrecy model to represent the average secrecy rate of the attacked system. With this angular model, the AAUC problem turns out to be nonconvex, and a successive convex optimization algorithm, which converges to a Karush-Kuhn-Tucker solution, is proposed. Furthermore, a closed-form solution and a Bregman first-order method are derived for the cases of large-scale antennas and large-scale users, respectively. Extension to the intelligent reflecting surfaces based scheme is also discussed. Simulation results demonstrate the effectiveness of the proposed successive convex optimization based AAUC scheme, and also validate the low-complexity nature of the proposed large-scale optimization algorithms.

Poor signal-to-noise ratios (SNRs) and low spatial resolutions have impeded low-cost pyroelectric infrared (PIR) sensors from many intelligent applications for thermal target detection/recognition. This article presents a cognitive signal conditioning and modulation learning framework for PIR sensing with the following two innovations to solve these problems: 1) a reconfigurable signal conditioning circuit design to achieve high SNRs and 2) an optimal sensor mask design to achieve high recognition performance. By using a programmable system on chip, the PIR signal amplifier gain and filter bandwidth can be adjusted automatically according to working conditions. Based on the modeling between PIR physics and thermal images, sensor masks can be optimized through training convolution neural networks with large thermal image datasets for feature extraction of specific thermal targets. The experimental results verify the improved performance of PIR sensors in various working conditions and applications by using the developed reconfigurable circuit and application-specific masks.

Unmanned aerial vehicles (UAVs) are advantageous in their high maneuverability for long-range outdoor target tracking. In this paper, we develop a UAV-based target tracking and recognition system using an intelligent gimbal system with the capabilities of accurate camera positioning, fast image processing, and multi-modality information fusion. Algorithms of consensus-based target tracking, moving background processing, and neural network-based target detection/recognition are optimized for embedded implementation. A geographic information system (GIS) is utilized to provide geo-location, environmental, and contextual information. The experimental results demonstrate the robust performance of the proposed UAV-based target tracking and recognition framework.

Simulation environments play a centric role in the research of sensor fusion and robot control. This paper presents Pavilion, a novel open-source simulation system, for robot perception and kinematic control based on the Unreal Engine and the Robot Operating System (ROS). The novelty of this work includes threefold: (1) developing a shader-based method to generate optical flow ground-truth data with the Unreal Engine, (2) developing a toolset to remove binary incompatibility between ROS and the Unreal Engine to enable real-time interaction, and (3) developing a method to directly import Simulation Description Format (SDF) robot models into the Unreal Engine at runtime. Finally, a Gazebo-compatible real-time simulation system is developed to enable training and evaluation of a large number of sensor fusion, planning, decision and control algorithms. The system can be implemented on both Linux and macOS, with the latest version of ROS. Various experiments have been performed to validate the superior performance of the proposed simulation environment over other state-of-the-art simulators in terms of number of modalities, simulation accuracy, latency and degree of integration difficulty.

This paper presents an encoded smart floor for multiple human localization, include binary sensor designing, space encoding and decoding scheme. This system can localize a group of people, as well as recognize associated scenarios, with high sensing efficiency and low computational complexity. The novelty of this work includes: (1) a set of code design for binary sensor deployment; (2) a Bayesian inference based decoding scheme in the context of activity and scenario recognition. The proposed scheme has been tested with pressure sensors, and the experiment results have demonstrated the superior performance of our design.

In this paper, we propose to build a comprehensive cyberphysical system (CPS) with virtual reality (VR) and intelligent sensors for motion recognition and training. We use both wearable wireless sensors (such as electrocardiogram, motion sensors) and nonintrusive wireless sensors (such as gait sensors) to monitor the motion training status. We first provide our CPS architecture. Then we focus on motion training from three perspectives: 1) VR-first we introduce how we can use motion capture camera to trace the motions; 2) gait recognition-we have invented low-cost small wireless pyroelectric sensor, which can recognize different gaits through Bayesian pattern learning. It can automatically measure gait training effects; and 3) gesture recognition-to quickly tell what motions the subject is doing, we propose a low-cost, low-complexity motion recognition system with 3-axis accelerometers. We will provide hardware and software design. Our experimental results validate the efficiency and accuracy of our CPS design.

This paper presents an information-gain-based sensor selection approach as well as a sensor sensing probability model-based calibration process for multihuman tracking in distributed binary pyroelectric infrared sensor networks. This research includes three contributions: 1) choose the subset of sensors that can maximize the mutual information between sensors and targets; 2) find the sensor sensing probability model to represent the sensing space for sensor calibration; and 3) provide a factor graph-based message passing scheme for distributed tracking. Our approach can find the solution for sensor selection to optimize the performance of tracking. The sensing probability model is efficiently optimized through the calibration process in order to update the parameters of sensor positions and rotations. An application for mobile calibration and tracking is developed. Simulation and experimental results are provided to validate the proposed framework.

Conventional pyroelectric infrared (PIR) motion sensors use paired elements for the detection of moving targets. This method makes them incapable of measuring thermal signals from static targets. We need an active sensor that can detect static thermal subjects. This paper presents our design of active PIR sensors. The proposed PIR sensing systems can actively detect static thermal targets by using three methods that are suitable to different applications: 1) a sensor that can be rotated by a self-controlled servo motor for the detection of moving or static thermal subjects nearby; 2) a sensor that is equipped with a mask for low-complexity posture recognition; and 3) a sensor that can be worn on the wrist for the recognition of surrounding subjects (this sensor is especially useful for blind users). Compressive sensing (CS) theory indicates that random down-sampling method can capture more accurate information of the original signal than the evenly spaced sampling. Based on CS theory, we have developed the random sampling structures for the active PIR systems, and have built a statistical feature space for human scenario recognition. The experimental results demonstrate that the active sensing system can efficiently measure the static thermal targets, and the random sampling scheme has a better recognition performance than the even sampling scheme.

This paper presents a flexible fiber-optic sensor-based pressure sensing system for human activity analysis and situation perception in indoor environments. In this system, a binary sensing technology is applied to reduce the data workload, and a bipedal movement-based space encoding scheme is designed to capture people's geometric information. We also develop a nonrepetitive encoding scheme to eliminate the ambiguity caused by the two-foot structure of bipedal movements. Furthermore, we propose an invariant activity representation model based on trajectory segments and their statistical distributions. In addition, a mixture model is applied to represent scenarios. The number of subjects is finally determined by Bayesian information criterion. The Bayesian network and region of interests are employed to facilitate the perception of interactions and situations. The results are obtained using distribution divergence estimation, expectation-maximization, and Bayesian network inference methods. In the experiments, we simulated an office environment and tested walk, work, rest, and talk activities for both one and two person cases. The experiment results have demonstrated that the average individual activity recognition is higher than 90%, and the situation perception rate can achieve 80%.

Distributed binary pyroelectric sensor network (PSN) is a low-cost alternative to video systems for human monitoring applications. This paper presents a PSN-based mobile target recognition system, which aims to achieve multitarget, complex scenario recognition. In this system, a novel pseudorandom visibility mode is designed for the sensor arrays to help capture statistical information of scenarios, and a sensor array fusion scheme is adopted to facilitate discriminative feature extraction. Moreover, we propose a statistical subspace representation model called probabilistic nonnegative matrix factorization (PNMF) to seek the scenario patterns rather than the object characteristics. We also further prove that our PNMF model is a generic model for NMF based algorithms. Original NMF, sparse NMF, and smooth NMF are special cases of the PNMF model. The simulation and experimental results demonstrate the advantages of our proposed method. Our system can be further developed to function as an independent facility for intelligent monitoring applications, especially under poor illumination circumstances.

Active stereo vision is a method of 3D surface scanning involving the projecting and capturing of a series of light patterns where depth is derived from correspondences between the observed and projected patterns. In contrast, passive stereo vision reveals depth through correspondences between textured images from two or more cameras. By employing a projector, active stereo vision systems find correspondences between two or more cameras, without ambiguity, independent of object texture. In this paper, we present a hybrid 3D reconstruction framework that supplements projected pattern correspondence matching with texture information. The proposed scheme consists of using projected pattern data to derive initial correspondences across cameras and then using texture data to eliminate ambiguities. Pattern modulation data are then used to estimate error models from which Kullback-Leibler divergence refinement is applied to reduce misregistration errors. Using only a small number of patterns, the presented approach reduces measurement errors versus traditional structured light and phase matching methodologies while being insensitive to gamma distortion, projector flickering, and secondary reflections. Experimental results demonstrate these advantages in terms of enhanced 3D reconstruction performance in the presence of noise, deterministic distortions, and conditions of texture and depth contrast.

Phase shifting structured light illumination for range sensing involves projecting a set of grating patterns where accuracy is determined, in part, by the number of stripes. However, high pattern frequencies introduce ambiguities during phase unwrapping. This paper proposes a process for embedding a period cue into the projected pattern set without reducing the signal-to-noise ratio. As a result, each period of the high frequency signal can be identified. The proposed method can unwrap high frequency phase and achieve high measurement precision without increasing the pattern number. Therefore, the proposed method can significantly benefit real-time applications. The method is verified by theoretical and experimental analysis using prototype system built to achieve 120 fps at 640 × 480 resolution.

Implantable medical devices (IMDs) have played an important role in many medical fields. Any failure in IMDs operations could cause serious consequences and it is important to protect the IMDs access from unauthenticated access. This study investigates secure IMD data collection within a telehealthcare [mobile health (m-health)] network. We use medical sensors carried by patients to securely access IMD data and perform secure sensor-to-sensor communications between patients to relay the IMD data to a remote doctor's server. To meet the requirements on low computational complexity, we choose N-th degree truncated polynomial ring (NTRU)-based encryption/decryption to secure IMD-sensor and sensor-sensor communications. An extended matryoshkas model is developed to estimate direct/indirect trust relationship among sensors. An NTRU hardware implementation in very large integrated circuit hardware description language is studied based on industry Standard IEEE 1363 to increase the speed of key generation. The performance analysis results demonstrate the security robustness of the proposed IMD data access trust model.

Many elder patients have multiple health conditions such as heart attacks (of various kinds), brain problems (such as seizure, mental disorder, etc.), high blood pressure, etc. Monitoring those conditions needs different types of sensors for analog signal data acquisition, such as electrocardiogram (ECG) for heart beats, electroencephalogram (EEG) for brain signals, and electromyogram (EMG) for muscles motions. To reduce mobile-health (m-health) cost, the above sensors should be made in tiny size, low memory, and long-term battery operations. We have designed a series of medical sensors with wireless networking capabilities. In this paper, we report our work in three aspects: (1) networked embedded system design, (2) network congestion reduction, and (3) network loss compensation. First, for networked embedded system design, we have designed an integrated wireless sensor network hardware / software platform for multi-condition patient monitoring. Such a system integrates ECG/EEG/other sensors with Radio Frequency Identification (RFID) into a Radio Frequency (RF) board through a programmable interface chip, called PSoc. Second, for network congestion reduction, the interface chip can use compressive signal processing to extract bio-signal feature parameters and only transmit those parameters. This provides an alternative approach to sensor network congestion reduction that aims to alleviate ?hot spot? issues. Third, for network loss compensation, we have designed wireless loss recovery schemes for different situations as follows. (1) If original sensor data streams are transmitted, network congestion will be a big concern due to the heavy traffic. A receiver-only loss prediction will be a good solution. (2) If the signal parameters are transmitted, the transmission loss mandates a 100% recovery rate. We have comprehensively compared the performance of those schemes. The proposed mechanisms for m-health system have potentially significant impacts on today's elder nursing home management and other mobile patient monitoring applications.

This work proposes an interface design of a low-power programmable system on chip for intelligent wireless sensor nodes to reduce the overall power consumption of the heart disease monitoring system, by lending them the capability of processing complex functions and performing rapid computations on a large amount of data at the node. This facilitates the node to intelligently monitor a medical signal for impending events instead of transmitting the signal to the base station constantly. Lowering the transmission data rate decreases the transmission power consumption in a node, thereby lengthening the node life and in turn increasing the reliability of the network. This work also implements a thresholding technique, which controls the data transmission rate depending on the value of the monitored signal, and a cardiac monitoring system that performs computations at the node for the detection of either a skipped heart beat or a reduced heart rate variability, in which event the signal is transmitted to the base station for monitoring/recording or alerting the crew. The performance analysis of the system shows that there are reductions in the system power consumption and data transmission rate, which in turn reduces the network traffic and averts congestion.