Perception and prediction modules are critical components of autonomous driving systems, enabling vehicles to navigate safely through complex environments. The perception module is responsible for perceiving the environment, including static and dynamic objects, while the prediction module is responsible for predicting the future behavior of these objects. These modules are typically divided into three tasks: object detection, object tracking, and motion prediction. Traditionally, these tasks are developed and optimized independently, with outputs passed sequentially from one to the next. However, this approach has significant limitations: computational resources are not shared across tasks, the lack of integrated optimization can amplify errors as they propagate throughout the pipeline, and uncertainty is rarely propagated between modules, resulting in significant information loss. To address these challenges, the joint perception and prediction paradigm has emerged, integrating perception and prediction into a unified model through multi-task learning. This strategy not only overcomes the limitations of previous methods, but also enables the three tasks to have direct access to raw sensor data, allowing richer and more nuanced environmental interpretations. This paper presents the first comprehensive survey of joint perception and prediction for autonomous driving. We propose a taxonomy that categorizes approaches based on input representation, scene context modeling, and output representation, highlighting their contributions and limitations. Additionally, we present a qualitative analysis and quantitative comparison of existing methods. Finally, we discuss future research directions based on identified gaps in the state-of-the-art.

In the context of robotics, accurate 3D human pose estimation is essential for enhancing human-robot collaboration and interaction. This manuscript introduces a multi-view 2D to 3D lifting optimization-based method designed for video-based 3D human pose estimation, incorporating temporal information. Our technique addresses key challenges, namely robustness to 2D joint detection error, occlusions, and varying camera perspectives. We evaluate the performance of the algorithm through extensive experiments on the MPI-INF-3DHP dataset. Our method demonstrates very good robustness up to 25 pixels of 2D joint error and shows resilience in scenarios involving several occluded joints. Comparative analyses against existing 2D to 3D lifting and multi-view methods showcase good performance of our approach.

Current approaches of applied in urban focus on decoupling the perception training from the driving policy training. The main reason is to avoid training a convolution encoder alongside a policy network, which is known to have issues related to sample efficiency, degenerated feature representations, and catastrophic self-overfitting. However, this paradigm can lead to representations of the environment that are not aligned with the downstream task, which may result in suboptimal performances. To address this limitation, this paper proposes RLAD, the first method applied in the urban ad domain. We propose several techniques to enhance the performance of an rlfp algorithm in this domain, including: 1) an image encoder that leverages both image augmentations and layers; 2) WayConv1D, which is a waypoint encoder that harnesses the 2D geometrical information of the waypoints using 1D convolutions; and 3) an auxiliary loss to increase the significance of the traffic lights in the latent representation of the environment. Experimental results show that RLAD significantly outperforms all state-of-the-art rlfp methods on the NoCrash benchmark. We also present an infraction analysis on the NoCrash-regular benchmark, which indicates that RLAD performs better than all other methods in terms of both collision rate and red light infractions. The source code of RLAD is available at https://github.com/DanielCoelho112/rlad.Note to Practitioners—The practical problem that motivated our work is the application of RLfP in urban AD. Unlike common approaches in AD, our main goal is to learn intermediate feature representations of the environment that are aligned with the driving task. Our method is the first RLfP algorithm applied in the urban AD domain, and results show that it surpasses all state-of-the-art RLfP methods on the NoCrash benchmark. The key insights and contributions of our work lie in the development of effective techniques for RLfP in urban AD, improving the overall performance and safety of AD systems. Beyond AD, the techniques and insights from our work could be extended to other domains that involve RL from pixel inputs, such as robotics, and video game agents. These applications would benefit from the enhanced performance offered by our proposed methods.

This article proposes a general approach to solve the hand–eye calibration problem. The system is general since it is able to calibrate any number of cameras and, moreover, is able to simultaneously perform the calibration of several instances of the two common hand–eye calibration use cases: eye-on-hand and eye-to-base. The calibration is solved with a nonlinear least squares method, and the reprojection error is used as a metric to guide the optimization procedure. Our approach is seamlessly integrated with the robot operating system framework and allows for the interactive positioning of sensors and labeling of data, facilitating both the data acquisition and labeling and the calibration procedures. Results show that the proposed approach is able to handle any calibration use case with a minimal initial configuration. The approach is compared with several other state-of-the-art hand–eye calibration algorithms. Results show that the proposed approach produces very accurate calibrations when compared to the state of the art.

Image mosaicking applications require both geometrical and photometrical registrations between the images that compose the mosaic. This paper proposes a probabilistic color correction algorithm for correcting the photometrical disparities. First, the image to be color corrected is segmented into several regions using mean shift. Then, connected regions are extracted using a region fusion algorithm. Local joint image histograms of each region are modeled as collections of truncated Gaussians using a maximum likelihood estimation procedure. Then, local color palette mapping functions are computed using these sets of Gaussians. The color correction is performed by applying those functions to all the regions of the image. An extensive comparison with ten other state of the art color correction algorithms is presented, using two different image pair data sets. Results show that the proposed approach obtains the best average scores in both data sets and evaluation metrics and is also the most robust to failures.

Service and assistance robots that must move in human environment must address the difficult issue of navigating in dynamic environments. As it has been shown in previous works, in such situations the robots can take advantage of the motion of persons by following them, managing to move together with humans in difficult situations. In those circumstances, the problem to be solved is how to choose a human leader to be followed. This work proposes an innovative method for leader selection, based on human experience. A learning framework is developed, where data is acquired, labeled and then used to train an AdaBoost classification algorithm, to determine if a candidate is a bad or a good leader, and also to study the contribution of features to the classification process.

Service robots have a great potential of improving human quality of life by aiding in everyday tasks. However, robots that share an environment and interact with humans still face some challenges that limits their acceptance. One of these challenges is how to move and behave among groups of people, which is a task performed seamlessly by humans and some animals. Motion planning in dynamic environments has been addressed mostly by predicting future position of humans and avoiding them. However, with the increase of the number of persons in such environments, techniques that are based only on the prediction of the movement of humans can fail, as they usually ignore the human's reaction to the presence of the robot. Instead of trying to model the complex human motion behavior, this work proposes to rely on humans to guide the robot through difficult situations, where modern approaches would fail to find a solution. This will be accomplished by a probabilistic approach for selecting a human leader, according to the robot's desired destination. In this way, the robot can take advantage of the humans' paths and behavior, effortlessly avoiding dynamic and static features together with the human leader, relieving the robot from the burden of having to generate its own path in difficult situations.

A crucial requirement for service robots is to be able to move in dynamic environments shared with humans as well as interact with them. Navigation in such environments is a challenging task, as the environment is constantly changing, future states have to be predicted and planning and execution must be carried on-line. However, even in very complex situations, humans can easily find a path that avoid both dynamic agents and static obstacles. This paper proposes a technique to take advantage of the human movement in such populated environments, using a probabilistic approach for the leader selection, according to the robot's desired destination. By choosing a leader to be followed in dynamic environments, the robot can take advantage of the paths traveled by humans or other robots, effortlessly avoiding dynamic and static features as its leader does, relieving the robot from the burden of having to generate its own path. Both the leader selection and the leader following algorithms have been tested in a real environment, with a robotic wheelchair.

The current paper proposes a novel color correction approach for onboard multi-camera systems. It works by segmenting the given images into several regions. A probabilistic segmentation framework, using 3D Gaussian Mixture Models, is proposed. Regions are used to compute local color correction functions, which are then combined to obtain the final corrected image. An image data set of road scenarios is used to establish a performance comparison of the proposed method with other seven well known color correction algorithms. Results show that the proposed approach is the highest scoring color correction method. Also, the proposed single step 3D color space probabilistic segmentation reduces processing time over similar approaches.

The current paper proposes a new parametric local color correction technique. Initially, several color transfer functions are computed from the output of the mean shift color segmentation algorithm. Secondly, color influence maps are calculated. Finally, the contribution of every color transfer function is merged using the weights from the color influence maps. The proposed approach is compared with both global and local color correction approaches. Results show that our method outperforms the technique ranked first in a recent performance evaluation on this topic. Moreover, the proposed approach is computed in about one tenth of the time.

This article describes a mathematical basis required to integrate features obtained for perception for topological navigation. It is intended for application to navigation in an environment that is not mapped, but in which a mission is described in the form of a semantic description of the perception stimulus that the robot is expected to encounter. The need to integrate features from different sensors led to the use of an uncertainty estimate employed in information theory; binary entropy. By using entropy, the features are ranked in order of decreasing uncertainty. This article describes the state of the work in an as yet preliminary stage, but appears promising for application to navigation using topological information. It also offers interesting perspectives on commonly used sensory data such as local intensity image features.

This paper describes a multi-loop, modular navigation architecture for mobile robots whose structure allows the execution of most types of navigation tasks in a highly robust manner. The modularity allows the clear separation of functions according to their complexity and priority. We propose a clear separation between local and global navigation in such a way that both can run independently, but an adequate alternation/competition between them allows accomplishment of trajectory execution including avoidance of unknown obstacles. The multi-loop nature of the architecture ensures adequate stability at different levels yielding safe navigation and accomplishment of higher level tasks. Examples may range from goal reaching in some point of the environment to a 3D-environment mapping application, as this is the case in this work. 1. Introduction Navigation is a major issue when addressing mobile robotics because the concept is so wide that it includes all aspects of directing a robot's ...

This paper describes a mobile robotics system that was implemented and is capable of building instantaneous representations of the free space available, and use those representations to perform navigation. The free space representations are done by means of perception maps specially designed to hold and combine ultrasonic data. The maps are built by neural network appropriately trained to minimise undesired ranging errors due to specular reflections that is achieved by taking advantage of the redundancy of sensorial data. The concept of local navigation is developed as a new navigation approach, and is based on full independence on the environment, relying purely on sensorial perception. Local motion is generated according to simple generic behaviour descriptions: the local navigation strategies. The system which was implemented guarantees a safe local motion throughout the environment, requiring no a priori information nor any pre-defined path to follow. An adequate navigation architecture integrates the local navigation module within a framework of other several modules for more complete navigation tasks. The results obtained are quite promising in pointing the way to a truly autonomous system.

Sensorial perception is a key issue for the problem of robot local navigation, that is, the immediate or short-range motion planning, reacting only to the free space around the robot, without requiring a pre-defined trajectory plan. Therefore, local navigation requires no environment model and relies entirely on sensorial data. Commonly used sensors such as ultrasonic ranging devices, are known for their associated problems: specular reflections and crosstalk, essentially. However if sensors are used in an appropriate number and geometric lay-outs, the resulting spatial redundancy offers the possibility of overcoming some of those problems. This paper deals with these problems by means of special perception maps using ultrasound data. A generalised grid serves as the base of maps, and its cells have simply binary values: free or occupied. The relation between the topology of the perception map and the environment is a determinant factor for accurate reasoning. A 3-layer feed-forward neural network is used to perform the mapping between sensorial scans and grid occupancy. It was verified that the neural network handles most of the situations of specular reflections, and gives good perception maps for mid-range distances. Changes in environment, such as obstacles in vehicle's trajectory, have also been detected, which stresses the network's ability to generalise.>