Abstract Estimating the position and orientation of a rigid object from an image is critical for situational awareness in robotics and autonomous systems. This study explores relative pose estimation using an ultra-wide fisheye camera for unmanned aircraft inspection vehicles. Ultra-wide fisheye lenses introduce radial distortion and capture features beyond the rectilinear image plane, rendering rectilinear Perspective-n-Point (PnP) algorithms inadequate. Designing a bespoke ultra-wide fisheye localization algorithm requires consideration of both the feature detection method and the pose estimator itself. This study proposes a novel method that combines (1) a fisheye-to-cubemap reprojection, (2) a You Only Look Once (YOLO) convolutional neural network trained for arbitrary airborne perspectives, and (3) an Angle-Agnostic and Multiple-Frame PnP (AMP) pose estimation algorithm. Our pipeline achieves a 97% success rate for valid pose estimates, with a mean absolute translational error of less than 12 cm on real ultra-wide fisheye imagery, outperforming conventional techniques, including OpenCV.

Abstract This paper presents a comprehensive approach to enhancing autonomous docking maneuvers through machine visual perception and sim-to-real transfer learning. By leveraging relative vectoring techniques, we aim to replicate the human ability to execute precise docking operations. Our study focuses on autonomous aerial refueling as a use case, demonstrating significant advancements in relative navigation and object detection. We introduce a novel method for aligning digital twins using fiducial targets and motion capture data, which facilitates accurate pose estimation from real-world imagery. Additionally, we develop cost-efficient annotation automation techniques for generating high-quality You Only Look Once training data. Experimental results indicate that our transfer learning methodologies enable accurate and reliable relative vectoring in real-world conditions, achieving error margins of less than 3 cm at contact (when vehicles are approximately 4 m from the camera) and maintaining performance at over 56 fps. The research findings underscore the potential of augmented reality and scene augmentation in improving model generalization and performance, bridging the gap between simulation and real-world applications. This work lays the groundwork for deploying autonomous docking systems in complex and dynamic environments, minimizing human intervention and enhancing operational efficiency.

Relative navigation methods are a critical enabling technology for the next generation of autonomous spacecraft conducting close proximity operations. This is especially true for multi-agent inspection operations in which safety including intra-agent or agent-target collisions are a serious concern. Additionally, in an on-orbit servicing operation various failure modes of the target may result in unreliable a-priori knowledge or cooperation from the target. The main contribution of this work is the demonstration of a method for multi-agent, relative pose estimation that is robust to A) sensor blinding and B) dynamic uncertainty. This objective is accomplished leveraging GTSAM, an existing toolbox for the formulation of factor graphs, along with an algorithm for the efficient, real-time solution of such factor graphs, iSAM2. This estimation method is demonstrated in an example scenario with uncertain dynamics and sensor blinding due to sun position. Results revealed that the iSAM2-based method is capable of handling sensor blinding through leveraging an inter-agent range measurement, despite a dynamically uncertain environment.

Abstract Achieving precise 6 degrees of freedom (6D) pose estimation of rigid objects from color images is a critical challenge with wide-ranging applications in robotics and close-contact aircraft operations. This study investigates key techniques in the application of YOLOv5 object detection convolutional neural network (CNN) for 6D pose localization of aircraft using only color imagery. Traditional object detection labeling methods suffer from inaccuracies due to perspective geometry and being limited to visible key points. This research demonstrates that with precise labeling, a CNN can predict object features with near-pixel accuracy, effectively learning the distinct appearance of the object due to perspective distortion with a pinhole camera. Additionally, we highlight the crucial role of knowledge about occluded features. Training the CNN with such knowledge slightly reduces pixel precision, but enables the prediction of 3 times more features, including those that are not initially visible, resulting in an overall better performing 6D system. Notably, we reveal that the data augmentation technique ofscalecan interfere with pixel precision when used during training. These findings are crucial for the entire system, which leverages the Solve Perspective-N-Point (Solve-PnP) algorithm, achieving 6D pose accuracy within 1 $$^\circ$$ ∘ and 7 cm at distances ranging from 7.5 to 35 m from the camera. Moreover, this solution operates in real-time, achieving sub-10ms processing times on a desktop PC.

Abstract Once realized, autonomous aerial refueling will revolutionize unmanned aviation by removing current range and endurance limitations. Previous attempts at establishing vision-based solutions have come close but rely heavily on near perfect extrinsic camera calibrations that often change midflight. In this paper, we propose dual object detection, a technique that overcomes such requirement by transforming aerial refueling imagery directly into receiver aircraft reference frame probe-to-drogue vectors regardless of camera position and orientation. These vectors are precisely what autonomous agents need to successfully maneuver the tanker and receiver aircraft in synchronous flight during refueling operations. Our method follows a common 4-stage process of capturing an image, finding 2D points in the image, matching those points to 3D object features, and analytically solving for the object pose. However, we extend this pipeline by simultaneously performing these operations across two objects instead of one using machine learning and add a fifth stage that transforms the two pose estimates into a relative vector. Furthermore, we propose a novel supervised learning method using bounding box corrections such that our trained artificial neural networks can accurately predict 2D image points corresponding to known 3D object points. Simulation results show that this method is reliable, accurate (within 3 cm at contact), and fast (45.5 fps).

Abstract Point set registration algorithms such as Iterative Closest Point (ICP) are commonly utilized in time-constrained environments like robotics. Finding the nearest neighbor of a point in a reference 3D point set is a common operation in ICP and frequently consumes at least 90% of the computation time. We introduce a novel approach to performing the distance-based nearest neighbor step based on Delaunay triangulation. This greedy algorithm finds the nearest neighbor of a query point by traversing the edges of the Delaunay triangulation created from a reference 3D point set. Our work integrates the Delaunay traversal into the correspondences search of ICP and exploits the iterative aspect of ICP by caching previous correspondences to expedite each iteration. An algorithmic analysis and comparison is conducted showing an order of magnitude speedup for both serial and vector processor implementation.

Bayesian estimators are commonly used to estimate the state of a system over time, whether the Kalman Filter (or its derivatives), a particle filter, a sliding window approach or batch-based approach. A common assumption with all these estimators is that the parameters of the dynamics and measurement noise sources are known a-priori. To overcome this weakness, previous approaches have simultaneously estimated the system state and the covariance of the inputs. All of these approaches, however, have made the assumption that the input noise is still uncorrelated over time. In this paper, we remove this assumption by inferring both system parameters (such as the parameters for a first-order Gauss-Markov process (FOGM)) and process noise covariance simultaneously. This estimation is performed using variational inference in conjunction with a Rauch-Tung-Striebel (RTS) smoother. We demonstrate significantly improved performance over a similar RTS-based approach that estimates Q but has the time constant of the FOGM added to the system state.

When attempting to estimate the state of a dynamic system, one of the common assumptions is that the uncertainty information for the inputs (measurements and dynamics) is known a-priori. Unfortunately, this is not a valid assumption in many cases, leading to the development of multiple covariance estimation techniques for both the Kalman filter and smoothing techniques. This paper presents a novel method to estimate the covariances of the inputs in a factor-graph formulation of the Bayesian estimation problem. A general solution, based on covariance estimation in linear regression problems, is presented that gives unbiased estimators of multiple variances from measured data. An iteratively re-weighted least squares (IRLS) algorithm is then used to estimate the input variances of a non-linear system using factor graph optimization. Simulation studies using a robot localization problem demonstrate the efficacy of our proposed techniques.

Previously the Gauss Newton method has been used to estimate the geo-location of an object from angle of arrival (AoA) measurements. This method has assumed, however, that all measurements were independent and identically distributed. Real sensor data, however, often has temporal correlations between measurements. If a detailed understanding of the measurement correlation exists, this correlation can be explicitly modeled and jointly estimated with the geo-location. Obtaining a detailed and accurate model of measurement error correlation, however, is often infeasible for a system where the unit producing measurements may be a black box. To overcome this unknown correlation between measurements, we propose a modified Gauss-Newton optimization algorithm based on prior Covariance Intersection work. A discussion on the efficacy of this modified technique, in terms of both geo-location accuracy and accurate prediction of geo-location uncertainty, concludes the paper.

The RANdom SAmple Consensus (RANSAC) algorithm, as a robust parameter estimator, has been widely used to remove gross errors. However, there is less work on analyzing the uncertainty produced by the RANSAC. This paper fills this gap by presenting an uncertainty estimation algorithm for the RANSAC. Based on a thorough analysis on the uncertainty of the model parameters generated during the random hypothesis sampling process of the RANSAC, we derive the probability that each hypothesis is selected as the best hypothesis by the RANSAC. Using the probability of the best hypothesis, we characterize the error expectation and error covariance of the model parameter estimates and compute the probability of each data point being an inlier. Three models including line fitting, homography, and essential matrix are used to evaluate the performance of the uncertainty estimation algorithm. Results demonstrate that the uncertainty produced by the RANSAC is characterized successfully by the proposed algorithm.

When observing an object of interest (herein simply called “object”), determining the geo-location of the object as accurately as possible, together with an accurate estimate of the geo-location uncertainty, is a key step in many remote sensing applications. To compute the geo-location of an object, Angle of Arrival (AoA) measurements can be used from multiple different sensor platforms such as aerostats, aircraft, and satellites. In a previous paper, nonlinear optimization (NLO) was used to provide an optimal geolocation estimate which converges to the minimum mean squared error (MMSE) for the problem of a stationary object and simultaneous measurements. This paper describes a generalization of the same (NLO method applied to fast-moving objects and nonsimultaneous measurements. For some sensor platforms, the object may be so far from the sensor that it moves significantly by the time the sensor measures the AoA. This is especially true for satellites in GEO or HEO. The time-delay from the Earth to such satellites may be as much as 200 ms, meaning the sensors produce measurements of where the object was rather than where it is at the time of measurement. In addition, sensors rarely produce measurements at the same time. This temporal incongruence of measurements can significantly complicate the geo-location problem for a rapidly moving object. Thus a scheme which can produce an optimal geolocation estimate and a confidence estimate for this scenario is desirable. Nonlinear optimization is proposed and examined as a means of satisfying this objective.

This paper addresses the problem of object localization using only angle-of-arrival (AoA) data from satellites. Traditionally, this is performed by a triangulation algorithm (TA) that minimizes the distances between the estimated object location to all lines-of-sight representing measurements of the object. However, when observing objects from satellites, the differences in distance from each satellite to the object can be significant. The error this induces in measurements from farther-distant satellites results in an inordinate impact on the geo-location error. To overcome this problem, we introduce a non-linear optimization (NLO) approach that models the measurement error at each satellites as a probability density function. By finding the most-probable geo-location of the object, this systematic error is eliminated. We found that the NLO provides a more accurate estimate of the object's location than the TA in 93% of instances. In addition, we analyze the uncertainty estimates generated by both the TA and NLO approaches. The NLO estimates of uncertainty are also considerably more accurate than the TA estimates in all cases.

In this paper, we investigate the nonlinear observability properties of bearing-only cooperative localization. We establish a link between observability and a graph that represents measurements and communication between the robots. It is shown that graph theoretic properties like the connectivity and the existence of a path between two nodes can be used to explain the observability of the system. We obtain the maximum rank of the observability matrix without global information and derive conditions under which the maximum rank can be achieved. Furthermore, we show that for complete observability, all of the nodes in the graph must have a path to at least two different landmarks of known location.

In this paper we propose a fast method for constructing multi-view stereo panoramas using a layering approach. Constructing panoramas requires accurate camera pose estimation and will often require an image blending or interpolation method to generate seamless results. We use a registration error correction method that provides globally corrected and fast results for paths that create cycles such as circular paths, back and forth straight sweeps, and even a single sweep. Then we apply our layering approach to generate multi-view stereo panoramas quickly for time sensitive applications that require immediate results and 3D perception.

In this work, we present a multimodal temporal panorama (MTP) representation that synchronizes visual, motion, and acoustic signatures of moving vehicles in the time axis. The MTP representation includes two layers: a synopsis layer and a snapshot layer. The temporal synopsis consists of 1) a panoramic view image (PVI) to represent vehicles' presence, which is constructed from 1D vertical detecting lines of a selected column location of all video frames, 2) an epipolar plane image (EPI) to characterize their motion (speeds and directions), generated from 1D horizontal scanning lines along the vehicles' moving paths, and 3) an audio wave scroll for visualizing moving vehicles' acoustic signatures. The MTP synopsis not only synchronizes all the three modalities (visual, motion and acoustic) of the vehicles, but also provides information that can perform automatic detection tasks including moving vehicle visual detection, motion estimation, and acoustic signature retrieval. Then in the snapshot layer, the occlusion-free, motion-blur-free, and view-invariant reconstruction of each vehicle (with both shape and motion information) and its acoustic signatures (e.g. spectrogram) are embedded. The MTP provides a very effective approach to (semi-)automatically labeling the multimodal data of uncontrolled traffic scenes in real time for further vehicle classification, check-point inspection and traffic analysis. The concept of MTP may not be only limited to visual, motion and audio modalities, it could also be applicable to other sensing modalities that can obtain data in the temporal domain.

Miniature Aerial Vehicles (MAVs) are quickly gaining acceptance as a platform for performing remote sensing. However, because MAVs are flown close to the ground (300 meters or less in altitude), their field of view for any one image is relatively small. In addition, the context of the video (where and at what orientation are the objects being observed, the relationship between images) is unclear from any one image. To overcome these problems, we propose a geo-referenced mo-saicing method that creates a mosaic from the captured images and geo-references the mosaic using information from the MAV IMU/GPS unit. Our method utilizes bundle adjustment within a constrained optimization framework. Using real MAV video, we have demonstrated our mosaic creation process on over 700 frames. Our method has been shown to produce the high quality mosaics to within 7m using tightly synchronized MAV telemetry data and to within 30m using only GPS information (i.e. no roll and pitch information).

Miniature UAVs (MAVs) are quickly gaining acceptance as a platform for performing remote sensing or surveillance of remote areas. However, due to their small size, MAVs are very susceptible to wind gusts and other atmospheric disturbances. These disturbances cause the video from an MAV to have a significant amount of movement, making the video frames difficult to watch and disorienting to the user. In addition, the context of the video (where and at what orientation are the objects being observed) is often as important as the video itself. To overcome the video movement problem and present location context to the user, we present a method for creating a mosaic from the frames of the video while simultaneously geo-referencing the mosaic. We demonstrate results using a 5-foot fixed-wing MAV.

We present a system to improve the estimation of MAV location and attitude by combining GPS, IMU and visual information in an unscented Kalman filter framework. Feature points are tracked and combined to create a homography matrix which is used as the measurement input to the filter. We present a novel method to transform uncertainty in feature tracking to uncertainty in the homography. Using a system developed with this framework, we present results which show that this method can substantially increase the accuracy of pose estimation, compared to GPS/IMU alone.

muUAVs have found increasing utility as a platform for visual data collection about an area of interest. One user-friendly way of presenting visual data about the area of interest is to create a mosaick of all data observed by the muUAV, absolving the user from having to watch the entire flight video. In addition, because multiple frames in the video sequence will in general observe an area of interest, super-resolution techniques can be applied to yield a mosaick with higher resolution than the camera on-board the muUAV. While super-resolution and mosaicking are both well studied problems, muUAVs present some interesting challenges in combining the two capabilities. Most super-resolution algorithms work within a constrained environment where the size of the superresolved image is fixed and known a priori. Mosaicking, however, is the process of creating a dynamically-sized image from multiple fixed-sized images, where each input image could contain visual data on previously unobserved areas. In this paper, we present a system that can do both: construct a mosaick, incorporating data on previously unobserved areas, while super-resolving the mosaick for areas that are observed by multiple input images

The transmission of video over wireless channels is a challenging problem, in part due to the rapid and wide variations in wireless channel bandwidth and error conditions. In this paper, we present a run-time method for assembling a video stream, VSHAPER, that shapes the video stream depending on the current wireless channel conditions. Previous methods for shaping video streams have either concentrated on bandwidth adaptation only, or have required the video stream to be compressed at run-time. VSHAPER assembles the output video stream from several pre-compressed videos, requiring minimal computational overhead at the video server. A modified rate-distortion optimization method for deciding how to shape the video stream is also used to minimize computational overhead. The output of VSHAPER is a video stream that can be read by any standard video decoder. Significant gains in the video quality of received video during varying wireless conditions are observed. In addition, a video server's capacity (number of video streams it can simultaneously serve) is increased significantly.

The transmission of digital video over wireless channels faces many problems common Io both wired and wireless networks such as bandwidth and buffer restrictions and low energy constraints, together with problems unique to wireless technologies such as extreme and rapidly varying channel noise. In this paper, we focus on enabling (he wireless communication of video clips despite the noisy and dynamic nature of wireless channels. Many efforts have been made in the past to increase the error resiliency of compressed video, but these approaches have not considered the effects of buffering on the received video quality. Traditional buffering (where a "sliding window" of the next few frames is kept in the buffer at all times), while useful for overcoming latency jitter, cannot be used Io guarantee against poor video quality and stalls due to wireless channel fluctuations. To help overcome the weaknesses of traditional buffering, we introduced out-of-order buffering (ORBit), a method for selecting important frames from throughout a video clip and buffering them before playback begins. ORBit can lead to significant gains in video quality despite variations in the wireless channel during the communication of a video clip, but requires the judicious selection of ORBit frames, with their corresponding quantization and channel coding levels. To enable run-time selection of ORBit frames, we introduce a fast video distortion estimation technique. Using our estimation technique, we present an ORBiI decision algorithm that selects ORBit frames depending on the wireless technology and end-user appliance constraints. Using the ORBit has to wait for a few seconds before the video display begins. During decision algorithm, significant gains of up to 9 dB in video quality are obtained.

As the technology sizes of semiconductor devices continue to decrease, the effect of nanometer technologies on interconnects, such as crosstalk glitches and timing variations, become more significant. In this paper, we study the effect of nanometer technologies on energy dissipation in interconnects. We propose a new power estimation technique which considers DSM effects, resulting in significantly more accurate energy dissipation estimates than transition-count based methods for on-chip interconnects. We also introduce an energy minimization technique which attempts to minimize large voltage swings across the cross-coupling capacitances between interconnects. Even though the number of transitions may increase, our method yields a decrease in power consumption of up to 50%.

To enable ubiquitous wireless multimedia communication, the bottlenecks to communicating multimedia data over wireless channels must be addressed. Two significant bottlenecks which need to be overcome are the bandwidth and energy consumption requirements for mobile multimedia communication. In this paper, we address the bandwidth and energy dissipation bottlenecks by adapting the image compression parameters to current communication conditions and constraints. We focus on the JPEG image compression algorithm, and present the results of varying some image compression parameters on energy dissipation, bandwidth required, and quality of image received. We present a methodology for selecting the JPEG image compression parameters in order to minimize energy consumption while meeting latency, bandwidth, and quality of image constraints.