Teleoperation can enable human intervention to help handle instances of failure in autonomy thus allowing for much safer deployment of autonomous vehicle technology. Successful teleoperation requires recreating the environment around the remote vehicle using camera data received over wireless communication channels. This paper develops a new predictive display system to tackle the significant time delays encountered in receiving camera data over wireless networks. First, a new high gain observer is developed for estimating the position and orientation of the ego vehicle. The novel observer is shown to perform accurate state estimation using only GNSS and gyroscope sensor readings. A vector field method which fuses the delayed camera and Lidar data is then presented. This method uses sparse 3D points obtained from Lidar and transforms them using the state estimates from the high gain observer to generate a sparse vector field for the camera image. Polynomial based interpolation is then performed to obtain the vector field for the complete image which is then remapped to synthesize images for accurate predictive display. The method is evaluated on real-world experimental data from the nuScenes and KITTI datasets. The performance of the high gain observer is also evaluated and compared with that of the EKF. The synthesized images using the vector field based predictive display are compared with ground truth images using various image metrics and offer vastly improved performance compared to delayed images.

Abstract Teleoperation can enable human intervention to help handle instances of failure in autonomy thus allowing for much safer deployment of autonomous vehicle technology. Successful teleoperation requires recreating the environment around the remote vehicle using camera data received over wireless communication channels. This paper develops a new predictive display system to tackle the significant time delays encountered in receiving camera data over wireless networks. First, a new high gain observer is developed for estimating the position and orientation of the ego vehicle. The novel observer is shown to perform accurate state estimation using only GNSS and gyroscope sensor readings. A vector field method which fuses the delayed camera and Lidar data is then presented. This method uses sparse 3D points obtained from Lidar and transforms them using the state estimates from the high gain observer to generate a sparse vector field for the camera image. Polynomial based interpolation is then performed to obtain the vector field for the complete image which is then remapped to synthesize images for accurate predictive display. The method is evaluated on real-world experimental data from the nuScenes and KITTI datasets. The performance of the high gain observer is also evaluated and compared with that of the EKF. The synthesized images using the vector field based predictive display are compared with ground truth images using various image metrics and offer vastly improved performance compared to delayed images.

Teleoperation could be used to replace a backup safety driver on autonomous vehicles and could play a valuable role in scaling up deployment of autonomous vehicles. The surrounding environment of the remote vehicle needs to be recreated for the teleoperator using video images received over wireless channels. To handle the significant time delays in receiving remote video data, this paper develops a predictive display system that uses deep learning to estimate the current video display for the teleoperator from old (delayed) camera images. First, old camera and Lidar data are fused to create a 3D reconstruction of the remote vehicle environment using conventional and deep-learning-based algorithms. Then the ego-vehicle’s real-time position and orientation variables are estimated using an extended Kalman filter. Predictive modification of the reconstructed old 3D scene is performed using the ego-vehicle’s estimated new trajectory variables. Deep-learning based image in-painting is used to improve image quality. Furthermore, this paper also introduces a new image comparison metric for evaluating the accuracy of the object detection and localization performance in the predictive display image. Real-world experimental data from the nuScenes and Kitti datasets are used for evaluation of the proposed system. The predictive display images are compared with ground truth images using various image comparison metrics and shown to provide significantly superior performance compared to the actual delayed images received over wireless channels.

This paper focuses on the design of a multi-output high gain observer for a vehicle trajectory tracking application. Tracking the trajectories of other vehicles on the road is needed for many applications ranging from collision avoidance to autonomous driving. Previously, such trajectory tracking has been done using linearized dynamic models, interacting-multiple-model (IMM) filters, or else by using LMI-based nonlinear observers. These estimation techniques suffer from some crucial shortcomings. Hence, this paper develops a high gain nonlinear observer for this application. The high gain observer approach offers the advantages of guaranteed feasibility and stability with just one constant observer gain for a wide range of motion. The challenges of transforming the vehicle dynamic model into the required companion form for applying the high gain observer technique are addressed. A coordinate transformation that allows for varying velocity and varying slip angle is shown to be appropriate. The high gain observer methodology for a dynamic system with multiple outputs is presented. Finally, simulation and experimental results on vehicle tracking are demonstrated. The experimental results show that, with a high gain observer, vehicle trajectories that span a large range of orientations can be accurately tracked using just one constant observer gain.

This paper focuses on activity recognition using a single wearable inertial measurement sensor placed on the subject's chest. The ten activities that need to be identified include lying down, standing, sitting, bending and walking, among others. The activity recognition approach is based on using and identifying a transfer function associated with each activity. The appropriate input and output signals for each transfer function are first determined based on the norms of the sensor signals excited by that specific activity. Then the transfer function is identified using training data and a Wiener filter based on the auto-correlation and cross-correlation of the output and input signals. The activity occurring in real-time is recognized by computing and comparing the input-output errors associated with all the transfer functions. The performance of the developed system is evaluated using data from a group of Parkinson's disease subjects, including data obtained in a clinical setting and data obtained through remote home monitoring. On average, the developed system provides better than 90% accuracy in identifying each activity as it occurs. Activity recognition is particularly useful for PD patients in order to monitor their level of activity, characterize their postural instability and recognize high risk-activities in real-time that could lead to falls.

This paper focuses on the development of an active sensing system for a bicycle to accurately detect and track rear vehicles. A collision detection sensor on a bicycle is required to be inexpensive, small, and lightweight. A single beam laser sensor that meets these constraints is mounted on a rotationally controlled platform for this sensing mission. The rotational orientation of the laser sensor needs to be actively controlled in real time in order to continue to focus on a rear vehicle, as the vehicle's lateral and longitudinal distances change. This tracking problem requires controlling the real-time angular position of the laser sensor without knowing the future trajectory of the vehicle. The challenge is addressed using a novel receding horizon framework for active control and an interacting multiple model framework for estimation. The features and benefits of this active sensing system are shown first using simulation results. Then, extensive experimental results are presented using an instrumented bicycle to show the performance of the system in detecting and tracking rear vehicles during both straight and turning maneuvers.

This paper focuses on the development of a portable roadside magnetic sensor system for vehicle counting, classification, and speed measurement. The sensor system consists of wireless anisotropic magnetic devices that do not require to be embedded in the roadway-the devices are placed next to the roadway and measure traffic in the immediately adjacent lane. An algorithm based on a magnetic field model is proposed to make the system robust to the errors created by larger vehicles driving in the nonadjacent lane. These false calls cause an 8% error if uncorrected. The use of the proposed algorithm reduces this error to only 1%. Speed measurement is based on the calculation of the cross correlation between longitudinally spaced sensors. Fast computation of the cross correlation is enabled by using frequency-domain signal processing techniques. An algorithm for automatically correcting for any small misalignment of the sensors is utilized. A high-accuracy differential Global Positioning System is used as a reference to measure vehicle speeds to evaluate the accuracy of the speed measurement from the new sensor system. The results show that the maximum error of the speed estimates is less than 2.5% over the entire range of 5-27 m/s (11-60 mi/h). Vehicle classification is done based on the magnetic length and an estimate of the average vertical magnetic height of the vehicle. Vehicle length is estimated from the product of occupancy and estimated speed. The average vertical magnetic height is estimated using two magnetic sensors that are vertically spaced by 0.25 m. Finally, it is shown that the sensor system can be used to reliably count the number of right turns at an intersection, with an accuracy of 95%. The developed sensor system is compact, portable, wireless, and inexpensive. Data are presented from a large number of vehicles on a regular busy urban road in the Twin Cities, MN, USA.

This paper focuses on the use of magnetoresistive and sonar sensors for imminent collision detection in cars. The magnetoresistive sensors are used to measure the magnetic field from another vehicle in close proximity, to estimate relative position, velocity, and orientation of the vehicle from the measurements. First, an analytical formulation is developed for the planar variation of the magnetic field from a car as a function of 2-D position and orientation. While this relationship can be used to estimate position and orientation, a challenge is posed by the fact that the parameters in the analytical function vary with the type and model of the encountered car. Since the type of vehicle encountered is not known a priori, the parameters in the magnetic field function are unknown. The use of both sonar and magnetoresistive sensors and an adaptive estimator is shown to address this problem. While the sonar sensors do not work at very small intervehicle distance and have low refresh rates, their use during a short initial time duration leads to a reliable estimator. Experimental results are presented for both a laboratory wheeled car door and for a full-scale passenger sedan. The results show that planar position and orientation can be accurately estimated for a range of relative motions at different oblique angles.

The integration of rollover prevention and yaw stability control objectives in electronic stability control (ESC) has traditionally been done based on a priority calculation. The control system nominally focuses on yaw stability control until a danger of rollover is detected. When a danger of rollover is detected, the control system switches from yaw stability control to rollover prevention. This paper focuses on an integrated ESC system wherein the objectives of yaw stability and rollover prevention are addressed simultaneously, rather than one at a time. First, we show that staying on a desired planar trajectory at a specified speed results in an invariant rollover index. This implies that rollover prevention can be achieved whenever there is a danger of rollover only by reducing vehicle speed, since changing the desired vehicle trajectory is not a desirable option. In this regard, it is shown that a vehicle that reduces its speed before entering a sharp curve performs significantly better than a vehicle that uses differential braking during the turn for yaw stability control. Second, this paper explores how the use of steer-by-wire technology can address the tradeoff between yaw stability, speed, and rollover prevention performance. It is shown that the use of traditional steer-by-wire simply as an additional actuator cannot by itself ameliorate the tradeoff. However, this tradeoff can be eliminated if steer-by-wire is used to invert the direction of the roll angle of the vehicle. A new steer-by-wire algorithm that uses transient countersteering is shown to change the location of the rollover dynamics from the neighborhood of an unstable to a stable equilibrium. In this case, a desired trajectory can indeed be achieved by the vehicle at the same speed with a much smaller danger of rollover. This is a novel and viable approach to integrating the yaw stability and rollover prevention functions and eliminating the inherent tradeoffs in the performance of both.

This paper focuses on the use of anisotropic magnetoresistive (AMR) sensors for imminent crash detection in cars. The AMR sensors are used to measure the magnetic field from another vehicle in close proximity to estimate relative position and velocity from the measurement. An analytical formulation for the relationship between magnetic field and vehicle position is developed. The challenges in the use of the AMR sensors include their nonlinear behavior, limited range, and magnetic signature levels that vary with each type of car. An adaptive filter based on the iterated extended Kalman filter (IEKF) is developed to automatically tune filter parameters for each encountered car and to reliably estimate relative car position. The utilization of an additional sonar sensor during the initial detection of the encountered vehicle is shown to highly speed up the parameter convergence of the filter. Experimental results are presented from a number of tests with various vehicles to show that the proposed sensor system is viable.

In active rollover prevention systems, a real-time rollover index, which indicates the likelihood of the vehicle to roll over, is used. This paper focuses on state and parameter estimation for reliable computation of the rollover index. Two key variables that are difficult to measure and play a critical role in the rollover index are found to be the roll angle and the height of the center of gravity of the vehicle. Algorithms are developed for real-time estimation of these variables. The algorithms investigated include a sensor fusion algorithm and a nonlinear dynamic observer. The sensor fusion algorithm requires a low-frequency tilt-angle sensor, whereas the dynamic observer utilizes only a lateral accelerometer and a gyroscope. The stability of the nonlinear observer is shown using Lyapunov's indirect method. The performance of the developed algorithms is investigated using simulations and experimental tests. Experimental data confirm that the developed algorithms perform reliably in a number of different maneuvers that include constant steering, ramp steering, double lane change, and sine with dwell steering tests.

This paper presents design, fabrication, and calibration results of MEMS-based two-axis capacitive force sensors capable of resolving forces up to 490/spl mu/N with a resolution of 0.01 /spl mu/N in x, and up to 900 /spl mu/N with a resolution of 0.24 /spl mu/N in y in the passive mode. Electrostatic microactuators are integrated to enable the force sensors to operate in an actively servoed mode, in which system stiffness is modulated using force compensation, greatly increasing force measurement dynamic ranges. When the microforce sensor is actively servoed, an externally applied force is balanced by the electrostatic forces generated by the electrostatic microactuators within the sensor. The movable parts of the sensor are maintained in the equilibrium position, making the system a regulator system. The force measurement is obtained by interpreting the actuation voltages. Probes of different shapes are integrated with the sensors for micromanipulation. Other types of end-effectors, such as microgrippers and microneedles for different micromanipulation tasks can be integrated by modifying the fabrication sequence. The current application of the force sensors is for providing real-time force feedback during microrobotic cell manipulation.

This paper develops a framework for the design and evaluation of spacing policies for adaptive cruise control. Spacing policies are evaluated from the point of view of string stability, traffic flow stability and traffic flow capacity. The standard constant time-gap spacing policy can guarantee string stability but is shown to suffer from poor traffic capacity and traffic flow instability. An "ideal" spacing policy is proposed that evolves naturally from the evaluation framework. The proposed spacing policy is a nonlinear function of speed. It provides string stability and traffic flow stability as well as a higher traffic flow capacity compared to the standard time-gap controller. An associated result proved in the paper is that traffic flow stability implies string stability for flow volumes up to a described maximum value.

This paper presents the design, fabrication, modeling, and control of a dual-axis electrostatic microactuation system. To form the 3D structure only three masks are used on silicon-on-insulator wafers using deep reactive ion etching. The bulk micromachined high aspect ratio structure produces large force output, achieving the full motion range with 10.7 V in x and 70.1 V in y. To provide position feedback for high precision manipulation, a capacitive position sensing mechanism, capable of resolving position changes up to 5 /spl mu/m with a resolution of 0.01 /spl mu/m in both x and y is integrated. A nonlinear model inversion technique is proposed for nonlinear electrostatic microactuation system identification and improving system linearity and response. The effectiveness of the technique was verified in experiments. Applications of the system include micromanipulation and microassembly.