Online Multi-Object Tracking (MOT) from videos is a challenging computer vision task which has been extensively studied for decades. Most of the existing MOT algorithms are based on the Tracking-by-Detection (TBD) paradigm combined with popular machine learning approaches which largely reduce the human effort to tune algorithm parameters. However, the commonly used supervised learning approaches require the labeled data (e.g., bounding boxes), which is expensive for videos. Also, the TBD framework is usually suboptimal since it is not end-to-end, i.e., it considers the task as detection and tracking, but not jointly. To achieve both label-free and end-to-end learning of MOT, we propose a Tracking-by-Animation framework, where a differentiable neural model first tracks objects from input frames and then animates these objects into reconstructed frames. Learning is then driven by the reconstruction error through backpropagation. We further propose a Reprioritized Attentive Tracking to improve the robustness of data association. Experiments conducted on both synthetic and real video datasets show the potential of the proposed model. Our project page is publicly available at: https://github.com/zhen-he/tracking-by-animation

Object detection is important in car sharing services. Accuracy, efficiency, and low memory consumption are desirable for object detection in car sharing services. This paper presents a network system that satisfies all these requirements. Our approach first divides the object detection task into multiple simpler local regression tasks. Then, we propose the generalized Haar filter-based convolutional neural network to reduce the consumption of memory and computing resource. To achieve real-time performance, we introduce a sparse window generation strategy to reduce the number of input image patches without sacrificing accuracy. We perform experiments on both vehicle and pedestrian data sets. Experimental results demonstrate that our approach can accurately detect objects under challenging conditions. Note to Practitioners-Object detection is an important part of intelligent vehicle technologies, which play an important role in car sharing services. Object detection provides metadata for collision avoidance, self-driving systems, and driver-assistance systems, which can result in better safety and consumer experiences in car sharing services. Although deep learning has achieved an excellent performance in object detection, they consume a large amount of storage and computing resource, which makes them difficult to be deployed for car sharing services. This paper suggests a novel approach which is based on the generalized Haar filter and the local regression strategy. Our approach is accurate, efficient, and light. The experimental results verify the effectiveness of the proposed approach in car sharing services.

Long Short-Term Memory (LSTM) is a popular approach to boosting the ability of Recurrent Neural Networks to store longer term temporal information. The capacity of an LSTM network can be increased by widening and adding layers. However, usually the former introduces additional parameters, while the latter increases the runtime. As an alternative we propose the Tensorized LSTM in which the hidden states are represented by tensors and updated via a cross-layer convolution. By increasing the tensor size, the network can be widened efficiently without additional parameters since the parameters are shared across different locations in the tensor; by delaying the output, the network can be deepened implicitly with little additional runtime since deep computations for each timestep are merged into temporal computations of the sequence. Experiments conducted on five challenging sequence learning tasks show the potential of the proposed model.

This paper presents a hierarchical model for robust image representation. We first introduce multi-level sparse coding algorithm and normalized max pooling strategy which are designed to obtain meaningful sparse codes and robust pooled codes, respectively. With the sparse codes and pooled codes, a hierarchical architecture is built and more robust features are extracted at the second layer. The proposed method has been evaluated on two widely used datasets: Caltech-101 and Caltech-256, and experimental results demonstrate that the proposed method is both effective and robust in image representation compared with the state-of-the-art.

Road detection is a crucial problem for autonomous navigation system (ANS) and advance driver-assistance system (ADAS). In this paper, we propose a hierarchical road detection method for robust road detection in challenging scenarios. Given an on-board road image, we first train a Gaussian mixture model (GMM) to obtain road probability density map (RPDM), and next oversegment the image into superpixels. Based on RPDM and superpixels, initial seeds are selected in an unsupervised way, and the seed superpixels iteratively try to occupy their neighbors according to GrowCut framework, the road segment is obtained after convergency. Finally, we refine the road segment with a conditional random field (CRF), which enforces the shape prior on the road segmentation task. Experiments on two challenging databases demonstrate that the proposed method exhibits high robustness compared with the state-of-the-art.

Many road detection algorithms require pre-learned information, which may be unreliable as the road scene is usually unexpectable. Single image based (i.e., without any pre-learned information) road detection techniques can be adopted to overcome this problem, while their robustness needs improving. To achieve robust road detection from a single image, this paper proposes a general road shape prior to enforce the detected region to be road-shaped by encoding the prior into a graph-cut segmentation framework, where the training data is automatically generated from a predicted road region of the current image. By iteratively performing the graph-cut segmentation, an accurate road region will be obtained. Quantitative and qualitative experiments on the challenging SUN Database validate the robustness and efficiency of our method. We believe that the road shape prior can also be used to yield improvements for many other road detection algorithms.

We address the issue of visual saliency from three perspectives. First, we consider saliency detection as a frequency domain analysis problem. Second, we achieve this by employing the concept of nonsaliency. Third, we simultaneously consider the detection of salient regions of different size. The paper proposes a new bottom-up paradigm for detecting visual saliency, characterized by a scale-space analysis of the amplitude spectrum of natural images. We show that the convolution of the image amplitude spectrum with a low-pass Gaussian kernel of an appropriate scale is equivalent to an image saliency detector. The saliency map is obtained by reconstructing the 2D signal using the original phase and the amplitude spectrum, filtered at a scale selected by minimizing saliency map entropy. A Hypercomplex Fourier Transform performs the analysis in the frequency domain. Using available databases, we demonstrate experimentally that the proposed model can predict human fixation data. We also introduce a new image database and use it to show that the saliency detector can highlight both small and large salient regions, as well as inhibit repeated distractors in cluttered images. In addition, we show that it is able to predict salient regions on which people focus their attention.

To address the path tracking problem of autonomous land vehicle, a new vehicle-road model named “Ribbon Model” is constructed under the constraints of road width and vehicle geometry structure. A new vehicle-road evaluation algorithm is developed based on this model, and new path tracking controller is designed. The difficulties of preview distance selection and parameters tuning with speed of pure following controller are avoided in this controller. Performance of the novel method is verified by simulation and vehicle experiments.

Kernel-based feature combination techniques such as Multiple Kernel Learning use arithmetical operations to linearly combine different kernels. We have observed that the kernel distributions of different features are usually very different. We argue that the similarity distributions amongst the data points for a given dataset should not change with their representation features and propose the concept of relative kernel distribution invariance (RKDI). We have developed a very simple histogram matching based technique to achieve RKDI by transforming the kernels to a canonical distribution. We have performed extensive experiments on various computer vision and machine learning datasets and show that calibrating the kernels to an empirically chosen canonical space before they are combined can always achieve a performance gain over state-of-art methods. As histogram matching is a remarkably simple and robust technique, the new method is universally applicable to kernel-based feature combination.

We propose a new saliency detection model by combining global information from frequency domain analysis and local information from spatial domain analysis. In the frequency domain analysis, instead of modeling salient regions, we model the nonsalient regions using global information; these so-called repeating patterns that are not distinctive in the scene are suppressed by using spectrum smoothing. In spatial domain analysis, we enhance those regions that are more informative by using a center-surround mechanism similar to that found in the visual cortex. Finally, the outputs from these two channels are combined to produce the saliency map. We demonstrate that the proposed model has the ability to highlight both small and large salient regions in cluttered scenes and to inhibit repeating objects. Experimental results also show that the proposed model outperforms existing algorithms in predicting objects regions where human pay more attention. 1

Task-specified computer vision systems usually need to detect certain targets at multiple scales of resolution and multiple orientations. For a vision-based lane detection system, it is essential to detect the lane-markings at different scales and orientations. In this paper, we illustrate an efficient local feature extraction algorithm for the lane detection system, which is tuned by the high-level information about the lane-markings. Firstly, we deduced the explicit expression of the scale and orientation for the local feature of the lane markings. Secondly, a filter bank for local feature extraction is designed using the SVD approach for certain orientation and scale. Thirdly, the filter bank is used to tune a special lane-marking detector to expected orientation and scale at different locations of the image. Then, non-maxima suppression is performed along the corresponding direction at that location. Lastly, a hysteresis thresholding is applied to identify the exact feature points. Unlike other works in which the authors try to remove the false local feature points with the help of high-level information, we prefer to introduce the high-level information to the local feature detection stage as early as possible. Experiment results show that the proposed algorithm is very efficient for lane detection especially in very complex road seniors.

Vision based lane detection is an essential task in both autonomous lane vehicles research and active safety system development. Hitherto, lane detection is, however, still a challenging issue due to the complexity of the real road scenes. In this paper, we consider lane detection as a visual attention problem. With a Bayesian attention framework, we address the issue from three perspectives: first, lane markings are assumed to be salient in the road scenes, which will pop out driven by the low level features combining with a bottom-up attention mechanism, second, the target-related features are designed guided by a top-down attention strategy, third, the location prior of the lane markings is also investigated. The experimental results show that the proposed lane detection approach is efficient and robust in real scenes.

This paper presents a real-time long-range lane detection and tracking approach to meet the requirements of the high-speed intelligent vehicles running on highway roads. Based on a linear-parabolic two-lane highway road model and a novel strong lane marking feature named Lane Marking Segmentation, the maximal lane detection distance of this approach is up to 120 meters. Then the lane lines are selected and tracked by estimating the ego vehicle lateral offset with a Kalman filter. Experiment results with test dataset extracted from real traffic scenes on highway roads show that the approaches proposed in this paper can achieve a high detection rate with a low time cost.

Vision-based lane detection is a key component for Driver-Assistance (DA) systems. It is still a challenging task in road scenes with complex shadows. This paper presents a novel local edge detector, using vanishing point position as a high level information to guide the use of steerable filters in lane detection, and its implementation on a Field Programmable Gate Array (FPGA) device. The FPGA technology has the advantages of high-performances for digital image processing and low cost, both of which are the requirements of DA systems. The main contributions of this work are twofold: 1) an edge extraction algorithm for lane detection is proposed, using the estimated vanishing point as high-level information to detect lanes. Firstly, a rough estimation of the vanishing point is used for calculating the expected local edge orientations. Secondly, a steerable filter is tuned to the expected direction for edge response. 2) a framework on FPGA is designed to implement the proposed algorithm. The framework is designed by using multi-engine technology, so it works in parallel for any order of steerable filters. Experiments and comparisons show that the proposed algorithm is very efficient in dealing with the complex shadow conditions, and works in real-time on FPGA device.

This paper describes a novel approach for fast detecting small maritime objects in infrared (IR) images. It is based on the local minimum patterns (LMP), which are theoretically the approximations of some stationary wavelet transforms (SWT). Using LMP to estimate the background with a single image, we obtain an object-aware saliency map by background subtraction. Regions of potential objects are then segmented by an adaptive threshold based on the histogram of the saliency map. We finally propose a fast clustering algorithm for localizing objects from segmented regions. Extensive experiments on challenging data sets show a competitive performance.

In vision-based driver assistance applications, the onboard cameras are supposed to be calibrated as simply and accurately as possible. This paper presents an interactive method for extrinsic parameter calibration of onboard camera. In the proposed method, the intrinsic camera parameters are assumed to be known. During the calibration process, we first design a calibration scene to compute the extrinsic camera parameters. There are three lines in the scene, two of which are parallel with the vehicle heading direction and the other one is perpendicular with the first two. Then, using this scene, we derive an interactive and analytical solution to obtain the extrinsic camera parameters, which can adequately compromise the calibration complexity and accuracy. We also extend the method to multiple cameras, in which case the calibration can be performed without absolute positions of the calibration lines. Theoretical and experimental results show the efficiency of the proposed calibration method.

Linear features such as line segments and contour fragments are important cues for object detection and scene analysis. Least square based and Hough-like approaches are quite popular and powerful. However, least square approaches are sensitive to outliers, and are unable to handle the case where there is more than one underlying line segment; while Hough-like approaches do not work well when extracting fragments which are not very `straight'. The goal of this work is to extract linear features locally in natural scenes using PCA and graph analysis. The whole process has five stages: first, for each local area, we create a graph using edge points as vertexes and distance measurements as links. Then, some edge-point sets which may contain underlying line segments are obtained by finding connected components in the graph; second, PCA is employed to estimate the existence of linear feature in each point set, and to compute the principal axis orientation; third, the relation between any two vertexes in each set is recalculated according to the principal axis orientation, and a new graph which only contains several trees is generated; fourth, one linear feature is detected by finding a best chain in each tree according to some criteria; fifth, all the line segments in the image are refined and organized according to global information.

With the fast development of robotics and intelligent vehicles, there has been much research work on modeling and motion control of autonomous vehicles. However, due to model complexity, and unknown disturbances from dynamic environment, the motion control of autonomous vehicles is still a difficult problem. In this paper, a novel self-learning path-tracking control method is proposed for a car-like robotic vehicle, where kernel-based approximate dynamic programming (ADP) is used to optimize the controller performance with little prior knowledge on vehicle dynamics. The kernel-based ADP method is a recently developed reinforcement learning algorithm called kernel least-squares policy iteration (KLSPI), which uses kernel methods with automatic feature selection in policy evaluation to get better generalization performance and learning efficiency. By using KLSPI, the lateral control performance of the robotic vehicle can be optimized in a self-learning and data-driven style. Compared with previous learning control methods, the proposed method has advantages in learning efficiency and automatic feature selection. Simulation results show that the proposed method can obtain an optimized path-tracking control policy only in a few iterations, which will be very practical for real applications.

In this paper, a novel hybrid learning method is proposed for reinforcement learning problems with continuous state and action spaces. The reinforcement learning problems are modeled as Markov decision processes (MDPs) and the hybrid learning method combines evolutionary algorithms with gradient-based adaptive heuristic critic (AHC) algorithms to approximate the optimal policy of MDPs. The suggested method takes the advantages of evolutionary learning and gradient-based reinforcement learning to solve reinforcement learning problems. Simulation results on the learning control of an acrobot illustrate the efficiency of the presented method.

The recursive least-squares (RLS) algorithm is one of the most well-known algorithms used in adaptive filtering, system identification and adaptive control. Its popularity is mainly due to its fast convergence speed, which is considered to be optimal in practice. In this paper, RLS methods are used to solve reinforcement learning problems, where two new reinforcement learning algorithms using linear value function approximators are proposed and analyzed. The two algorithms are called RLS-TD(lambda) and Fast-AHC (Fast Adaptive Heuristic Critic), respectively. RLS-TD(lambda) can be viewed as the extension of RLS-TD(0) from lambda=0 to general lambda within interval [0,1], so it is a multi-step temporal-difference (TD) learning algorithm using RLS methods. The convergence with probability one and the limit of convergence of RLS-TD(lambda) are proved for ergodic Markov chains. Compared to the existing LS-TD(lambda) algorithm, RLS-TD(lambda) has advantages in computation and is more suitable for online learning. The effectiveness of RLS-TD(lambda) is analyzed and verified by learning prediction experiments of Markov chains with a wide range of parameter settings. The Fast-AHC algorithm is derived by applying the proposed RLS-TD(lambda) algorithm in the critic network of the adaptive heuristic critic method. Unlike conventional AHC algorithm, Fast-AHC makes use of RLS methods to improve the learning-prediction efficiency in the critic. Learning control experiments of the cart-pole balancing and the acrobot swing-up problems are conducted to compare the data efficiency of Fast-AHC with conventional AHC. From the experimental results, it is shown that the data efficiency of learning control can also be improved by using RLS methods in the learning-prediction process of the critic. The performance of Fast-AHC is also compared with that of the AHC method using LS-TD(lambda). Furthermore, it is demonstrated in the experiments that different initial values of the variance matrix in RLS-TD(lambda) are required to get better performance not only in learning prediction but also in learning control. The experimental results are analyzed based on the existing theoretical work on the transient phase of forgetting factor RLS methods.