Predicting urban traffic flow is a challenging task, due to the complicated spatio-temporal dependencies on traffic networks. Urban traffic flow usually has both short-term neighboring and long-term periodic temporal dependencies. It is also noticed that the spatial correlations over different traffic nodes are both local and non-local. What’s more, the traffic flow is affected by various external factors. To capture the non-local spatial correlations, we propose a Dilated Attentional Graph Convolution (DAGC). The DAGC utilizes a dilated graph convolution kernel to expand the nodes’ receptive field and exploit multi-order neighborhood. Technically, the lower-order neighborhood corresponds to local spatial dependencies, while the higher-order neighborhood corresponds to non-local spatial dependencies between nodes. Based on DAGC, a Multi-Source Spatio-Temporal Network (MS-Net) is designed, which suffices to integrate long-range historical traffic data as well as multi-modal external information. MS-Net consists of four components: a spatial feature extraction module, a temporal feature fusion module, an external factors embedding module, and a multi-source data fusion module. Extensive experiments on three real traffic datasets demonstrates that the proposed model performs well on both the public transportation networks, road networks, and can handle large-scale traffic networks in particular the Beijing bus network which has more than 4,000 traffic nodes.

Glass-like objects such as windows, bottles, and mirrors exist widely in the real world. Sensing these objects has many applications, including robot navigation and grasping. However, this task is very challenging due to the arbitrary scenes behind glass-like objects. This paper aims to solve the glass-like object segmentation problem via enhanced boundary learning. In particular, we first propose a novel refined differential module that outputs finer boundary cues. We then introduce an edge-aware point-based graph convolution network module to model the global shape along the boundary. We use these two modules to design a decoder that generates accurate and clean segmentation results, especially on the object contours. Both modules are lightweight and effective: they can be embedded into various segmentation models. In extensive experiments on three recent glass-like object segmentation datasets, including Trans10k, MSD, and GDD, our approach establishes new state-of-the-art results. We also illustrate the strong generalization properties of our method on three generic segmentation datasets, including Cityscapes, BDD, and COCO Stuff. Code and models will be available for further research.

Skeleton-based gesture recognition is very challenging, as the high-level information in gesture is expressed by a sequence of complexly composite motions. Previous works often learn all the motions with a single model. In this paper, we propose to decouple the gesture into hand posture variations and hand movements, which are then modeled separately. For the former, the skeleton sequence is embedded into a 3D hand posture evolution volume (HPEV) to represent fine-grained posture variations. For the latter, the shifts of hand center and fingertips are arranged as a 2D hand movement map (HMM) to capture holistic movements. To learn from the two inhomogeneous representations for gesture recognition, we propose an end-to-end two-stream network. The HPEV stream integrates both spatial layout and temporal evolution information of hand postures by a dedicated 3D CNN, while the HMM stream develops an efficient 2D CNN to extract hand movement features. Eventually, the predictions of the two streams are aggregated with high efficiency. Extensive experiments on SHREC'17 Track, DHG-14/28 and FPHA datasets demonstrate that our method is competitive with the state-of-the-art.

Transferring image-based object detectors to the domain of videos remains a challenging problem. Previous efforts mostly exploit optical flow to propagate features across frames, aiming to achieve a good trade-off between accuracy and efficiency. However, introducing an extra model to estimate optical flow can significantly increase the overall model size. The gap between optical flow and high-level features can also hinder it from establishing spatial correspondence accurately. Instead of relying on optical flow, this paper proposes a novel module called Progressive Sparse Local Attention (PSLA), which establishes the spatial correspondence between features across frames in a local region with progressively sparser stride and uses the correspondence to propagate features. Based on PSLA, Recursive Feature Updating (RFU) and Dense Feature Transforming (DenseFT) are proposed to model temporal appearance and enrich feature representation respectively in a novel video object detection framework. Experiments on ImageNet VID show that our method achieves the best accuracy compared to existing methods with smaller model size and acceptable runtime speed.

Existing conditional video prediction approaches train a network from large databases and generalise to previously unseen data. We take the opposite stance, and introduce a model that learns from the first frames of a given video and extends its content and motion, to, e.g., double its length. To this end, we propose a dual network that can use in a flexible way both dynamic and static convolutional motion kernels, to predict future frames. We demonstrate experimentally the robustness of our approach on challenging videos in-the-wild and show that it is competitive w.r.t related baselines.

We propose a robust method for estimating road curb 3-D parameters (size, location, and orientation) using a calibrated monocular camera equipped with a fisheye lens. Automatic curb detection and localization is particularly important in the context of an advanced driver assistance system, i.e., to prevent possible collision and damage to the vehicle's bumper during perpendicular and diagonal parking maneuvers. Combining 3-D geometric reasoning with advanced vision-based detection methods, our approach is able to estimate the vehicle to curb distance in real time with a mean accuracy of more than 90%, as well as its orientation, height, and depth. Our approach consists of two distinct components-curb detection in each individual video frame and temporal analysis. The first part is comprised of sophisticated curb edges extraction and parameterized 3-D curb template fitting. Using a few assumptions regarding the real-world geometry, we can thus retrieve the curb's height and its relative position with respect to the moving vehicle on which the camera is mounted. Support vector machine classifier fed with histograms of oriented gradients is used for appearance-based filtering out outliers. In the second part, the detected curb regions are tracked in the temporal domain, so as to perform a second pass of false positives rejection. We have validated our approach on a newly collected database of 11 videos under different conditions. We have used point-wise LIDAR measurements and manual exhaustive labels as a ground truth.

We extract 3D curb from video sequence, using a single camera equipped with fish-eye lens and located at the front/rear of the vehicle. The challenge in extracting curbs from images lies in their small size and their lack of texture. We show that by appropriately exploiting appearance features, 3D geometry, and temporal information, one can reliably detect and localize the curbs in the 3D scene. The main underlying assumption of our model is that the road surface is flat and that the curb is approximately orthogonal to the road plane. We collected nine videos with ground truth, under day-time sunny weather condition, up to 2m range. Our experimental results compare favorably wrt the current the state-of-the-art on our database -90% precision rate in average and over 85% accuracy in curb height estimation.

We estimate illuminant chromaticity from temporal sequences, for scenes illuminated by either one or two dominant illuminants. While there are many methods for illuminant estimation from a single image, few works so far have focused on videos, and even fewer on multiple light sources. Our aim is to leverage information provided by the temporal acquisition, where either the objects or the camera or the light source are/is in motion in order to estimate illuminant color without the need for user interaction or using strong assumptions and heuristics. We introduce a simple physically-based formulation based on the assumption that the incident light chromaticity is constant over a short space-time domain. We show that a deterministic approach is not sufficient for accurate and robust estimation: however, a probabilistic formulation makes it possible to implicitly integrate away hidden factors that have been ignored by the physical model. Experimental results are reported on a dataset of natural video sequences and on the Gray Ball benchmark, indicating that we compare favorably with the state-of-the-art.

We propose a novel method for detecting and enhancing specular highlights from a video sequence, thereby obtaining a better visual perception of the specularity. To this end, we first generate a specularity map, defined over the space-time domain, by leveraging information provided by the temporal acquisition. We then amplify the highlights in each video frame to create the visual sensation of high dynamic range data. Results are illustrated on several videos taken under different acquisition scenarios.

Satellite observations are currently of major importance in geosciences. Remote sensing is a strong tool to study atmospheric and earth phenomena. In this work, we propose a new motion estimation approach with application to dust storm tracking from remote sensing images. Dynamic natural phenomena in the atmosphere are generally turbulent due to a high Reynolds number. Meteorological images are still poor in time and space resolution compared to the turbulence characteristics of the flow. To tackle this problem, we define a new formulation of the flow equation based on a filtered scalar transport equation. Using Large Eddy Simulation theory, we propose a sub-grid model which incorporates small scale effects as missing (ie non-observed) information of remote sensing images. For day light changes, a uniform brightness variation term is incorporated to the model. We validated our approach on synthetic Direct Numerical Simulation (DNS) of scalar propagation. Promising results are obtained on real MTSAT-1R visible images of a dust storm event over Australia.

Optical flow estimation is one of the main subjects in computer vision. Many methods developed to compute the motion fields are built using standard heuristic formulation. In this paper, however, we learn a motion model. We develop a hybrid model by combining the learnt model with Markov Random Field (MRF). And then we introduce a method based on "Radial Basis Function Neural Network" (RBF) to learn the model. When computing the displacement field, a Gaussian pyramidal down-sampling decomposition technique is employed. At each pyramidal level, we use bi-linear interpolation combined with an efficient warping technique to generate a residual image, which is then used at the finer level to compute the flow. To minimize the energy, we use two different discrete optimization methods: Graph-Cut algorithm, Tree-Reweighted Message Passing (TRW-S) algorithm. Results are demonstrated for our approach on synthetic images and fluid images from the real world.

The present work concerns the analysis of dynamic scenes from earth observation images. We are interested in building a map which, on one hand locates places of change, on the other hand, reconstructs a unique visual information of the non-change areas. We show in this paper that such a problem can naturally be takled with conditional mixed-state random field modeling (mixed-state CRF), where the ¿mixed state¿ refers to the symbolic or continous nature of the unknown variable. The maximum a posteriori (MAP) estimation of the CRF is, through the Hammersley-Clifford theorem, turned into an energy minimisation problem. We tested the model on several Quickbird images and illustrate the quality of the results.

For its simplicity and efficiency, the bag-of-words representation based on appearance features is widely used in image and text classification. Its drawback is that shape patterns of the image are neglected. This paper presents a novel image classification approach using a bag-of-words representation of textons while taking into account spatial information. A generative probabilistic modeling of the distribution of textons is proposed. The parameters of the mixture’s components are estimated using a EM algorithm. We show that the number of classes in a database can be found automatically and exactly by MDL. This modeling gives very good results for the task of weakly supervised classification in satellite images. 1

We introduce a new image representation that encompasses both the general layout of groups of quantized local invariant descriptors as well as their relative frequency. A graph of interest points clusters is constructed and we use the matrix of commute times between the different nodes of the graph to obtain a description of their relative arrangement that is robust to large intra class variation. The obtained high dimensional representation is then embedded in a space of lower dimension by exploiting the spectral properties of the graph made of the different images. Classification tasks can be performed in this embedding space. We expose classification and labelling results obtained on three different datasets, including the challenging PASCAL VOC2007 dataset. The performances of our approach compare favorably with the standard bag of features, which is a particular case of our representation.

We propose in this paper a robust multi-resolution technique to estimate dense velocity field from image sequences. It couples a Gaussian pyramidal down-sampling decomposition together with a multi-grid approach. At each pyramid level, bilinear interpolation and efficient warping techniques are performed to generate a residual images. The displacement field is computed in a Markov random field (MRF) framework. We compare two different methods to minimize the Gibbs energy: a modified iterative conditional mode (ICM) and a graph-cut algorithm extended to multi-grid scheme. We validate and demonstrate the robustness of our approach on synthetic and real images for fluid experiment applications.

In this paper we provide an overview of an image representation approach based on the description of layout and appearance properties of groups of features. In each image a graph of quantized features of interest is constructed. The features that are assigned to the same codebook bin are then grouped to produce a collapsed graph; the image content is represented by the matrix of commute times of this collapsed graph. This novel image descriptor can be used to label satellite image databases; we demonstrate the relevance and the efficiency of our approach by addressing classification problems on a dataset of 0.6 m resolution Quick bird images.

We address the problem of updating road maps in dense urban areas by extracting the main road network from a very high resolution (VHR) satellite image. Our model of the region occupied by the road network in the image is innovative. It incorporates three different types of prior geometric knowledge: generic boundary smoothness constraints, equivalent to a standard active contour prior; knowledge of the geometric properties of road networks (i.e. that they occupy regions composed of long, low-curvature segments joined at junctions), equivalent to a higher-order active contour prior; and knowledge of the road network at an earlier date derived from GIS data, similar to other ‘shape priors’ in the literature. In addition, we represent the road network region as a ‘phase field’, which offers a number of important advantages over other region modelling frameworks. All three types of prior knowledge prove important for overcoming the complexity of geometric ‘noise’ in VHR images. Promising results and a comparison with several other techniques demonstrate the effectiveness of our approach.

We propose in this paper a new formulation of the equation of the optical flow enabling to compute global and local motions of multi-structure objects (flowers and petals, trees and leaves). The displacement fields are computed using a Markovian random field (MRF) model. Local and global components of the vector flow are both explicitly retrieved. The minimization of the Gibbs energy is achieved with a down-scaling approach, in which we first analyze the motion of the compact object, the sub-structures movement being retrieved in a second stage. We validate and demonstrate the efficiency of our approach on synthetic and real images for various applications.

Landuse classification is an important problem in the remote sensing field. It can be used in a wide range of applications. In this paper, we propose a hybrid method fusing edges and regions information for the landuse classification of multispectral images. It mainly includes the steps of image pre-processing, initial segmentation and region merging. Especially, a novel spatial mean shift procedure is proposed so that some information can be extracted and used in the successive steps. Aiming at the multispectral images processing, we also design a band weighting strategy that give a proper weight to each band adoptively according to the region to be processed. Experimental results on the Landsat TM and ETM+ images validate the performance of the proposed method.

This paper proposes a probability model and efficient scheme to extract buildings object from high-resolution panchromatic image in dense urban area. The overall idea of this work is to segment the image into regions, to treat all extracted regions' contours as candidates, and to make use of specific probability model to select 'true' buildings. Discriminative features that characterize buildings are proposed. Application is performed on Quickbird images over Beijing city.

Three fusion methods are presented for landuse classification in this paper. They make fully use of the spectral and terrain information obtained from the optical image and DEM, and combine the remarkable features of each source in the final classification. Their performance is demonstrated by experiments and a comparison is made. Keywords-image fusion;landuse classification;DEM

A building extraction method based on segmentation is proposed. The high-resolution satellite imagery and GIS data are used in our application. GIS data provides the prior knowledge of building to segmentation algorithm. The method consists of three steps: data preprocessing, objects segmentation using the fuzzy connectedness segmentation algorithm and results post-processing. Experimental results show the efficiency of the method but improvement is still needed.

For interferometric SAR processing (InSAR), the quality of the interferogram is affected by many factors such as Doppler centroid difference, baseline, atmospheric effect. It can be measured by the estimation of the coherence of the data. It has been shown that the coherence image can be used to do the classification in the area such as water, soil, plants, city or something else. But the coherence estimation is always biased somehow. To find an asymptotically unbiased coherence estimation method is therefore challenging and important for InSAR processing. In this paper, several different coherence estimators are introduced. An accurate coherence computation method using an existing DEM is proposed and illustrated

Baseline is a very important parameter in SAR interferometry (InSAR) processing and applications. It impact generation and quality of DEM from interferogram. Although a variety of baseline estimation methods are reported in the literature, to our knowledge, a systematical comparison on such methods is still unavailable. In this paper, some representative methods of baseline estimation are compared and their advantages and disadvantages are assessed and outlined. The comparison is carried out with ERS-1/2 tandem data over the Jingjiang area of China.

In interferometric synthetic aperture radar (InSAR) processing, simulation of interferogram is a common practice. It is used as synthetic data to test and validate the whole chain of InSAR processing from the interferogram creation to the DEM reconstruction. Simulators for interferometry, those have been reported in the literature, are developed in very simplified imagery geometry model. In this paper, a interferogram simulator which deal with real radar sensor parameters and orbit data is presented.

A new phase unwrapping method based on active contour model is presented in this paper. Active contour model offers a general frame for the edge feature extraction. It defines an initial contour by some high level information, and then deforms it towards the desired feature. Then contour map is obtained, on which the fringe lines are continuous and closed. Actually the interferogram is divided into some blocks by the fringe lines and the border of the image. Finally, we do phase unwrapping for the whole image block by block. The advantages of our method are its robust to image noise and easy contour initialization. Experiment on the real data and the comparison with one traditional method demonstrate the efficiency and robust of our method.

A dike detection method based on active contour model is introduced in this paper for our flood management project. Experimental result on DEM (Digital Elevation Model) of Jingjiang River of China shows that a continuous and closed dike description could be gotten by this method.

This paper introduces an original entropy-based method for water body extraction from multi-source satellite images. The proposed method is remarkably simple and relies only on the assumption that water body in the image is a smooth area. This assumption is most of the time verified in optical and radar images, which makes the approach insensitive to image sensors. Experimental results illustrate the robustness of the proposed approach.

A rough-to-fine satellite image registration method that can realize robust alignment with sub-pixel accuracy is described. The registration procedure is composed of several steps. Firstly, a globally rough registration using single-template normalized crosscorrelation is realized to get an overlapped area. Secondly, multiple sub-images are randomly generated in the roughly overlapped area, and templates are selected according to empirical entropy. In the third step, a locally rough-to-fine registration is performed in order to get tie-points, and during this step, an optimal matching accuracy degree is achieved by means of "gradient descent" searching method. The last part is the estimation of mapping function and image transformation. Experimental results on satellite optical images validate the efficiency and accuracy of the proposed algorithm.

A new method derived from the Frost filter kernel for SAR (synthetic aperture radar) image speckle reduction is introduced. In the speckle reduction procedure, both the statistical information of the central pixel in the filter window and that of the neighborhood pixels are used. The experimental results on ERS-1 PRI images demonstrate the efficiency of the proposed method compared to other common filters.

Thin network extraction from three dimensional (3D) images is a new issue in image processing. It is of major importance in medical vascular imaging for diagnostics, therapy planning and surgery. Here, the authors develop a framework for automatic vascular network extraction from the volumic image. The approach consists in treating the 3D image as a hyper-surface of IR/sup 4/. It is shown that the crest points of this hyper-surface correspond to the center line of the thin network in the image. Promising results are shown on synthetic and real vascular images.

Thin network extraction from three dimensional images is a new issue in computer vision. It is of major importance in medical vascular imaging for diagnostic, therapy planning and surgery. In this paper, we develop a framework for automatic thin network extraction from the volumic image. The approach consists in treating the 3D image as a hyper-surface of IR/sup 4/. It is shown that the crest points of this hyper-surface correspond to the center line of the thin network in the image. Using a simple mathematical model, we establish the relationship between the computed principal curvatures of the hyper-surface and the geometry of the network. Promising results are shown on synthetic and real vascular images.