Spiking neural networks (SNNs) are rich in spatio-temporal dynamics and are suitable for processing event-based neuromorphic data. However, event-based datasets are usually less annotated than static datasets. This small data scale makes SNNs prone to overfitting and limits their performance. In order to improve the generalization ability of SNNs on event-based datasets, we use static images to assist SNN training on event data. In this paper, we first discuss the domain mismatch problem encountered when directly transferring networks trained on static datasets to event data. We argue that the inconsistency of feature distributions becomes a major factor hindering the effective transfer of knowledge from static images to event data. To address this problem, we propose solutions in terms of two aspects: feature distribution and training strategy. Firstly, we propose a knowledge transfer loss, which consists of domain alignment loss and spatio-temporal regularization. The domain alignment loss learns domain-invariant spatial features by reducing the marginal distribution distance between the static image and the event data. Spatio-temporal regularization provides dynamically learnable coefficients for domain alignment loss by using the output features of the event data at each time step as a regularization term. In addition, we propose a sliding training strategy, which gradually replaces static image inputs probabilistically with event data, resulting in a smoother and more stable training for the network. We validate our method on neuromorphic datasets, including N-Caltech101, CEP-DVS, and N-Omniglot. The experimental results show that our proposed method achieves better performance on all datasets compared to the current state-of-the-art methods. Code is available at https://github.com/Brain-Cog-Lab/Transfer-for-DVS.

Extracting robust and discriminative local features from images plays a vital role for long term visual localization, whose challenges are mainly caused by the severe appearance differences between matching images due to the day-night illuminations, seasonal changes, and human activities. Existing solutions resort to jointly learning both keypoints and their descriptors in an end-to-end manner, leveraged on large number of annotations of point correspondence which are harvested from the structure from motion and depth estimation algorithms. While these methods show improved performance over non-deep methods or those two-stage deep methods, i.e., detection and then description, they are still struggled to conquer the problems encountered in long term visual localization. Since the intrinsic semantics are invariant to the local appearance changes, this paper proposes to learn semantic-aware local features in order to improve robustness of local feature matching for long term localization. Based on a state of the art CNN architecture for local feature learning, i.e., ASLFeat, this paper leverages on the semantic information from an off-the-shelf semantic segmentation network to learn semantic-aware feature maps. The learned correspondence-aware feature descriptors and semantic features are then merged to form the final feature descriptors, for which the improved feature matching ability has been observed in experiments. In addition, the learned semantics embedded in the features can be further used to filter out noisy keypoints, leading to additional accuracy improvement and faster matching speed. Experiments on two popular long term visual localization benchmarks (Aachen Day and Night v1.1, Robotcar Seasons) and one challenging indoor benchmark (InLoc) demonstrate encouraging improvements of the localization accuracy over its counterpart and other competitive methods.

This paper performs a comprehensive and comparative evaluation of the state-of-the-art local features for the task of image-based 3D reconstruction. The evaluated local features cover the recently developed ones by using powerful machine learning techniques and the elaborately designed handcrafted features. To obtain a comprehensive evaluation, we choose to include both float type features and binary ones. Meanwhile, two kinds of datasets have been used in this evaluation. One is a dataset of many different scene types with groundtruth 3D points, containing images of different scenes captured at fixed positions, for quantitative performance evaluation of different local features in the controlled image capturing situation. The other dataset contains Internet scale image sets of several landmarks with a lot of unrelated images, which is used for qualitative performance evaluation of different local features in the free image collection situation. Our experimental results show that binary features are competent to reconstruct scenes from controlled image sequences with only a fraction of processing time compared to using float type features. However, for the case of a large scale image set with many distracting images, float type features show a clear advantage over binary ones. Currently, the most traditional SIFT is very stable with regard to scene types in this specific task and produces very competitive reconstruction results among all the evaluated local features. Meanwhile, although the learned binary features are not as competitive as the handcrafted ones, learning float type features with CNN is promising but still requires much effort in the future.

Deep convolutional neural networks (CNN) has achieved state-of-the-art result on traffic sign classification, which plays a key role in intelligent transportation system. However, it usually requires a large number of labeled training data, which is not always available, to guarantee a good performance. In this paper, we propose to synthesize traffic sign images by generative adversarial networks (GANs). It takes a standard traffic sign template and a background image as input to the generative network in GANs, where the template defines which class of traffic sign to include and the background image controls the visual appearance of the synthetic images. Experiments show that our method could generate more realistic traffic sign images than the conventional image synthesis method. Meanwhile, by adding the synthesis images to train a typical CNN for traffic sign classification, we obtained a better accuracy.

The human stereo vision process begins in primary visual cortex where complex cells are deemed as disparity detectors. Besides correct matches, complex cells could also respond to false matches which cause ambiguous depth perception. This reveals that there exist some inherent mechanisms in the higher visual processing areas for eliminating false matches and thus recovering true disparity. Due to the least understanding of this hierarchical process, this paper aims to investigate this problem by proposing a deep stereo vision model. There are three characteristics of the proposed model: 1) the obtained disparity maps become more accurate from lower layers to higher layers; 2) it tends to promote neural responses of correct matches while inhibiting those of incorrect matches; 3) it generalizes well to unseen data.

Early stages of the human visual system consist of retinal cones, retinal ganglion cells(RGC), lateral geniculate nucleus(LGN) and V1. Modeling early visual stages is conducive to reveal the mechanism of visual signal preprocessing and representation inside brain, as well as settle challenges artificial intelligence confronts. However, a majority of previous work often models RGC/LGN or V1 separately, seldom modeling them together hierarchically. In order to be consistent with the biological results, we propose HEVS (a Hierarchical computational model for Early stages of the Visual System), a feedforward neural network composed of three layers, which represent receptor neurons, RGC/LGN and V1 successively. Exactly as the visual system, the proposed locally connected model is derived in the unsupervised scenario on natural images and trained in the bottom-up order. In order to learn the two connection weights among three layers, we formulate two optimization problems based on the reconstruction error and sparse learning. Unlike traditional models on RGC/LGN, we perform weighted similarity measuring as a regular term to simulate the strong correlations among nearby neuron spikes in the same stage. Different from existing researches on modeling V1 neurons from image pixels directly, we transmit the signals represented by the ganglion cells in the second layer to the V1 neurons in the third layer. Moreover, solutions to these objectives are provided as well. Experimental results demonstrate that the characteristics of HEVS are consistent with those of the corresponding biological stages. The results further verify the performance of HEVS on dealing with the de-blurring and de-noising tasks.

Feature description for local image patch is widely used in computer vision. While the conventional way to design local descriptor is based on expert experience and knowledge, learning-based methods for designing local descriptor become more and more popular because of their good performance and data-driven property. This paper proposes a novel data-driven method for designing binary feature descriptor, which we call receptive fields descriptor (RFD). Technically, RFD is constructed by thresholding responses of a set of receptive fields, which are selected from a large number of candidates according to their distinctiveness and correlations in a greedy way. Using two different kinds of receptive fields (namely rectangular pooling area and Gaussian pooling area) for selection, we obtain two binary descriptors RFDR and RFDG .accordingly. Image matching experiments on the well-known patch data set and Oxford data set demonstrate that RFD significantly outperforms the state-of-the-art binary descriptors, and is comparable with the best float-valued descriptors at a fraction of processing time. Finally, experiments on object recognition tasks confirm that both RFDR and RFDG successfully bridge the performance gap between binary descriptors and their floating-point competitors.

Local image descriptors are one of the key components in many computer vision applications. Recently, binary descriptors have received increasing interest of the community for its efficiency and low memory cost. The similarity of binary descriptors is measured by Hamming distance which has equal emphasis on all elements of binary descriptors. This paper improves the performance of binary descriptors by learning a weighted Hamming distance for binary descriptors with larger weights assigned to more discriminative elements. What is more, the weighted Hamming distance can be computed as fast as the Hamming distance on the basis of a pre-computed look-up-table. Therefore, the proposed method improves the matching performance of binary descriptors without sacrificing matching speed. Experimental results on two popular binary descriptors (BRIEF [1] and FREAK [2]) validate the effectiveness of the proposed method.

Although feature-based methods have been successfully developed in the past decades for the registration of optical images, the registration of optical and synthetic aperture radar (SAR) images is still a challenging problem in remote sensing. In this letter, an improved version of the scale-invariant feature transform is first proposed to obtain initial matching features from optical and SAR images. Then, the initial matching features are refined by exploring their spatial relationship. The refined feature matches are finally used for estimating registration parameters. Experimental results have shown the effectiveness of the proposed method.