As the third generation of neural networks, spiking neural networks (SNNs) are dedicated to exploring more insightful neural mechanisms to achieve near-biological intelligence. Intuitively, biomimetic mechanisms are crucial to understanding and improving SNNs. For example, the associative long-term potentiation (ALTP) phenomenon suggests that in addition to learning mechanisms between neurons, there are associative effects within neurons. However, most existing methods only focus on the former and lack exploration of the internal association effects. In this paper, we propose a novel Adaptive Internal Association (AIA) neuron model to establish previously ignored influences within neurons. Consistent with the ALTP phenomenon, the AIA neuron model is adaptive to input stimuli, and internal associative learning occurs only when both dendrites are stimulated at the same time. In addition, we employ weighted weights to measure internal associations and introduce intermediate caches to reduce the volatility of associations. Extensive experiments on prevailing neuromorphic datasets show that the proposed method can potentiate or depress the firing of spikes more specifically, resulting in better performance with fewer spikes. It is worth noting that without adding any parameters at inference, the AIA model achieves state-of-the-art performance on DVS-CIFAR10 (83.9%) and N-CARS (95.64%) datasets.

Neuromorphic vision sensors (event cameras) are inherently suitable for spiking neural networks (SNNs) and provide novel neuromorphic vision data for this biomimetic model. Due to the spatiotemporal characteristics, novel data augmentations are required to process the unconventional visual signals of these cameras. In this paper, we propose a novel Event Spatio Temporal Fragments (ESTF) augmentation method. It preserves the continuity of neuromorphic data by drifting or inverting fragments of the spatiotemporal event stream to simulate the disturbance of brightness variations, leading to more robust spiking neural networks. Extensive experiments are performed on prevailing neuromorphic datasets. It turns out that ESTF provides substantial improvements over pure geometric transformations and outperforms other event data augmentation methods. It is worth noting that the SNNs with ESTF achieve the state-of-the-art accuracy of 83.9% on the CIFAR10-DVS dataset.

Neuromorphic vision sensors (event cameras) simulate biological visual perception systems and have the advantages of high temporal resolution, less data redundancy, low power consumption, and large dynamic range. Since both events and spikes are modeled from neural signals, event cameras are inherently suitable for spiking neural networks (SNNs), which are considered promising models for artificial intelligence (AI) and theoretical neuroscience. However, the unconventional visual signals of these cameras pose a great challenge to the robustness of spiking neural networks. In this paper, we propose a novel data augmentation method, View-Point Transform and SpatioTemporal Stretching (VPT-STS). It improves the robustness of SNNs by transforming the rotation centers and angles in the spatiotemporal domain to generate samples from different viewpoints. Furthermore, we introduce the spatiotemporal stretching to avoid potential information loss in viewpoint transformation. Extensive experiments on prevailing neuromorphic datasets demonstrate that VPT-STS is broadly effective on multi-event representations and significantly outperforms pure spatial geometric transformations. Notably, the SNNs model with VPT-STS achieves a state-of-the-art accuracy of 84.4% on the DVS-CIFAR10 dataset.

Convolutional neural networks (CNNs) have achieved superior performance but still lack clarity about the nature and properties of feature extraction. In this paper, by analyzing the sensitivity of neural networks to frequencies and scales, we find that neural networks not only have low- and mediumfrequency biases but also prefer different frequency bands for different classes, and the scale of objects influences the preferred frequency bands. These observations lead to the hypothesis that neural networks must learn the ability to extract features at various scales and frequencies. To corroborate this hypothesis, we propose a network architecture based on Gaussian derivatives, which extracts features by constructing scale space and employing partial derivatives as local feature extraction operators to separate high-frequency information. This manually designed method of extracting features from different scales allows our GSSDNets to achieve comparable accuracy with vanilla networks on various datasets.

Existing super-resolution (SR) methods commonly assume that the degradation kernels are fixed and known (e.g., bicubic downsampling or single Gaussian blurring kernel). However, these methods suffer a severe performance drop when the real degradations deviate from this assumption. To address this issue, this paper proposes a novel kernel estimation network (KENet) for kernel prediction. Specifically, KENet predicts the degradation kernels by optimizing the kernel space loss in a supervised way, without extra iterations at the inference time. Moreover, we introduce an adaptive attention loss to constrain the kernel optimization space, which can bias the allocation of trainable model parameters towards the most informative components of the estimation kernels. Extensive experiments on synthetic and real images show that the proposed KENet not only encourages a more accurate way to predict degradation kernels but also outperforms existing state-of-the-art blind SR methods when combined with non-blind SR methods.

It is a common paradigm in object detection frameworks to perform multi-scale detection. However, each scale is treated equally during training. In this paper, we carefully study the objective imbalance of multi-scale detector training. We argue that the loss in each scale is neither equally important nor independent. Different from the existing solutions of setting fixed multi-task weights, we dynamically optimize the loss weight of each scale in the training process. Specifically, we propose an Adaptive Variance Weighting (AVW) to balance multi-scale loss according to the statistical variance. Then we develop a novel Reinforcement Learning Optimization (RLO) to decide the weighting scheme probabilistically during training. The proposed dynamic methods make better utilization of multi-scale training loss without extra computational complexity and learnable parameters for backpropagation. Experiments on Pascal VOC and MS COCO benchmark validate the effectiveness of our proposed methods.

Feature Pyramid Network (FPN) has become a common detection paradigm by improving multi-scale features with strong semantics. However, most FPN-based methods typically treat each feature map equally and sum the loss without distinction, which might lead to suboptimal overall performance. In this paper, we propose a Multi-scale Reinforcement Learning Strategy (MRLS) for balanced multi-scale training. First, we design Dynamic Feature Fusion (DFF) to dynamically magnify the impact of more important feature maps in FPN. Second, we introduce Compensatory Scale Training (CST) to enhance the supervision of the under-training scale. We regard the whole detector as a reinforcement learning system while the state bases on multi-scale loss. And we develop the corresponding action, reward, and policy. Compared with adding more rich model architectures, MRLS would not add any extra modules and computational burdens on the baselines. Experiments on MS COCO and PASCAL VOC benchmark demonstrate that our method significantly improves the performance of commonly used object detectors.

Modern efficient Convolutional Neural Networks(CNNs) always use Depthwise Separable Convolutions(DSCs) and Neural Architecture Search(NAS) to reduce the number of parameters and the computational complexity. But some inherent characteristics of networks are overlooked. Inspired by visualizing feature maps and N×N(N>1) convolution kernels, several guidelines are introduced in this paper to further improve parameter efficiency and inference speed. Based on these guidelines, our parameter-efficient CNN architecture, called VGNetG, achieves better accuracy and lower latency than previous networks with about 30%~50% parameters reduction. Our VGNetG-1.0MP achieves 67.7% top-1 accuracy with 0.99M parameters and 69.2% top-1 accuracy with 1.14M parameters on ImageNet classification dataset. Furthermore, we demonstrate that edge detectors can replace learnable depthwise convolution layers to mix features by replacing the N×N kernels with fixed edge detection ker-nels. And our VGNetF-1.5MP archives 64.4%(-3.2%) top-1 accuracy and 66.2%(-1.4%) top-1 accuracy with additional Gaussian kernels.

A proper strategy to alleviate overfitting is critical to a deep neural network (DNN). In this paper, we introduce the cross-loss-function regularization for boosting the generalization capability of the DNN, which results in the multi-loss regularized DNN (ML-DNN) framework. For a particular learning task, e.g., image classification, only a single-loss function is used for all previous DNNs, and the intuition behind the multiloss framework is that the extra loss functions with different theoretical motivations (e.g., pairwise loss and LambdaRank loss) may drag the algorithm away from overfitting to one particular single-loss function (e.g., softmax loss). In the training stage, we pretrain the model with the single-core-loss function and then warm start the whole ML-DNN with the convolutional parameters transferred from the pretrained model. In the testing stage, the outputs by the ML-DNN from different loss functions are fused with average pooling to produce the ultimate prediction. The experiments conducted on several benchmark datasets (CIFAR-10, CIFAR-100, MNIST, and SVHN) demonstrate that the proposed ML-DNN framework, instantiated by the recently proposed network in network, considerably outperforms all other state-of-the-art methods.

The authors introduce the concept of a spline resampling in the particle filter to deal with high accuracy and sample impoverishment. The resampling is usually based on a linear transformation on the weights of the particles, so it affects the filtering accuracy. The spline resampling consists of two parts: the spline transformation of weights and the spread transformation of states. The former is based on a spline transformation on the weights of the particles to obtain highly accurate particle filtering, and the latter is based on a point spread transformation on states of particles to prevent sample impoverishment due to a decline in the diversity of hypothesis after resampling. Two transformations are sequentially implemented to incorporate with each other. Then, the authors propose a global transition model in the particle filter, which takes account of the background variation caused by the camera motion, to decrease error from real object position. The authors test the performance of their spline resampling and the global transition model in the particle filter in an object‐tracking scenario. Experimental results demonstrate that the particle filter with the spline resampling and the global transition model has promising discriminative capability in comparison with others.

In this letter, we introduce a subcategory-aware object detection framework to detect generic object classes with high intra-class variance. Motivated by the observation that the object appearance demonstrates some clustering property, we split the training data into subcategories and train a detector for each subcategory. Since the proposed ensemble of detectors relies heavily on subcategory clustering, we propose an effective subcategories generation method that is tuned for the detection task. More specifically, we first initialize subcategories by constrained spectral clustering based on mid-level image features used in object recognition. Then we jointly learn the ensemble detectors and the latent subcategories in an alternative manner. Our performance on the PASCAL VOC 2007 detection challenges and INRIA Person dataset is comparable with state-of-the-art, even with much less computational cost.

To efficiently deal with the complex nonlinear variations of face images, a novel Lie group (LG) kernel is proposed in this paper to address the facial analysis problems. First, we present a linear dynamic model (LDM)-based face representation to capture both the appearance and spatial information of the face image. Second, the derived LDM can be parameterized as a specially structured upper triangular matrix, the space of which is proved to constitute an LG. An LG kernel is then designed to characterize the similarity between the LDMs for any two face images and the kernel can be fed into classical kernel-based classifiers for different types of facial analysis. Finally, experimental evaluations on face recognition and head pose estimation are conducted on several challenging data sets and the results show that the proposed algorithm outperforms other facial analysis methods.

In this paper, we study discriminative analysis of symmetric positive definite (SPD) matrices on Lie groups (LGs), namely, transforming an LG into a dimension-reduced one by optimizing data separability. In particular, we take the space of SPD matrices, e.g., covariance matrices, as a concrete example of LGs, which has proved to be a powerful tool for high-order image feature representation. The discriminative transformation of an LG is achieved by optimizing the within-class compactness as well as the between-class separability based on the popular graph embedding framework. A new kernel based on the geodesic distance between two samples in the dimension-reduced LG is then defined and fed into classical kernel-based classifiers, e.g., support vector machine, for various visual classification tasks. Extensive experiments on five public datasets, i.e., Scene-15, Caltech101, UIUC-Sport, MIT-Indoor, and VOC07, well demonstrate the effectiveness of discriminative analysis for SPD matrices on LGs, and the state-of-the-art performances are reported.

Inspired by the recent image feature learning work, we propose a novel key point detection approach for object tracking. Our approach can select mid-level interest key points by max pooling over the local descriptor responses from a set of filters. Linear filters are first learned from targets in first frames. Then max pooling is performed over data driven spatial supporting field to detect discriminant key points, and thus the detected key points bear higher level semantic meanings, which we apply in tracking by structured key point matching. We show that our tracking system is robust to occlusions and cluttered background. Testing on several challenging tracking sequences, we demonstrate that our proposed tracking system can achieve competitive or better performances than the state-of-the-art trackers.

Extensive neurophysiological studies have shown that the receptive field plays a significant role in the human visual system. It has various kinds of properties such as orientation-selectivity, correlativity, etc. Motivated by these structural and functional properties, we propose in this paper a novel local image descriptor namely the Discriminative Transform of Receptive Fields (DTRF). Specifically, Receptive Field Patterns (RFP) are defined around each sample pixel and then divided into two kinds of components: RFP-Surround and RFP-Center. The RFP-Surround serves as the basic feature structure, which is extracted based on Local Annular Discrete Cosine Transform (LADCT) algorithm. The RFP-Center is used to pool these local features to simulate the correlative property of receptive field. Experimental results on the standard Oxford data set demonstrate the superiority of DTR-F over the state-of-the-art descriptors under various types of image transformations such as rotation and scaling changes, viewpoint changes, image blurring, JPEG compression, illumination changes, and image noise.

We present a new fuzzy c-means algorithm for image segmentation by introducing a novel spatially constrained Student's t-distribution and a new regularization term. Firstly, considering that conventional distribution models lack spatial information and the multivariate Student's t-distribution is heavily tailed, we propose a new way to incorporate spatial information between neighboring pixels into the Student's t-distribution based on Markov random field (MRF) in order to enhance robustness. Secondly, the new regularization term, inspired by the geodesic active contour (GAC) with a strong ability in capturing boundary, can preserve the details of edges and further enhance its robustness to noise and outliers by capitalizing on the local context information and edge information. Finally, in comparison to other Markov random fields that are complex and computationally expensive, the parameters are easily optimized with the EM algorithm in our proposed method. The proposed algorithm demonstrates the robustness and effectiveness, compared with other state-of-the-art methods on synthetic and real images.

This paper first introduces the concept of a geogram that captures richer features to represent the objects. The spatiogram contains some moments upon the coordinates of the pixels corresponding to each bin, while the geogram contains information about the perimeter of grouped regions in addition to features in the spatiogram. Then we consider that a convergence process of mean shift is divided into obvious dynamic and steady states, and introduce a hybrid technique of feature description, to control the convergence process. Also, we propose a spline resampling to control the balance between computational cost and accuracy of particle filtering. Finally, we propose a boosting-refining approach, which is boosting the particles positioned in the ill-posed condition instead of eliminating the ill-posed particles, to refine the particles. It enables the estimation of the object state to obtain high accuracy. Experimental results show that our approach has promising discriminative capability in comparison with the state-of-the-art approaches.

Dimensionality reduction (DR) has been considered as one of the most significant tools for data analysis. One type of DR algorithms is based on latent variable models (LVM). LVM-based models can handle the preimage problem easily. In this paper we propose a new LVM-based DR model, named thin plate spline latent variable model (TPSLVM). Compared to the well-known Gaussian process latent variable model (GPLVM), our proposed TPSLVM is more powerful especially when the dimensionality of the latent space is low. Also, TPSLVM is robust to shift and rotation. This paper investigates two extensions of TPSLVM, i.e., the back-constrained TPSLVM (BC-TPSLVM) and TPSLVM with dynamics (TPSLVM-DM) as well as their combination BC-TPSLVM-DM. Experimental results show that TPSLVM and its extensions provide better data visualization and more efficient dimensionality reduction compared to PCA, GPLVM, ISOMAP, etc.

This paper presents an ordered-patch-based image classification framework integrating the image Grassmannian manifold to address handwritten digit recognition, face recognition, and scene recognition problems. Typical image classification methods explore image appearances without considering the spatial causality among distinctive domains in an image. To address the issue, we introduce an ordered-patch-based image representation and use the autoregressive moving average (ARMA) model to characterize the representation. First, each image is encoded as a sequence of ordered patches, integrating both the local appearance information and spatial relationships of the image. Second, the sequence of these ordered patches is described by an ARMA model, which can be further identified as a point on the image Grassmannian manifold. Then, image classification can be conducted on such a manifold under this manifold representation. Furthermore, an appropriate Grassmannian kernel for support vector machine classification is developed based on a distance metric of the image Grassmannian manifold. Finally, the experiments are conducted on several image data sets to demonstrate that the proposed algorithm outperforms other existing image classification methods.

Accurate head pose estimation is significant for many applications such as face recognition and human-computer interaction. In this paper, we treat the head pose estimation as a classification problem and employ the Lie Algebrized Gaussians (LAG) feature as the representation approach for head image. The LAG feature, which is built on Gausssian Mixture Model (GMM), has the capability to preserve the structure of Gaussian components in the original Lie group manifold. Moreover, to keep more spatial structure information of the image, LAG is operated on many subregions of the image. As a result, these properties of LAG enable it to reflect the pose characteristic of the head image well and possess powerful discriminative ability in pose classification. Experiments on CMU Pose, Illumination, and Expression (PIE) and Pointing'04 benchmarks show state-of-the-art performance and demonstrate that LAG represents the head pose characteristic well.

Dimensionality reduction (DR) has been considered as one of the most significant tools for data analysis. In this paper we propose a new latent variable model based on the thin plate splines, named Thin Plate Spline Latent Variable Model (TPSLVM). It has strong connection with the so-called Gaussian Process Latent Variable Model (GPLVM). We demonstrate that the proposed TPSLVM can be viewed as the GPLVM with a fairly peculiar covariance function. Moreover, compared to GPLVM, TPSLVM is more powerful especially when the dimensionality of the latent space is very low (e.g., 2D or 3D). One of main purposes of DR algorithms is to visualize data in 2D/3D spaces. Therefore, TPSLVM will benefit this process. Experimental results show that TPSLVM provides better data visualization and more efficient dimensionality reduction than GPLVM.

The Gaussian process (GP) latent variable model (GPLVM) has the capability of learning low-dimensional manifold from highly nonlinear data of high dimensionality. As an unsupervised dimensionality reduction (DR) algorithm, the GPLVM has been successfully applied in many areas. However, in its current setting, GPLVM is unable to use label information, which is available for many tasks; therefore, researchers proposed many kinds of extensions to the GPLVM in order to utilize extra information, among which the supervised GPLVM (SGPLVM) has shown better performance compared with other SGPLVM extensions. However, the SGPLVM suffers in its high computational complexity. Bearing in mind the issues of the complexity and the need of incorporating additionally available information, in this paper, we propose a novel SGPLVM, called supervised latent linear GPLVM (SLLGPLVM). Our approach is motivated by both SGPLVM and supervised probabilistic principal component analysis (SPPCA). The proposed SLLGPLVM can be viewed as an appropriate compromise between the SGPLVM and the SPPCA. Furthermore, it is also appropriate to interpret the SLLGPLVM as a semiparametric regression model for supervised DR by making use of the GP to model the unknown smooth link function. Complexity analysis and experiments show that the developed SLLGPLVM outperforms the SGPLVM not only in the computational complexity but also in its accuracy. We also compared the SLLGPLVM with two classical supervised classifiers, i.e., a GP classifier and a support vector machine, to illustrate the advantages of the proposed model.

This paper introduces a feature descriptor called Shape of Gaussian (SOG), which is based on a general feature descriptor design framework called Shape of Signal Probability Density Function (SOSPDF). SOSPDF takes the shape of a signal's probability density function (pdf) as its feature. Under such a view, both histogram and region covariance often used in computer vision are SOSPDF features. Histogram describes SOSPDF by a discrete approximation way. Region covariance describes SOSPDF as an incomplete parameterized multivariate Gaussian distribution. Our proposed SOG descriptor is a full parameterized Gaussian, so it has all the advantages of region covariance and is more effective. Furthermore, we identify that SOGs form a Lie group. Based on Lie group theory, we propose a distance metric for SOG. We test SOG features in tracking problem. Experiments show better tracking results compared with region covariance. Moreover, experiment results indicate that SOG features attempt to harvest more useful information and are less sensitive against noise.

Spatiograms were generalization of histograms, which can harvest spatial information of images. The similarity measure is important when applying spatiograms to various computer vision problems such as tracking and image retrieval. The original proposed measures use Mahalanobis distance of coordinate mean to measure spatial information in spatiograms. However, spatial information which is described by spatiograms does not lie on vector space. Measures for vector space such as Mahalanobis distance are not effective measures for them. In this paper, We model spatial information as Gaussian approximation of coordinate distributions. Then we parameterize them as a Lie group. Based on Lie group theory, we analyze function space structure of Gaussian pdfs (probability density function) and propose an effective spatiogram similarity measure. We test our measure in object tracking scenarios. Experiments show better tracking results compared with previously proposed measures.

In this paper, a fast and robust video copy detection scheme is proposed, which is suitable for the DCT-coded video sequences. To address the efficiency and effectiveness issue, we extract the video signature directly from the compressed domain. The video sequence clusters are constructed with a fixed length. Each cluster consists of several fictional key-frames. For each key-frame, some low-middle frequency full DCT coefficients are obtained directly from block DCT coefficients, and their ordinal measure is computed and acts as video signature. A rotation compensational strategy is further employed to resist the rotation attacks. The experimental results show that the proposed scheme can be resilient to various types of video transformations, including scaling, rotation, speed change, text insertion, and subsequence insertion/deletion etc.. The most important thing is that the proposed approach not only handles geometric distortion perfectly, but also reduces the computation costs substantially.

As an effective means of multi-agent problem solving, autonomous individual programming and interaction that are essential need, are limited in their ability to accommodate the interests of others, and therefore, may unnecessarily constrain the solving ability and negotiability of an agent, particularly in a distributed cooperative environments founded on private and uncertain information. In this paper, a multi-agent group programming model is presented, where each agent executes local programming by evolutionary search. Based on co-evolution idea, agents resolve conflicts and revise their own search direction to optimize local and social objectives in an interactive process by means of clustering and group choice. Finally, the paper presents simulation results that illustrate the operational effectiveness of our agent group programming model.

In this paper we describe a new approach to distinguish and extract text from images with various objects and complex backgrounds. The goal of our approach is to present characters in images with clear background and without other objects. The proposed approach mainly includes two steps. Firstly, a density-based clustering method is employed to segment candidate characters by integrating spatial connectivity and color feature of characterspsila pixels. In most images, colors of pixels in one character are commonly non-uniform due to the noise. So a new histogram segmentation method is proposed in this step to obtain the color thresholds of characters. Secondly, priori knowledge and texture-based method are performed on the candidate characters to filter the non-characters. Experimental results show that the proposed approach has a good performance in character extraction rate.