Long short-term memory (LSTM) is a robust recurrent neural network architecture for learning spatiotemporal sequential data. However, it requires significant computational power for learning and implementing from both software and hardware aspects. This paper proposes a novel LiteLSTM architecture based on reducing the computation components of the LSTM using the weights sharing concept to reduce the overall architecture cost and maintain the architecture performance. The proposed LiteLSTM can be significant for learning big data where time-consumption is crucial such as the security of IoT devices and medical data. Moreover, it helps to reduce the CO2 footprint. The proposed model was evaluated and tested empirically on two different datasets from computer vision and cybersecurity domains.

It has been shown that the core reasons that complex and hypercomplex valued neural networks offer improvements over their real-valued counterparts is the fact that aspects of their algebra forces treating multi-dimensional data as a single entity. However, both are constrained to a set number of dimensions, two for complex and four for quaternions. These observations motivate us to introduce novel vector map convolutions which capture this property, while dropping the unnatural dimensionality constraints their algebra imposes. This is achieved by introducing a system that mimics the unique linear combination of input dimensions via the Hamilton product using a permutation function, as well as batch normalization and weight initialization for the system. We perform two experiments to show that these novel vector map convolutions seem to capture all the benefits of complex and hyper-complex networks, while avoiding the dimensionality restriction.

In this paper, we proposed a novel deep-learning method called Inception LSTM for video frame prediction. A standard convolutional LSTM uses a single size kernel for each of its gates. Having multiple kernel sizes within a single gate would provide a richer features that would otherwise not be possible with a single kernel. Our key idea is to introduce inception like kernels within the LSTM gates to capture features from a bigger area of the image while retaining the fine resolution of small information. We implemented the proposed idea of inception LSTM network on PredNet network with both inception version 1 and inception version 2 modules. The proposed idea was evaluated on both KITTI and KTH data. Our results show that the Inception LSTM has better predictive performance compared to convolutional LSTM. We also observe that LSTM with Inception version 1 has better predictive performance compared to Inception version 2, but Inception version 2 has less computational cost.

Abstract Spatiotemporal sequence prediction is an important problem in deep learning. We study next‐frame(s) video prediction using a deep‐learning‐based predictive coding framework that uses convolutional LSTM (convLSTM) modules. We introduce a novel rgcLSTM architecture that requires a significantly lower parameter budget than a comparable convLSTM. By using a single multifunction gate, our reduced‐gate model achieves equal or better next‐frame(s) prediction accuracy than the original convolutional LSTM while using a smaller parameter budget, thereby reducing training time and memory requirements. We tested our reduced gate modules within a predictive coding architecture on the moving MNIST and KITTI datasets. We found that our reduced‐gate model has a significant reduction of approximately 40% of the total number of training parameters and a 25% reduction in elapsed training time in comparison with the standard convolutional LSTM model. The performance accuracy of the new model was also improved. This makes our model more attractive for hardware implementation, especially on small devices. We also explored a space of 20 different gated architectures to get insight into how our rgcLSTM fits into that space.

Heart diseases are mainly diagnosed by the electrocardiogram (ECG) or (EKG). The correct classification of ECG signals helps in diagnosing heart diseases. In this paper, we study and analyze the univariate and multivariate ECG signal classification problems to find the optimal classifier for ECG signals from existing state-of-the-art time series classification models.

Spatiotemporal sequence prediction is an important problem in deep learning. We study next-frame video prediction using a deep-learning-based predictive coding framework that uses convolutional, long short-term memory (convLSTM) modules. We introduce a novel reduced-gate convolutional LSTM architecture that achieves better next-frame prediction accuracy than the original convolutional LSTM while using a smaller parameter budget, thereby reducing training time and memory requirements. We tested our reduced gate modules within a predictive coding architecture on the gray-scale and RGB video datasets. We found that our reduced-gate model has a significant reduction of approximately 40 percent of the total number of training parameters and training time in comparison with the standard LSTM model which makes it attractive for hardware implementation especially on small devices.

This paper empirically evaluates and analyzes the effect of the choice of recurrent activation and unit activation functions on the unsupervised convolutional LSTM learning process. The goal of this work is to provide guidance for selecting the optimal non-linear activation function for the convolutional LSTM models which target the video prediction problem. This paper shows an empirical analysis of different non-linear activation functions that are commonly implemented in different deep learning APIs. We used the moving MNIST dataset as the most common benchmark for video prediction problems.

The field of deep learning has seen significant advancement in recent years. However, much of the existing work has been focused on real-valued numbers. Recent work has shown that a deep learning system using the complex numbers can be deeper for a fixed parameter budget compared to its real-valued counterpart. In this work, we explore the benefits of generalizing one step further into the hyper-complex numbers, quaternions specifically, and provide the architecture components needed to build deep quaternion networks. We develop the theoretical basis by reviewing quaternion convolutions, developing a novel quaternion weight initialization scheme, and developing novel algorithms for quaternion batch-normalization. These pieces are tested in a classification model by end-to-end training on the CIFAR -10 and CIFAR -100 data sets and a segmentation model by end-to-end training on the KITTI Road Segmentation data set. These quaternion networks show improved convergence compared to real-valued and complex-valued networks, especially on the segmentation task, while having fewer parameters.

This paper proposes a novel unsupervised rank aggregation method using parameterized function optimization (PFO). This algorithm derives a parameterized rank aggregation model by minimizing the energy of weighted standard deviations of rank lists associated with different rankers or attributes. Parameters, in this problem, are weights representing the impact of rank lists on the final aggregated rank. The proposed learning to rank aggregation method is efficient (linear time complexity) and its accuracy compares favorably with pairwise preference methods (with polynomial time complexity). Two rounds of experiments are run to show the success of PFO in rank aggregation: one on the learning to rank (LETOR) benchmark dataset to show its success in unsupervised rank aggregation and the other on three university ranking datasets to solve a practical problem in education. The experimental results on the LETOR show that PFO significantly outperforms the baseline results and show promising performances in comparison with recent high performance methods developed for unsupervised rank aggregation. The university ranks obtained by our model compare favorably with the ranks reported by well-known organizations. Success of the PFO model for performing unsupervised rank aggregation, specifically on practical problems, supports the use of the algorithm in difficult ranking scenarios without ground truth.

This paper proposes a new method for training multi-layer spiking convolutional neural networks (CNNs). Training a multi-layer spiking network poses difficulties because the output spikes do not have derivatives and the commonly use backpropagation method for non-spiking networks is not easily applied. Our method uses a novel version of layered spike-timing- dependent plasticity (STDP) that incorporates supervised and unsupervised components. Our method starts with conventional learning methods and converts them to spatio-temporally local rules suited for spiking neural networks (SNNs). The training process uses two components for unsupervised feature extraction and supervised classification. The first component is a new STDP rule for spike-based representation learning which trains convolutional filters. The second introduces a new STDP-based supervised learning rule for spike pattern classification via an approximation to gradient descent. Stacking these components implements a novel spiking CNN of integrate-and-fire (IF) neurons with performances comparable with the state-of-the-art deep SNNs. The experimental results show the success of the proposed model for the MNIST handwritten digit classification. Our network architecture is the only high performance, spiking CNN which provides bio-inspired STDP rules in a hierarchy of feature extraction and classification in an entirely spike-based framework.

While cancer is a heterogeneous complex of distinct diseases, the common underlying mechanism for uncontrolled tumor growth is due to mutations in proto-oncogenes and the loss of the regulatory function of tumor suppression genes. In this paper we propose a novel deep learning model for predicting tumor suppression genes (TSGs) and proto-oncogenes (OGs) from their Protein Data Bank (PDB) three dimensional structures. Specifically, we develop a convolutional neural network (CNN) to classify the feature map sets extracted from the tertiary protein structures. Each feature map set represents particular biological features associated with the atomic coordinates appearing on the outer surface of protein's three dimensional structure. The experimental results on the collected dataset for classifying TSGs and OGs demonstrate promising performance with 82.57% accuracy and 0.89 area under ROC curve. The initial success of the proposed model warrants further study to develop a comprehensive model to identify the cancer driver genes or events using the principle cancer genes (TSG and OG).

Spiking neural networks (SNNs) have advantages over traditional, non-spiking networks with respect to biorealism, potential for low-power hardware implementations, and theoretical computing power. However, in practice, spiking networks with multi-layer learning have proven difficult to train. This paper explores a novel, bio-inspired spiking convolutional neural network (CNN) that is trained in a greedy, layer-wise fashion. The spiking CNN consists of a convolutional/pooling layer followed by a feature discovery layer, both of which undergo bio-inspired learning. Kernels for the convolutional layer are trained using a sparse, spiking auto-encoder representing primary visual features. The feature discovery layer uses a probabilistic spike-timing-dependent plasticity (STDP) learning rule. This layer represents complex visual features using WTA-thresholded, leaky, integrate-and-fire (LIF) neurons. The new model is evaluated on the MNIST digit dataset using clean and noisy images. Intermediate results show that the convolutional layer is stack-admissible, enabling it to support a multi-layer learning architecture. The recognition performance for clean images is above 98%. This performance is accounted for by the independent and informative visual features extracted in a hierarchy of convolutional and feature discovery layers. The performance loss for recognizing the noisy images is in the range 0.1% to 8.5%. This level of performance loss indicates that the network is robust to additive noise.

This paper proposes a novel method for protein function recognition using deep learning. Recently, deep convolutional neural networks (DCNNs) demonstrated high performances in many areas of pattern recognition. Protein function is often associated with its tertiary structure denoting the active domain of a protein. This investigation develops a novel DCNN for protein functionality recognition based on its tertiary structure. Two rounds of experiments are performed. The initial experiment on tertiary protein structure alignment shows promising performances (94% accuracy rate) such that it shows the model robustness against rotations, local translations, and scales of the 3D structure. With these results, the main experiments contain five different datasets obtained by similarity measures between pairs of gene ontology terms. The experimental results for protein function recognition on selected datasets show 87.6% and 80.7% maximum and average accuracy rates respectively. The initial success of the DCNN in tertiary protein structure recognition supports further investigations with respect to tertiary protein retrieval and pattern mining on large scale problems.

Bayesian filters provide a robust and powerful technique for integrating noisy information in dynamic environments. However, the computational cost of the filtering algorithm depends on the size of the problem, and an effective solution may be constrained by execution time. This paper applies basic concepts of clustering and message passing to particle filters making them substantially faster to compute, while still maintaining the original accuracy. An example from vehicle state estimation is provided to illustrate how to implement the technique. Our results indicate that modularization can produce a speed up of over 28 times even on this small problem.

This paper explores modifications to a feedforward five-layer spiking convolutional network (SCN) of the ventral visual stream [Masquelier, T., Thorpe, S., Unsupervised learning of visual features through spike timing dependent plasticity. PLoS Computational Biology, 3(2), 247-257]. The original model showed that a spike-timing-dependent plasticity (STDP) learning algorithm embedded in an appropriately selected SCN could perform unsupervised feature discovery. The discovered features where interpretable and could effectively be used to perform rapid binary decisions in a classifier. In order to study the robustness of the previous results, the present research examines the effects of modifying some of the components of the original model. For improved biological realism, we replace the original non-leaky integrate-and-fire neurons with Izhikevich-like neurons. We also replace the original STDP rule with a novel rule that has a probabilistic interpretation. The probabilistic STDP slightly but significantly improves the performance for both types of model neurons. Use of the Izhikevich-like neuron was not found to improve performance although performance was still comparable to the IF neuron. This shows that the model is robust enough to handle more biologically realistic neurons. We also conclude that the underlying reasons for stable performance in the model are preserved despite the overt changes to the explicit components of the model.

Visual inspection of neuroimagery is susceptible to human eye limitations. Computerized methods have been shown to be equally or more effective than human clinicians in diagnosing dementia from neuroimages. Nevertheless, much of the work involves the use of domain expertise to extract hand-crafted features. The key technique in this paper is the use of cross-domain features to represent MRI data. We used a sparse autoencoder to learn a set of bases from natural images and then applied convolution to extract features from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset.Using this new representation, we classify MRI instances into three categories: Alzheimer’s Disease (AD), Mild Cognitive Impairment (MCI) and Healthy Control (HC).Our approach, in spite of being very simple, achieved high classification performance, which is competitive with or better than other approaches.

This paper demonstrates that feature acquisition systems composed of spiking neurons trained by spike-timing-dependent plasticity (STDP) can effectively be scaled using General-Purpose computing on Graphics Processing Units (GPGPU). Previous studies have demonstrated the efficacy of such systems for classes with low intra-class variability. Parallelization substantially increases the range of classes to which such a system can be applied. This system uses a hierarchical design based on the ventral visual pathway, alternating between selective layers that combine inputs and invariance layers that sample from inputs. Most systems do not, however, use spiking neurons as the brain does. Masquelier and Thorpe presented a system that produces highly informative features using a spiking convolutional network trained by STDP. However, this approach is costly in time and consequently is difficult to scale to a large library of internally complex features. Specifically, they reported that their system was incapable of learning the class “animal”, which was a major goal of that study. The present paper demonstrates that GPGPU parallelism can be leveraged to overcome the scaling limitations of the serial version. Highly informative features can then be generated in large quantities. The maximum complexity of classes learnable by this system is increased to encompass the natural class “animal” by GPGPU paralleization.

Unsupervised of different types of activity in video data has many applications such as anomaly detection, automated tagging of video for search, and cognitive modeling. Topic models originally used in corpus analysis have recently been used to identify different types of activities in videos. Among topic models, probabilistic latent semantic analysis (pLSA) provides an efficient method for identifying clusters of activity in video. This paper integrates pLSA with the causal graphical models of Pearl [1] to learn visual event structures and their temporal relationships simultaneously. The model is fully generative. A noisy-OR style temporal dependence is used for learning which is well known to identify the same causal patterns that human learners do. The addition of temporal learning allows the system to model temporally ordered and long range temporal dependencies that traditional topic models cannot. The model successfully identifies human recognizable event structures in video and successfully classifies videos of human activity learning.

Understanding how sequences are learned and encoded is a key component to understanding cognition. We present a recruitment model in which sequences are learned via the hierarchical binding of features across time. Learning in the model is unsupervised and occurs within a single presentation of the input. The topology and learning mechanisms allow the network to exploit the temporal structure of the input in order to recruit localized representations of sequences, using leaky integrate-and-fire neurons with biologically-grounded learning mechanisms. The model learns a temporal XOR-style task, and ablation tests are performed to justify the inclusion of particular features in the model. The model is then extended and applied to the task of learning 7-digit sequences. Both sets of simulations demonstrate the ability of the model to acquire and reuse chunks. Limitations and future extensions of the model are then discussed.

The generalized multilayer perceptron (gMLP) augments the connections in the multilayered perceptron (MLP) architecture to include all possible non-recurrent connections. The layered arbitrarily connected network (lACN) has connections from input nodes to output nodes in addition to the connections included in a MLP. In this paper the performance of MLP, lACN and gMLP networks trained using the Levenberg-Marquardt method are compared. A number of different function approximation tasks were examined. The effect of varying the number of hidden layer neurons, the error termination condition, and the training set size were also evaluated. The results presented here represent preliminary findings. In particular, additional testing on benchmark real data sets is needed.

Our work describes the simulation of a planar network of spiking I/F neurons on graphics processing hardware. The described approach adds to the fast-growing field of general-purpose computation on GPUs (GPGPU). We provide an in-depth explanation of the steps involved in implementing the network using programmable shading hardware. We replicated simulation results by Hopfield et al. and Maida et al. and give qualitative and quantitative measures of our implementation.

This research models human performance in the Dalla Bella, Peretz, and Aronoff melody recognition study. They compared performance between musicians and nonmusicians in the recognition (and perception) of melodies. They used a gating task to identify three events in the melody perception/recognition process. These were the familiarity emergence point (FEP), the isolation point (IP), and the recognition point (RP). We develop a simulation to model hypothesized cognitive processes underlying these events. The IP is modeled using a winner-take-all connectionist network adapted to operate with temporal input sequences. Meta-level processes examine the dynamic state of the recognition network to model the FEP and the RP.

Weight-perturbation (WP) algorithms for supervised and/or reinforcement learning offer improved biological plausibility over backpropagation because of their reduced circuitry requirements for realization in neural hardware. This paper explores the hypothesis that biological synaptic noise might serve as the substrate by which weight perturbation is implemented. We explore the basic synaptic noise hypothesis (BSNH), which embodies the weakest assumptions about the underlying neural circuitry required to implement WP algorithms. This paper identifies relevant biological constraints consistent with the BSNH, taxonomizes existing WP algorithms with regard to consistency with those constraints, and proposes a new WP algorithm that is fully consistent with the constraints. By comparing the learning effectiveness of these algorithms via simulation studies, it is found that all of the algorithms can support traditional neural network learning tasks and have similar generalization characteristics, although the results suggest a trade-off between learning efficiency and biological accuracy.

The temporal correlation hypothesis proposes using distributed synchrony for the binding stimulus features. However, synchronized spikes must travel over cortical circuits that have varying-length pathways, leading to mismatched arrival times. This raises the question of how initial stimulus-dependent synchrony might be preserved at a destination binding site. Earlier, we proposed constraints on tolerance and segregation parameters for a phase-coding approach, within cortical circuits, to address this question [C. Gunay et. al., July 2001] The purpose of the present paper is twofold. First, we conduct simulation experiments to test the proposed constraints. Second, we explore the practicality of temporal binding to drive a process of long-term memory formation based on a recruitment learning method [J.A. Feldman, 1982]. A network based on Valiant's neuroidal architecture [Leslie G. Valiant, 1994] is used to demonstrate the coalition between temporal binding and recruitment. Complementing similar approaches, we implement a continuous-time learning procedure allowing computation with spiking neurons. Our results indicate that when tolerance and segregation parameters obey our proposed constraints, the assemblies of correct bindings are dominant over assemblies of spurious bindings in reasonable operating conditions.

A simulator for the Burgess, Recce, and O'Keefe (1994) BRO model of rodent navigation was built. Initial experiments did not reproduce the navigation performance reported by BRO. To determine the cause of the discrepancy, we reconstructed and reanalyzed their model. We were able to verify their main claim that if phase-modulated place cells act as radial basis functions, then they computationally have the ability to support memory-based navigation. However, our simulations were unable to reproduce their model of hippocampal cell place field acquisition.

Neural maps have been recently proposed as an alternative method for mobile robot path planning. However, these proposals are mostly theoretical and primarily concerned with biological plausibility. This paper addresses the applicability of neural maps to mobile robot navigation with focus on efficient implementations It is suggested that neural maps offer a promising alternative compared to the traditional distance transform and harmonic function methods. Applications of neural maps are presented for both global and local navigation. Experimental results (both simulated and real-world on a Nomad 200 mobile robot) demonstrate the validity of the approach. Our work reveals that a key issue for success of the method is the organization of the map that needs to be optimized for the situation at hand.

. This paper analyses description-based communication for autonomous agents when it takes place under ideal conditions. By ideal conditions, is meant the following: agents operate in a domain consisting of novel objects but each object has a finite and unique set of basic properties. Agents are able to completely and accurately observe the basic properties of an object. Furthermore, when an agent describes an object, the agent completely and accurately describes the basic properties of the object. Finally, there are no transmission errors in the communication process and an agent that receives a message believes everything it is told. Under these conditions, it is shown that the object recognition problem is identical to the referent identification problem. Further, referent identification failures and object misrecognitions do not occur. Also, agents never acquire false beliefs, and for any object that an agent knows about, its knowledge of the basic properties of the object is complete. Sufficient conditions are stated to infer that a causal reference chain (description forwarding) is well grounded and it is shown that agents operating under the ideal conditions described above satisfy these requirements.

This paper considers the problem of representing the knowledge state of an observed agent who has incomplete knowledge of existence. Agents in such knowledge states are said to have existential misconceptions. We describe preliminary work on representing the knowledge state of an agent who possesses existential misconceptions and how to detect when this occurs. The method is based on a simulation of the observed agent's processes of information acquisition. We describe an implementation of this system and study its operation in a simulated microworld. The agent's acquisition of existential misconceptions is correlated with the agent's inaccurate beliefs of equality. This makes it possible to characterize an agent's existential misconceptions by describing its inaccurate beliefs of equality.

A semantic framework for nonmonotonic reasoning is developed. Specifically, a reliability measure theory based on multivalued logics that view a knowledge base as a set of contexts is proposed, and the contexts are compared according to some reliability measure. The proposed theory shows higher precision of nonmonotonic reasoning than other approaches. In particular, the qualification problem does not occur and many unwanted conflicts are eliminated.>

We propose a psychologically and psychophysically motivated texture segmentation algorithm. The algorithm is implemented as a computer program which parses visual images into regions on the basis of texture. The program's output matches human judgements on a very large class of stimuli. The program and algorithm offer very detailed hypotheses of how humans might segment stimuli, and also suggests plausible alternative explanations to those presented in the literature. In particular, contrary to Julesz & Bergen (1983), the program does not use crossings as textons and the program does use corners as textons. Nonetheless, the program is able to account for the same data. The program accounts for much of the linking phenomena of Beck et al. (1983). It does so by matching structures between feature maps on the basis of spatial overlap. These same mechanisms are also used to account for the feature integration phenomena of Treisman (1985).