Efficient processing of large-scale time-series data is an intricate problem in machine learning. Conventional sensor signal processing pipelines with hand-engineered feature extraction often involve huge computational costs with high dimensional data. Deep recurrent neural networks have shown promise in automated feature learning for improved time-series processing. However, generic deep recurrent models grow in scale and depth with the increased complexity of the data. This is particularly challenging in presence of high dimensional data with temporal and spatial characteristics. Consequently, this work proposes a novel deep cellular recurrent neural network (DCRNN) architecture to efficiently process complex multidimensional time-series data with spatial information. The cellular recurrent architecture in the proposed model allows for location-aware synchronous processing of time-series data from spatially distributed sensor signal sources. Extensive trainable parameter sharing due to cellularity in the proposed architecture ensures efficiency in the use of recurrent processing units with high-dimensional inputs. This study also investigates the versatility of the proposed DCRNN model for the classification of multiclass time-series data from different application domains. Consequently, the proposed DCRNN architecture is evaluated using two time-series data sets: a multichannel scalp electroencephalogram (EEG) data set for seizure detection, and a machine fault detection data set obtained in-house. The results suggest that the proposed architecture achieves state-of-the-art performance while utilizing substantially less trainable parameters when compared to comparable methods in the literature.

Malicious software (malware) is designed to cause unwanted or destructive effects on computers. Since modern society is dependent on computers to function, malware has the potential to do untold damage. Therefore, developing techniques to effectively combat malware is critical. With the rise in popularity of polymorphic malware, conventional anti-malware techniques fail to keep up with the rate of emergence of new malware. This poses a major challenge towards developing an efficient and robust malware detection technique. One approach to overcoming this challenge is to classify new malware among families of known malware. Several machine learning methods have been proposed for solving the malware classification problem. However, these techniques rely on hand-engineered features extracted from malware data which may not be effective for classifying new malware. Deep learning models have shown paramount success for solving various classification tasks such as image and text classification. Recent deep learning techniques are capable of extracting features directly from the input data. Consequently, this paper proposes an end-to-end deep learning framework for multimodels (henceforth, multimodel learning) to solve the challenging malware classification problem. The proposed model utilizes three different deep neural network architectures to jointly learn meaningful features from different attributes of the malware data. End-to-end learning optimizes all processing steps simultaneously, which improves model accuracy and generalizability. The performance of the model is tested with the widely used and publicly available Microsoft Malware Challenge Dataset and is compared with the state-of-the-art deep learning-based malware classification pipeline. Our results suggest that the proposed model achieves comparable performance to the state-of-the-art methods while offering faster training using end-to-end multimodel learning.

Deep learning classifiers are assisting humans in making decisions and hence the user's trust in these models is of paramount importance. Trust is often a function of constant behavior. From an AI model perspective it means given the same input the user would expect the same output, especially for correct outputs, or in other words consistently correct outputs. This paper studies a model behavior in the context of periodic retraining of deployed models where the outputs from successive generations of the models might not agree on the correct labels assigned to the same input. We formally define consistency and correct-consistency of a learning model. We prove that consistency and correct-consistency of an ensemble learner is not less than the average consistency and correct-consistency of individual learners and correct-consistency can be improved with a probability by combining learners with accuracy not less than the average accuracy of ensemble component learners. To validate the theory using three datasets and two state-of-the-art deep learning classifiers we also propose an efficient dynamic snapshot ensemble method and demonstrate its value. Code for our algorithm is available at https://github.com/christa60/dynens.

Efficient learning of data distribution is necessary for many different applications such as classification, recognition, decision making and segmentation. Generative probabilistic models have been extensively used to effectively learn distribution of the input data. Moreover, generative models are considered as the first and foremost building blocks for developing highly expressive deep neural network architectures. Throughout the past decade, researchers have developed several undirected and directed probabilistic generative models. The two most popular generative models are Restricted Boltzman Machine (RBM) and Sigmoid Belief Network (SBN). Both these models utilize a two layer architecture with feedforward information processing inspired by the biological systems for learning data distributions. However, none of these models exhibit another well-known property found in biology in the form of recurrent neuronal information processing, which maybebeneficial for learning more complex data distribution. Consequently, this paper, for the first time in literature, proposes a directed recurrent generative model known as Simultaneous Recurrent Belief Network (SRBN) for efficiently learning the distribution of the input data. The efficacy of the proposed SRBN model is evaluated using two benchmark datasets: MNIST and Caltech 101 Silhouettes. Our experimental results suggest the SRBN model shows improved data distribution learning performance while utilizing minimal trainable parameters.

In cognitive radio networks, it is desirable to determine radio spectrum usage in frequency, time, and spatial domains. Spectrum data improves cognitive radio network planning, sensing, routing, and security. Due to cost concerns, spectrum monitors are deployed sparsely in space, spectrum usage at nearby locations can be modeled for use in these applications. Previous work using neural networks for spatial spectrum prediction involved prior knowledge of transmitter locations as input to the models. In practical scenarios, the prior knowledge is not available. Hence, this work considers prediction of the spatial spectrum without knowing the transmitter location information. The prediction task is achieved by using a specialized recurrent neural network known as Cellular Simultaneous Recurrent Network (CSRN). Our investigation shows the proposed recurrent neural network operates in real-time and is generalized to offer spectrum estimations without further changes to the network, even when a transmitter location is changed. The experiments are conducted in a challenging indoor environment to assess the performance in a practical scenario. Our results suggest the CSRN can learn efficiently to predict signal across an indoor space while transmitters move to different locations. We perform a performance comparison of our proposed technique with an MLP based estimation method. Our analysis further suggests that the CSRN achieves comparable prediction accuracy to that of the MLP based method. The major advantage of the proposed CSRN based method is the ability to perform prediction from new radio configurations without retraining the network, and, hence is more suitable for a real-time practical environment.

Facial expression recognition is a challenging task that involves detection and interpretation of complex and subtle changes in facial muscles. Recent advances in feed-forward deep neural networks (DNNs) have offered improved object recognition performance. Sparse feature learning in feed-forward DNN models offers further improvement in performance when compared to the earlier handcrafted techniques. However, the depth of the feed-forward DNNs and the computational complexity of the models increase proportional to the challenges posed by the facial expression recognition problem. The feed-forward DNN architectures do not exploit another important learning paradigm, known as recurrency, which is ubiquitous in the human visual system. Consequently, this paper proposes a novel biologically relevant sparse-deep simultaneous recurrent network (S-DSRN) for robust facial expression recognition. The feature sparsity is obtained by adopting dropout learning in the proposed DSRN as opposed to usual handcrafting of additional penalty terms for the sparse representation of data. Theoretical analysis of S-DSRN shows that the dropout learning offers desirable properties such as sparsity, and prevents the model from overfitting. Experimental results also suggest that the proposed method yields better performance accuracy, requires reduced number of parameters, and offers reduced computational complexity than that of the previously reported state-of-the-art feed-forward DNNs using two of the most widely used publicly available facial expression data sets. Furthermore, we show that by combining the proposed neural architecture with a state-of-the-art metric learning technique significantly improves the overall recognition performance. Finally, a graphical processing unit (GPU)-based implementation of S-DSRN is obtained for real-time applications.

In recent years, Convolutional Neural Networks (CNNs) have become very popular and have achieved great success in many computer vision tasks - particularly in object recognition. Partially inspired by neuroscience, CNNs share many properties with the visual system of the brain. However, the filters of convolutional layers play a vital role on overall accuracy of CNNs. In this paper, the Cellular Simultaneous Recurrent Networks (CSRNs) are applied to generate initial filters of Convolutional Networks (CNs) for features extraction and Regularized Extreme Learning Machines (RELM) are used for classification. Furthermore, Deep Belief Networks (DBN), CNNs with random and Gabor filters are implemented to evaluate the overall performance against the proposed CSRN's filters based CNs with RELM. Experiments were conducted on three popular datasets for object recognition (such as face, pedestrian, and car) to evaluate the performance of the proposed system. The experimental results show that in most of the cases, the proposed approach provides better performance on the extracted features using CSRN's filters with CNs compare to initialize with Gaussian random and DBN for object recognition.

This work proposes multiclass deep learning classification of Alzheimer's disease (AD) using novel texture and other associated features extracted from structural MRI. Two distinct learning models (Model 1 and 2) are presented where both include subcortical area specific feature extraction, feature selection and stacked auto-encoder (SAE) deep neural network (DNN). The models learn highly complex and subtle differences in spatial atrophy patterns using white matter volumes, gray matter volumes, cortical surface area, cortical thickness, and different types of Fractal Brownian Motion co-occurrence matrices for texture as features to classify AD from cognitive normal (CN) and mild cognitive impairment (MCI) in dementia patients. A five layer SAE with state-of-the-art dropout learning is trained on a publicly available ADNI dataset and the model performances are evaluated at two levels: one using in-house tenfold cross validation and another using the publicly available CADDementia competition. The in-house evaluations of our two models achieve 56.6% and 58.0% tenfold cross validation accuracies using 504 ADNI subjects. For the public domain evaluation, we are the first to report DNN to CADDementia and our methods yield competitive classification accuracies of 51.4% and 56.8%. Further, both of our proposed models offer higher True Positive Fraction (TPF) for AD class when compared to the top-overall ranked algorithm while Model 1 also ties for top diseased class sensitivity at 58.2% in the CADDementia challenge. Finally, Model 2 achieves strong disease class sensitivity with improvement in specificity and overall accuracy. Our algorithms have the potential to provide a rapid, objective, and non-invasive assessment of AD.

Many practical applications such as large scale image processing requires cost function optimization that is rather complex in nature. This renders the generic function approximators such as MLP quite inefficient in approximating such functions. A particular variant of universal approximators (UA) known as Cellular Simultaneous Recurrent Network (CSRN) is shown to be efficient in solving this type of complex functions. This cellular formation of SRN is specifically adept at approximating the topological mapping tasks typically found in image data processing. However, training CSRN for topological image processing is a challenging task because of the complexity of the network and the sheer size of the image data. Several representative training algorithms such as Back-propagation Through Time (BPTT), Extended Kalman Filtering (EKF), and Particle Swarm Optimization (PSO) have been used to train CSRN for different applications. However, none of the works in literature shows systematic approach to choose an appropriate learning algorithm for training a complex UA such as the CSRN in generalized image processing applications. Consequently, for the first time in literature, this work obtains generalization of the network architecture for image processing and simultaneously performs a systematic comparison to solve complex topological image mapping problems. It is shown that introduction of unconstrained Jacobian free learning algorithms such as Unscented Kalman Filter (UKF) and PSO alleviate the high computational cost otherwise associated with calculating Jacobian for EKF. Our result also shows that Jacobian-free training algorithms such as UKF and PSO perform better than Jacobian-based algorithm such as EKF in large scale image data processing.

Metric learning has been successful in distance based classification tasks. However, metric learning tends to become increasingly complex with the increase of input feature dimensionality. Therefore, application of an efficient feature extraction and dimensionality reduction technique prior to metric learning has been pursued. Conventional feature extraction and dimensionality reduction techniques used for metric learning are usually hand-crafted and may not offer the best overall performance. Contemporary methods such as deep neural networks (DNNs) along with metric learning have been used for improved feature extraction and dimensionality reduction through learning. While DNNs have exhibited excellent performance, such deep structures tend to get cumbersome with increasing complexity of the task. Consequently, this work attempts to introduce an efficient feature extraction and dimensionality reduction technique using a simultaneous recurrent network (SRN) architecture. The proposed SRN architecture with metric learning is tested on solving two complex classification tasks: facial expression and character recognition. Our results show that the proposed SRN feature extraction and metric learning classification pipeline achieves superior performance in comparison to a DNN-based feature reduction and metric learning pipeline. We also demonstrate that the proposed SRN manages to utilize far less trainable parameters than the comparable DNN model such as stacked autoencoders (SAEs) for the same set of tasks.

EEG is one the most effective tools used in the diagnosis of epilepsy. However, proper diagnosis of epilepsy requires the detection and analysis of epileptic seizures for a long period of time. Manual monitoring of long term EEG is tedious and costly. Therefore, a reliable automated seizure detection system is desirable. Most current state-of-the-art methods use hand crafted feature extraction and simple classification techniques, which often leads to sub-par performance due to lack of generalizability across patient dataset. Consequently, this work introduces novel deep recurrent neural network (DRNN) architecture to perform automated patient specific seizure detection using scalp EEG. We further propose a unique mapping of seizure EEG signal for efficient processing with the DRNN. This mapping allows the proposed deep architecture to simultaneously learn both temporal and spatial features of raw seizure EEG respectively. The proposed DRNN architecture is tested with long term patient specific scalp EEG data of 5 subjects with approximately 34 hours of EEG extracted from a publically available dataset. Overall, the proposed network successfully detects 100% of total seizure events with an average detection delay of ~7.0 sec. The results demonstrate superior performance to that of the current state-of-art seizure detection methods. The proposed DRNN architecture also obtains a runtime of approximately 30ms for 1 sec segment of 18 channel EEG. The low processing time with sparse use of computing resources and superior performance make the proposed architecture appropriate for real-time use.

Large scale feed forward neural networks have seen intense application in many computer vision problems. However, these networks can get hefty and computationally intensive with increasing complexity of the task. This work, for the first time in literature, introduces a Cellular Simultaneous Recurrent Network (CSRN) based hierarchical neural network for object detection. CSRN has shown to be more effective to solving complex tasks such as maze traversal and image processing when compared to generic feed forward networks. While deep neural networks (DNN) have exhibited excellent performance in object detection and recognition, such hierarchical structure has largely been absent in neural networks with recurrency. This work attempts to introduce deep hierarchy in CSRN for object detection. We propose a novel CSRN based DNN feature extractor that utilizes highly efficient filters derived from CSRNs used in the hidden convolutional layer. Experiments using a face detection task show that the CSRN based DNN offers comparable performance to the state-of-the-art convolutional neural network (CNN), while utilizing only as few as five filters in each of its hidden layers. In comparison, the CNN requires from a few to thousands of filters in each of its hidden layers for the same task. We also demonstrate the flexibility of the proposed architecture by introducing hybridization concept to the network to enhance its scale invariance. The hybrid scale invariant architecture is tested with randomly scaled object image dataset for face detection. Finally, we show that the CSRN based hybrid DNN performance is also comparable to that of the hybrid CNN.

Cellular Simultaneous Recurrent Network (CSRN) is a unique type of recurrent networks that is designed to solve complex optimization problems. This network has already shown to successfully solve many challenging problems such as 2D maze navigation, image registration and affine transformation, game of go, and power system voltage profile prediction. One of the main challenges of using a complex network structure as CSRN is to efficiently train the network. Many representative training algorithms such as Back-propagation Through Time (BPTT), Extended Kalman Filtering (EKF) and Particle Swarm Optimization (PSO) have been used to train CSRN. Our prior works with CSRN suggest that for large number of network inputs, which is very common for large scale maze and image data, computational complexity of computing Jacobian in EKF training becomes prohibitive. In this paper, we propose Unscented Kalman Filter (UKF) for the training of CSRN to avoid computing Jacobian. We show that CSRN trained with UKF can solve the 2D maze traversal problem with better convergence rate than that of EKF. We also report preliminary results on binary image affine transformation wherein CSRN trained with UKF offers comparable performance to that of EKF. A comparison has been obtained between CSRN with GMLP core versus an Elman core trained with UKF for Affine transform results. Finally, we show that for more complex applications such as large scale image processing, UKF is much faster than EKF in training CSRN.

We propose an adaptive channel allocation algorithm applicable in multi-carrier cellular CDMA systems. In our method, traffic channels are assigned to a mobile user based on cell traffic load and user mobility pattern. Depending on the mobility pattern (and the probability of moving to a neighboring cell), the algorithm selects the best-suited carrier (for the mobile), which may not always be the least loaded carrier in the cell. Performance measurements include percentage of calls blocked, percentage of calls dropped, percentage of supported call duration (in relation to its total required duration), and call quality (average number of legs supported during soft handoff). Simulation results show that the algorithm offers significant system performance improvements over hashing based algorithms and is suitable for next generation cellular system.