Applying machine learning techniques to the quickly growing data in science and industry requires highly-scalable algorithms. Large datasets are most commonly processed "data parallel" distributed across many nodes. Each node's contribution to the overall gradient is summed using a global allreduce. This allreduce is the single communication and thus scalability bottleneck for most machine learning workloads. We observe that frequently, many gradient values are (close to) zero, leading to sparse of sparsifyable communications. To exploit this insight, we analyze, design, and implement a set of communication-efficient protocols for sparse input data, in conjunction with efficient machine learning algorithms which can leverage these primitives. Our communication protocols generalize standard collective operations, by allowing processes to contribute arbitrary sparse input data vectors. Our generic communication library, SparCML1, extends MPI to support additional features, such as non-blocking (asynchronous) operations and low-precision data representations. As such, SparCML and its techniques will form the basis of future highly-scalable machine learning frameworks.

This paper presents the implementation of a low-power and implantable neuroprocessor on low-cost nano-FPGA for data reduction and on-the-fly spike sorting in Brain Machine Interface applications. Detailed analysis of efficient utilization of the hardware resources, power consumption and design scalability are provided. The prototype we report here enables simultaneous processing of 32-channel data sampled at 25 kHz/channel with 8-bit/sample resolution with less than 5 mW power consumption for all modes of operation (monitoring, compression and sensing) at 1.2 V core voltage supply on a 5 mm × 5 mm nano-FPGA.

An essential step towards understanding how the brain orchestrates information processing at the cellular and population levels is to simultaneously observe the spiking activity of cortical neurons that mediate perception, learning, and motor processing. In this paper, we formulate an information theoretic approach to determine whether cooperation among neurons may constitute a governing mechanism of information processing when encoding external covariates. Specifically, we show that conditional independence between neuronal outputs may not provide an optimal encoding strategy when the firing probability of a neuron depends on the history of firing of other neurons connected to it. Rather, cooperation among neurons can provide a "message-passing" mechanism that preserves most of the information in the covariates under specific constraints governing their connectivity structure. Using a biologically plausible statistical learning model, we demonstrate the performance of the proposed approach in synergistically encoding a motor task using a subset of neurons drawn randomly from a large population. We demonstrate its superiority in approximating the joint density of the population from limited data compared to a statistically independent model and a maximum entropy (MaxEnt) model.

Estimating rate functions underlying neural point processes is essential for characterizing the firing patterns of cortical neurons involved in sensory and motor processing. This paper introduces a new method for directly estimating neuronal firing rates from a compressed representation of their extracellular recordings. The approach is based on extending a near-optimal sparse representation of the extracellular recordings to time scales matching those of the underlying rate functions, thereby performing the same role as a kernel-density estimator but at a much lower computational cost. Experimental results demonstrate that this method achieves comparable performance to the conventional Gaussian kernel estimation methods at a fraction of the computational cost. This makes it suitable to brain machine interface applications where real time decoding of neural activity for neuro-motor prosthetic control using firing rate estimates is strongly desired.

Understanding the mechanism underlying distributed neural coding is a fundamental goal in computational neuroscience. With the ability to simultaneously observe the activity of large networks of neurons in response to external stimuli, a natural question arises: how the outside world is represented in the collective activity of these neurons? In this work, we provide an information theoretic approach for determining the role of cooperation among neurons in encoding external stimuli. Specifically, we show that statistical independence between neuronal outputs may not provide the best coding strategy when these outputs depend on the history of other neuronal constituents in the network. Rather, cooperation among neurons can provide a near optimal and lossless coding strategy under specific constraints governing their network structure. Using a statistical learning model, we demonstrate the performance of the proposed approach in decoding a motor task with both discrete targets and continuous trajectory using spike trains from a small subset of a large network. We demonstrate its superiority in minimizing the decoding error compared to a statistically independent model and to other classical decoders reported in the literature.

Mutual information has been used in many clustering algorithms for measuring general dependencies between random data variables, but its difficulties in computing for small size datasets has limited its efficiency for clustering in many applications. A novel clustering method is proposed which estimates mutual information based on information potential computed pair-wise between data points and without any prior assumptions about cluster density function. The proposed algorithm increases the mutual information in each step in an agglomerative hierarchy scheme. We have shown experimentally that maximizing mutual information between data points and their class labels will lead to an efficient clustering. Experiments done on a variety of artificial and real datasets show the superiority of this algorithm, besides its low computational complexity, in comparison to other information based clustering methods and also some ordinary clustering algorithms.

This paper presents a novel method for detecting multiple frontal faces in still images using multi-scale processing. The main characteristic of this algorithm is its stability in detecting faces with seldom false detections and a high correct detection rate. The novelty of this work comes from the utilization of multiscale detection and using two classifiers to reduce false detections. The algorithm generally has two stages: in the first stage, a face is detected in a unique scale and in the second stage, only the faces that are located in the neighbor scales are accepted as real faces. Consequently, a still image is first resized and scanned block wise, and then each enhanced block is tested for being face. One dimensional Harr wavelet is used for feature extraction, which gives appropriate discriminating features between the face and nonface classes. Detection results at each scale are accumulated in an internal database, so the ultimate detection is prepared based on the mutual detection information between consequent scales. To parameterize both the Bayesian and the simple proposed classifier, 2,643 faces were congregated from famous face databases and more than 10,000 non-face samples were selected from nature images. Experimental results using images gathered from known databases like MIT-CMU show great ability of the proposed algorithm in detecting faces.