LCN

Deep spiking neural networks (SNNs) offer the promise of low-power artificial intelligence. However, training deep SNNs from scratch or converting deep artificial neural networks to SNNs without loss of performance has been a challenge. Here we propose an exact mapping from a network with Rectified Linear Units (ReLUs) to an SNN that fires exactly one spike per neuron. For our constructive proof, we assume that an arbitrary multi-layer ReLU network with or without convolutional layers, batch normalization and max pooling layers was trained to high performance on some training set. Furthermore, we assume that we have access to a representative example of input data used during training and to the exact parameters (weights and biases) of the trained ReLU network. The mapping from deep ReLU networks to SNNs causes zero percent drop in accuracy on CIFAR10, CIFAR100 and the ImageNet-like data sets Places365 and PASS. More generally our work shows that an arbitrary deep ReLU network can be replaced by an energy-efficient single-spike neural network without any loss of performance.

Continually learning new classes from fresh data without forgetting previous knowledge of old classes is a very challenging research problem. Moreover, it is imperative that such learning must respect certain memory and computational constraints such as (i) training samples are limited to only a few per class, (ii) the computational cost of learning a novel class remains constant, and (iii) the memory footprint of the model grows at most linearly with the number of classes observed. To meet the above constraints, we propose C-FSCIL, which is architecturally composed of a frozen meta-learned feature extractor, a trainable fixed-size fully connected layer, and a rewritable dynamically growing memory that stores as many vectors as the number of encountered classes. C-FSCIL provides three update modes that offer a trade-off between accuracy and compute-memory cost of learning novel classes. C-FSCIL exploits hyperdimensional embedding that allows to continually express many more classes than the fixed dimensions in the vector space, with minimal interference. The quality of class vector representations is further improved by aligning them quasi-orthogonally to each other by means of novel loss functions. Experiments on the CIFAR100, mini-ImageNet, and Omniglot datasets show that C-FSCIL outperforms the baselines with remarkable accuracy and compression. It also scales up to the largest problem size ever tried in this few-shot setting by learning 423 novel classes on top of 1200 base classes with less than 1.6% accuracy drop. Our code is available at https://github.com/IBM/constrained-FSCIL.

Developing energy-saving neural network models is a topic of rapidly increasing interest in the artificial intelligence community. Spiking neural networks (SNNs) are biologically inspired models that strive to leverage the energy efficiency stemming from a long process of evolution under limited resources. In this paper we propose a SNN model where each neuron integrates piecewise linear postsynaptic potentials caused by input spikes and a positive bias, and spikes maximally once. Transformation of such a network into the ANN domain yields an approximation of a standard ReLU network, leading to a facilitated training based on backpropagation and an adaptation of the batch normalization. With backpropagation-trained weights, SNN inference offers a sparse-signal and low-latency classification, which can be readily adapted for a stream of input patterns, lending itself to an efficient hardware implementation. The supervised classification of MNIST and Fashion-MNIST datasets, using this approach, provides accuracy close to that of an ANN and surpassing other single-spike SNNs.

In this paper, we propose a system for file classification in large data sets based on spiking neural networks (SNNs). File information contained in key-value metadata pairs is mapped by a novel correlative temporal encoding scheme to spike patterns that are input to an SNN. The correlation between input spike patterns is determined by a file similarity measure. Unsupervised training of such networks using spike-timing-dependent plasticity (STDP) is addressed first. Then, supervised SNN training is considered by backpropagation of an error signal that is obtained by comparing the spike pattern at the output neurons with a target pattern representing the desired class. The classification accuracy is measured for various publicly available data sets with tens of thousands of elements, and compared with other learning algorithms, including logistic regression and support-vector machines. Simulation results indicate that the proposed SNN-based system using memristive synapses may represent a valid alternative to classical machine learning algorithms for inference tasks, especially in environments with asynchronous ingest of input data and limited resources.

In multi-tier storage systems with large amounts of data, most of the data is stored on inexpensive slower tiers such as cloud or tape to achieve cost savings. This also implies that retrieving the data from the slower storage tiers incurs high latency. Therefore, it would be beneficial to proactively prefetch data from slower tiers to faster tiers by predicting future data accesses. State-of-the-art access prediction methods typically record access history of individual files, data objects, or data segments. However, in systems with large amounts of infrequently accessed (or cold) data, file-level access history is often unavailable for much of the data due to the low frequency of access. In this paper, we extract information from file metadata to predict file accesses in a storage system. The proposed method relies on the hypothesis that users and applications access data stored in the system in a given context and that the context and, therefore, the set of files that are likely to be accessed can be identified by detecting access patterns in file metadata. As an application, we consider the LOFAR radio telescope's long term archive, where the access patterns are learned based on a rich set of metadata, and these patterns are then used to make predictions as to likely future accesses by the astronomers.

Imaging techniques in radio interferometry often face a significant challenge posed by the large number of antenna signals received, from which the image information needs to be extracted. Beamforming is envisaged to reduce the rate required for transporting data from groups of antennas to a central site for further processing. We propose a novel method for image reconstruction based on the iterative scanning of a region of interest, combined with randomized beamforming. A modified approximate message-passing algorithm is adopted to extract relevant image information from beamformed signals received at the antenna stations. The method is illustrated by simulations, with reference to the LOFAR radio interferometer, and compared with the CLEAN algorithm.

The Square Kilometre Array (SKA) will form the largest radio telescope ever built and such a huge instrument in the desert poses enormous engineering and logistic challenges. Algorithmic and architectural breakthroughs are needed. Data is collected and processed in groups of antennas before transport for central processing. This processing includes beamforming, primarily so as to reduce the amount of data sent. The principal existing technique points to a region of interest independently of the sky model and how the other stations beam-form. We propose a new collaborative beamforming algorithm in order to maximize information captured at the stations (thus reducing the amount of data transported). The method increases the diversity in measurements through randomized beamforming. We demonstrate through numerical simulation the effectiveness of the method. In particular, we show that randomized beamforming can achieve the same image quality while producing 40% less data when compared to the prevailing method matched beamforming.

Probe-storage devices employ large arrays of probes to write/read data in parallel in some storage medium, and combine ultra-high density, low access times, and low power consumption. A particular probe-storage technique utilizes thermomechanical means to store and retrieve information in thin polymer films. In this paper, a system-level channel model for the thermomechanical probe-storage channel is presented. Each of the components of the proposed model is derived by extensive characterization of experimentally obtained readback signals from probe recording tests. Moreover, detection techniques that are actually utilized in a probe-storage prototype implementation are described, followed by coding techniques for added reliability in the presence of particles or other impurities of the storage medium. In addition to low-complexity coding constructs, a concatenated coding scheme with an outer LDPC and inner modulation code is considered, in order to establish a benchmark for overall system performance. A novel methodology for joint decoding of outer LDPC and inner (d,k) modulation codes is developed. Furthermore, an optimal soft decoder for the modulation code is proposed, based on a modification of the decoder metrics to accurately account for the probe storage channel output statistics. Experimental results are used throughout the paper to validate the channel model and identify its relevant parameters, as well as to verify the system performance obtained by simulations.

Frame synchronization is studied for pulse-position modulation-encoded longitudinal position (LPOS) words that are embedded in servo patterns written on tape. After introducing soft-output detection of LPOS symbols, the problem of LPOS frame synchronization using soft outputs from the LPOS detector is considered. An efficient LPOS frame-synchronization algorithm that is robust in the presence of transition noise found in magnetic recording channels is then proposed. Finally, the performance of hard-decision LPOS frame synchronization is compared by simulations with that of the proposed soft-decision algorithm, and a hardware implementation of the overall scheme is illustrated.

Probe-storage devices employ large arrays of probes to write/read data in parallel in some storage medium. Because of their inherent parallelism and the independence of data retrieved from different probes, these devices lend themselves naturally to low-density parity-check (LDPC) codes and associated soft/iterative decoding techniques. In this paper, a concatenated coding scheme for a particular probe-storage channel is presented that comprises an inner (d, k)-constrained code and an outer LDPC code, and the problem of probabilistic data retrieval for this channel is addressed. In particular, soft information is generated by explicitly accounting for the channel output statistics in the presence of jitter and additive noise, based on derived analytical expressions.

Scanning-probe data storage is emerging as a viable alternative to conventional data storage, offering ultra-high density, low access times, and low power consumption. One probe-storage technique utilizes a thermomechanical means to store and retrieve information in thin polymer films. We describe the readback signal path and characterize the thermomechanical-based probe-storage recording channel. It is shown that this channel exhibits a particular nonlinear behavior at high storage densities or high recording power, that is, the energy per unit time used to write a bit of information. A simple model is proposed that accurately captures the characteristics of this nonlinearity. Experimental results from single-probe recording setups are used to verify the validity of this model and identify its relevant parameters.

A filter-bank modulation technique called filtered multitone (FMT) and its application to data transmission for very high-speed digital subscriber line technology are described. The proposed scheme leads to significantly lower spectral overlapping between adjacent subchannels than for known multicarrier techniques such as discrete multitone (DMT) or discrete wavelet multitone. FMT modulation mitigates interference due to echo and near-end crosstalk signals, and increases the system throughput and reach. Signal equalization in an FMT receiver is accomplished in the form of per-subchannel symbol-spaced or fractionally spaced linear or decision-feedback equalization. The problem of channel coding for this type of modulation is also addressed, and an approach that allows combined removal of intersymbol-interference via precoding and trellis coding is described. Furthermore, practical design aspects regarding filter-bank realization, initial transceiver training, adaptive equalization, and timing recovery are discussed. Finally, simulation results of the performance achieved by FMT modulation for very high-speed digital subscriber line systems, where upstream and downstream signals are separated by frequency-division duplexing, are presented and compared with DMT modulation.

The problem of multiuser detection for upstream very-high-speed digital subscriber line (VDSL) transmission, where far-end crosstalk (FEXT) signals at the input of a VDSL receiver are viewed as interferers that share the same channel as the remote user signal, is addressed. Knowledge of the FEXT impulse responses at the central office, as well as upstream transmission based on multicarrier modulation by all remote users, are assumed. The joint application of upstream power back-off with a reduced-complexity decision-feedback (DFE) multiuser detector, where only the most significant crosstalk interferers are suppressed, is considered, and the coefficients of the minimum mean-square error structure are determined. Then a novel optimum power back-off algorithm is introduced, and its performance in terms of achievable rates evaluated. Finally, numerical results that illustrate the impact of power back-off on achievable performance of a VDSL system with multiuser detection are presented.

100BASE-T2 is a new physical layer specification for IEEE 802.3 LANs operating at 100 Mbit/s ("fast Ethernet"). It enables users of the 10BASE-T Ethernet LAN technology to upgrade their networks from 10 Mbit/s to 100 Mbit/s performance while retaining an existing voice-grade cabling infrastructure. 100BASE-T2 transceivers will operate over two pairs in unshielded twisted-pair cables corresponding to EIA/TIA category 3 (UTP-3), which is the minimum requirement for 10BASE-T. In a four-pair UTP-3 cable, simultaneous operation of two 100BASE-T2 links, or one 100BASE-T2 and one 10BASE-T link, is permitted. Since voice-grade cables exhibit more signal attenuation and significantly higher crosstalk coupling between adjacent pairs than data-grade cables, sophisticated digital signal processing techniques are needed to achieve reliable 100 Mbit/s transmission. The 100BASE-T2 standard defines dual-duplex baseband transmission at a modulation rate of 25 MBaud. During each modulation interval, a four-bit data nibble or Ethernet specific control information is encoded into a pair of quinary signals. These signals are transmitted simultaneously on the two wire pairs in both signalling directions. In the receivers, adaptive digital filters are required for echo and NEXT cancellation, equalization, and interference suppression.

This paper deals with multilevel partial-response class-IV (PRIV) transmission over unshielded twisted-pair (UTP) cables. Specifically, transmission at a rate of 155.52 Mb/s over data-grade UTP cables for local-area networking is considered. As a low-complexity method used to compensate for cable-length dependent signal distortion, adaptive analog equalization with two controlled parameters is proposed: one parameter determines a frequency-independent receiver gain, the other parameter controls the transfer characteristic of a variable analog receive-filter section. For the stepwise design of the transmit and receive filters, a combination of analytic techniques and simulated annealing is employed. First, the variable equalizer section, then the remaining fixed analog receive filter section are developed and finally the analog transmit filter is determined. The paper also describes the adjustment of the equalizer section, and the control of the sampling phase in the receiver front-end. The two equalizer parameters are controlled by an algorithm that operates on the sampled signals and adjusts these parameters to optimum settings independently of the sampling phase. The latter is controlled by a decision-directed phase-locked loop algorithm that becomes effective when equalization has been achieved. The dynamic behaviour and mean-square error in steady-state obtained with these control algorithms are investigated.

The paper describes an experimental transceiver for full-duplex transmission at a rate of 125 Mbit/s over unshielded twisted-pair cables of ordinary voice-grade quality, intended for use in a fiber distributed data interface (FDDI) network. Quaternary partial-response class-IV (QPRIV) overall-channel signaling with near-end crosstalk (NEXT) cancellation and maximum-likelihood sequence detection is employed. The spectral shape of the QPRIV signals facilitates equalization and achieving compliance with EMC regulations. Since in an FDDI system each transmitter can be clocked independently, the receiver must cope with phase drift between NEXT signals to be cancelled and signals received from the remote transmitter. With the chosen transceiver architecture, digital-to-analog conversion of transmit signals, analog-to-digital conversion of receive signals, and adaptive NEXT cancellation are performed synchronously with the transmitter clock. The rate change from transmit timing to controlled receive timing is accomplished by an adaptive equalizer in conjunction with an elastic buffer and occasional coefficient shifts. The equalizer is adjusted rapidly enough to allow for a maximal phase drift of ±100 ppm. The implementation of all digital signal-processing functions in a single 0.5 μm CMOS VLSI prototype chip is discussed. The employed standard-cell design resulted in a power consumption of 6 W. Significantly lower power consumption can be achieved by custom design of highly repetitive processing elements

The authors discuss a new self-training algorithm for adaptive equalization of multilevel partial-response class-IV (PRIV) systems. Adaptive distributed-arithmetic equalizers are considered, where the process of multiplying the tap signals with tap gains and summing the resulting products is replaced by a procedure involving only table look-up and shift-and-add operations. Classical self-training adaptation schemes for linear adaptive equalizers do not converge if applied to distributed-arithmetic equalizers, because of the inherent nonlinearity of the system during the adaptation process. The authors show that, by adopting a generalized stochastic gradient to adjust the look-up tables, the mean-square error converges to a value which depends on the system parameters. For practical system implementation, a two-step modified algorithm is proposed to closely approach in the steady state the minimum achievable mean-square error. Numerical results are presented for multilevel PRIV systems for high-rate data transmission over twisted-pair cables.>

Adaptive distributed-arithmetic echo cancellers are well suited for full-duplex high-speed data transmission. They allow a simpler implementation than adaptive linear transversal filters, since multiplications are replaced by table look-up and shift-and-add operations. Various tradeoffs between the number of operations and the number of memory locations of the look-up tables can be achieved by segmenting the echo canceller delay line into sections of shorter length. Adaptivity is achieved by a decision-directed stochastic gradient algorithm to adjust the contents of the look-up tables. The author adopts the mean-square error criterion to investigate the convergence behavior of adaptive distributed-arithmetic echo cancellers. Under the assumption that the look-up values are statistically independent of the symbols stored in the echo canceller delay line, he obtains an analytical expression for the mean-square error as a function of time. The maximum speed of convergence and the corresponding optimum adaptation gain are also determined. Simulation results for a full-duplex quaternary partial response class-IV system are presented and compared with theoretical results.>

The authors present and analyze a model for both hybrid and frequency-hopped spread-spectrum systems in which a fully digital coherent receiver is used to demodulate the data. A receiver for a hybrid DS/FH (direct-sequence/frequency-hopped) system using a digital delay-lock loop is considered. In the absence of frequency uncertainty, it is shown that the tracking error can be modeled as an ergodic Markov chain with a finite-state set, and the probability density function of the steady-state tracking error is evaluated. When there is a frequency uncertainty, the dynamics of the resulting nonstationary phase error can be obtained, and an expression to evaluate the probability distribution of the first time at which the phase error hits predetermined boundary values is derived. Bit error rate performance is determined in the presence of both additive white Gaussian noise and various types of interference, and the performance is compared to that of noncoherent FSK systems.>

For pt.I see ibid., vol.37, no.6, p.601-11 (1989). A pure phase-coherent frequency-hop (FH) system is described, and a digital receiver based on maximum-likelihood estimation of the phase of the frequency-hopped carrier is suggested. The theory developed in pt.I is used in the analysis of the receiver whose performance in the presence of both additive white Gaussian noise and partial-band interference is evaluated. By comparison to the bit error rate performance of noncoherent FSK systems, it is found that for channels without excessive phase distortion, an improvement is achievable with the coherent system, but at a price of a more complex transmitter and receiver.>

The performance analysis of a fully digital acquisition circuit for direct sequence spread-spectrum systems has been made. The disturbance is considered Gaussian but with a colored spectral density. The effects of the disturbance correlation show remarkable effects on the detection probability and on the mean acquisition time at low signal-to-noise ratios. An example of an application is reported.