We introduce a new neural network model, together with a tractable and monotone online learning algorithm. Our model describes feed-forward networks for classification, with one output node for each class. The only nonlinear operation is rectification using a ReLU function with a bias. However, there is a rectifier on every edge rather than at the nodes of the network. There are also weights, but these are positive, static, and associated with the nodes. Our rectified wire networks are able to represent arbitrary Boolean functions. Only the bias parameters, on the edges of the network, are learned. Another departure in our approach, from standard neural networks, is that the loss function is replaced by a constraint. This constraint is simply that the value of the output node associated with the correct class should be zero. Our model has the property that the exact norm-minimizing parameter update, required to correctly classify a training item, is the solution to a quadratic program that can be computed with a few passes through the network. We demonstrate a training algorithm using this update, called sequential deactivation (SDA), on MNIST and some synthetic datasets. Upon adopting a natural choice for the nodal weights, SDA has no hyperparameters other than those describing the network structure. Our experiments explore behavior with respect to network size and depth in a family of sparse expander networks.

Buoyed by the success of deep multilayer neural networks, there is renewed interest in scalable learning of Bayesian neural networks. Here, we study algorithms that utilize recent advances in Bayesian inference to efficiently learn distributions over network weights. In particular, we focus on recently proposed assumed density filtering based methods for learning Bayesian neural networks -- Expectation and Probabilistic backpropagation. Apart from scaling to large datasets, these techniques seamlessly deal with non-differentiable activation functions and provide parameter (learning rate, momentum) free learning. In this paper, we first rigorously compare the two algorithms and in the process develop several extensions, including a version of EBP for continuous regression problems and a PBP variant for binary classification. Next, we extend both algorithms to deal with multiclass classification and count regression problems. On a variety of diverse real world benchmarks, we find our extensions to be effective, achieving results competitive with the state-of-the-art.

We address the problem of planning collision-free paths for multiple agents using optimization methods known as proximal algorithms. Recently this approach was explored in Bento et al. (2013), which demonstrated its ease of parallelization and decentralization, the speed with which the algorithms generate good quality solutions, and its ability to incorporate different proximal operators, each ensuring that paths satisfy a desired property. Unfortunately, the operators derived only apply to paths in 2D and require that any intermediate waypoints we might want agents to follow be preassigned to specific agents, limiting their range of applicability. In this paper we resolve these limitations. We introduce new operators to deal with agents moving in arbitrary dimensions that are faster to compute than their 2D predecessors and we introduce landmarks, space-time positions that are automatically assigned to the set of agents under different optimality criteria. Finally, we report the performance of the new operators in several numerical experiments.

We describe a new instance-based learning algorithm called the Boundary Forest (BF) algorithm, that can be used for supervised and unsupervised learning. The al- gorithm builds a forest of trees whose nodes store previ- ously seen examples. It can be shown data points one at a time and updates itself incrementally, hence it is nat- urally online. Few instance-based algorithms have this property while being simultaneously fast, which the BF is. This is crucial for applications where one needs to respond to input data in real time. The number of chil- dren of each node is not set beforehand but obtained from the training procedure, which makes the algorithm very flexible with regards to what data manifolds it can learn. We test its generalization performance and speed on a range of benchmark datasets and detail in which settings it outperforms the state of the art. Empirically we find that training time scales as O(DN log(N )) and testing as O(Dlog(N)), where D is the dimensionality and N the amount of data.

We describe a novel approach for computing collision-free \emph{global} trajectories for $p$ agents with specified initial and final configurations, based on an improved version of the alternating direction method of multipliers (ADMM) algorithm. Compared with existing methods, our approach is naturally parallelizable and allows for incorporating different cost functionals with only minor adjustments. We apply our method to classical challenging instances and observe that its computational requirements scale well with $p$ for several cost functionals. We also show that a specialization of our algorithm can be used for {\em local} motion planning by solving the problem of joint optimization in velocity space.

We present an approach to designing capacity-approaching high-girth low-density parity-check (LDPC) codes that are friendly to hardware implementation, and compatible with some desired input code structure defined using a protograph. The approach is based on a mapping of any class of codes defined using a protograph into a family of hierarchical quasi-cyclic (HQC) LDPC codes. Whereas the parity check matrices of standard quasi-cyclic (QC) LDPC codes are composed of circulant submatrices, those of HQC LDPC codes are composed of a hierarchy of circulant submatrices that are, in turn, constructed from circulant submatrices, and so on, through some number of levels. Next, we present a girth-maximizing algorithm that optimizes the degrees of freedom within the family of codes to yield a high-girth HQC LDPC code, subject to bounds imposed by the fact that HQC codes are still quasi-cyclic. Finally, we discuss how certain characteristics of a code protograph will lead to inevitable short cycles and show that these short cycles can be eliminated using a “squashing” procedure that results in a high-girth QC LDPC code, although not a hierarchical one. We illustrate our approach with three design examples of QC LDPC codes-two girth-10 codes of rates 1/3 and 0.45 and one girth-8 code of rate 0.7-all of which are obtained from protographs of one-sided spatially coupled codes.

Cooperation between the nodes of wireless multihop networks can increase communication reliability, reduce energy consumption, and decrease latency. The possible improvements are even greater when nodes perform mutual information accumulation. In this paper, we investigate resource allocation for unicast and multicast transmission in such networks. Given a network, a source, and a destination, our objective is to minimize end-to-end transmission delay under energy and bandwidth constraints. We provide an algorithm that determines which nodes should participate in forwarding the message and what resources (time, energy, bandwidth) should be allocated to each. Our approach factors into two sub-problems, each of which can be solved efficiently. For any transmission order we show that solving for the optimum resource allocation can be formulated as a linear programming problem. We then show that the transmission order can be improved systematically by swapping nodes based on the solution of the linear program. Solving a sequence of linear programs leads to a locally optimal solution in a very efficient manner. In comparison to the proposed cooperative routing solution, it is observed that conventional shortest path multihop routing typically incurs additional delays and energy expenditures on the order of 70%. Drawing inspiration from this first, centralized, algorithm, we also present two distributed algorithms. These algorithms require only local channel state information. Simulations indicate that they yield solutions about two to five percent less efficient than the centralized algorithm.

The “Divide and Concur” (DC) algorithm introduced by Gravel and Elser can be considered a competitor to the belief propagation (BP) algorithm, in that both algorithms can be applied to a wide variety of constraint satisfaction, optimization, and inference problems. We show that DC can be interpreted as a message-passing algorithm on a “normal” factor graph. The “difference-map” dynamics of the DC algorithm enables it to avoid “traps” which may be related to the “trapping sets” or “pseudo-codewords” that plague BP decoders of low-density parity check (LDPC) codes in the error-floor regime. We investigate two decoders for LDPC codes based on these ideas. The first decoder is based directly on DC, while the second decoder borrows the important “difference-map” concept from the DC algorithm and translates it into a BP-like decoder. We show that this “difference-map belief propagation” (DMBP) decoder has dramatically improved error-floor performance compared to standard BP decoders, while maintaining a similar computational complexity. We present simulation results for LDPC codes comparing DC and DMBP decoders with other decoders based on sum-product BP, linear programming, and mixed-integer linear programming. We also describe the close relation of the DMBP decoder to reweighted min-sum algorithms, including those recently proposed by Ruozzi and Tatikonda.

In this paper we present a three-stage decoding strategy that combines quantized and un-quantized belief propagation (BP) decoders with a mixed-integer linear programming (MILP) decoder. Each decoding stage is activated only when the preceding stage fails to converge to a valid codeword. The faster BP decoding stages are able to correct most errors, yielding a short average decoding time. Only in the rare cases when the iterative stages fail is the slower but more powerfulMILP decoder used. The MILP decoder iteratively adds binary constraints until either the maximum likelihood codeword is found or some maximum number of binary constraints has been added. Simulation results demonstrate a large improvement in the word error rate (WER) of the proposed multi-stage decoder in comparison to belief propagation. The improvement is particularly noticeable in the low crossover probability (error floor) regime. Through introduction of an accelerated ¿active-set¿ version of the quantized BP decoder we significantly speed up the pace of simulation to simulate low density parity check (LDPC) codes of length up to around 2000 down to a WER of around 10-10on the binary symmetric channel. We demonstrate that for certain codes our approach can efficiently approach the optimal ML decoding performance for low crossover probabilities.

Cooperation between the nodes of wireless multi-hop networks can increase communication reliability, reduce energy consumption, and decrease latency. The possible improvements are even greater when nodes perform mutual-information accumulation, e.g., by using rateless codes. In this paper, we investigate routing problems in such networks. Given a network, a source and a destination, our objective is to minimize end-to-end transmission delay under a sum energy constraint. We provide an algorithm that determines which nodes should participate in forwarding the message and what resources (time, energy, bandwidth) should be allocated to each. Our approach factors into two sub-problems, each of which can be solved efficiently. For any node decoding order we show that solving for the optimum resource allocation can be formulated as a linear problem. We then show that the decoding order can be improved systematically by swapping nodes based on the solution of the linear program. Solving a sequence of linear program leads to a locally optimum solution in a very efficient manner. In comparison to the cooperative routings, it is observed that conventional shortest-path multihop routings incur additional delays and energy expenditures on the order of 70%. Since this initial solution is centralized, requiring full channel state information, we exploit the insights to design two distributed routing algorithms that require only local channel state information. We provide simulations showing that in the same networks the distributed algorithms find routes that are only about 2-5% less efficient than the centralized solution.

We present an information-theoretically secure biometric storage system using graph-based error correcting codes in a Slepian-Wolf coding framework. Our architecture is motivated by the noisy nature of personal biometrics and the requirement to provide security without storing the true biometric at the device. The principal difficulty is that real biometric signals, such as fingerprints, do not obey the i.i.d. or ergodic statistics that are required for the underlying typicality properties in the Slepian-Wolf coding framework. To meet this challenge, we propose to transform the biometric data into binary feature vectors that are i.i.d. Bernoulli(0.5), independent across different users, and related within the same user through a BSC-p channel with small p< 0.5. Since this is a standard channel model for LDPC codes, the feature vectors are now suitable for LDPC syndrome coding. The syndromes serve as secure biometrics for access control. Experiments on a fingerprint database demonstrate that the system is information-theoretically secure, and achieves very low false accept rates and low false reject rates.

We describe a method to encode fingerprint biometrics securely for use, e.g., in encryption or access control. The system is secure because the stored data does not suffice to recreate the original fingerprint biometric. Therefore, a breach in database security does not lead to the loss of biometric data. At the same time the stored data suffices to validate a probe fingerprint. Our approach is based on the use of distributed source coding techniques implemented with graph-based codes. We present a statistical model of the relationship between the enrollment biometric and the (noisy) biometric measurement taking during authentication. We describe how to validate or reject a candidate biometric probe given the probe and the stored encoded data. We report the effectiveness of our method as tested on a database consisting of 579 data sets, each containing roughly 15 measurements of a single finger. We thereby demonstrate a working secure biometric system for fingerprints.

We present a newly developed airborne InSAR system incorporating a novel phase unwrapping algorithm, capable of retrieving a highly accurate Digital Surface Model (DSM). The SAR sensor system, with a spatial resolution of 30 cm, is carried on an airborne platform which has 2 antennas placed in a baseline length of 1 m. We have established a DSM reconstruction processing technique, which includes the new "Iterated Conditional Modes-Minimum Cost Flow" (ICM-MCF) phase-unwrapping algorithm. The ICM-MCF algorithm finds a locally optimal configuration of unwrapped phases under a well-characterized statistical model of the terrain and noise. An experimental field observation was carried out in Tsukuba, Japan. The DSM was generated, and the height accuracy of the SAR-DSM was evaluated by comparing with laser profiler data. For 50 cm x 50 cm mesh, an accuracy of better than 50 cm in height was confirmed.

Linear programming (LP) decoding was introduced by Feldman et al. (IEEE Trans. Inform. Theory Mar. 2005) as a novel way to decode binary low-density parity-check codes. Taghavi and Siegel (Proc. ISIT 2006) describe a computationally simplified decoding approach they term "adaptive" LP decoding. Adaptive LP decoding starts with a sub-set of the LP constraints, and iteratively adds violated constraints until an optimum of the original LP is found. Usually only a tiny fraction of the original constraints need to be reinstated, leading to huge efficiency gains compared to ordinary LP decoding. Here we describe a modification of the adaptive LP decoder that results in a maximum likelihood (ML) decoder. Whenever the adaptive LP decoder returns a pseudo-codeword rather than a codeword, we add an integer constraint on the least certain symbol of the pseudo-codeword. For certain codes, and especially in the high-SNR (error floor) regime, only a few integer constraints are required to force the resultant mixed-integer LP to the ML solution. We demonstrate that our approach can efficiently achieve the optimal ML decoding performance on a (155,64) LDPC code introduced by Tanner et al.

We investigate the performance of iterative decoding algorithms for multistep majority logic decodable (MSMLD) codes of intermediate length. We introduce a new bit-flipping algorithm that is able to decode these codes nearly as well as a maximum-likelihood decoder on the binary-symmetric channel. We show that MSMLD codes decoded using bit-flipping algorithms can outperform comparable Bose-Chaudhuri-Hocquenghem (BCH) codes decoded using standard algebraic decoding algorithms, at least for high bit-flip rates (or low and moderate signal-to-noise ratios (SNRs)).

Replica shuffled versions of iterative decoders for low-density parity-check (LDPC) codes and turbo codes are presented. The proposed schemes can converge faster than standard and plain shuffled approaches. Two methods, density evolution and extrinsic information transfer (EXIT) charts, are used to analyze the performance of the proposed algorithms. Both theoretical analysis and simulations show that the new schedules offer good tradeoffs with respect to performance, complexity, latency, and connectivity

Cooperative communications, where parallel relays forward information to a destination node, can greatly improve the energy efficiency and latency in ad-hoc networks. However, current networks do not fully exploit its potential as they only use traditional energy-accumulation, which is often used in conjunction with repetition coding or cooperative space-time codes. In this paper, we show that the concept of mutual- information-accumulation can be realized with the help of fountain codes, and leads to a lower energy expenditure and a lower transmission time than energy accumulation. We then provide an analysis of the performance of mutual information accumulation in relay networks with N relay nodes. We first analyze the quasi-synchronuous scenario where the source stops transmitting and the relay nodes start transmitting after L relay nodes have successfully decoded the source data. We show that an optimum L exists, and is typically on the order of 3 or 4. We also give closed-form equations for the energy savings that can be achieved by the use of mutual-information-accumulation at the receiver. We then analyze and provide bounds for an alternate scenario where each relay node starts its transmission to the destination as soon as it has decoded the source data, independent of the state of the other relay nodes. This approach further reduces the transmission time, because the transmission by the relay nodes helps the other relay nodes that are still receiving.

We investigate a cooperative communications scheme withNparallel relays, where both the transmissions from the source to the relays and from the relays to the destination use fountain codes. Receivers for fountain codes can accumulate mutual information, while traditional energy collection methods, such as repetition or cooperative space-time codes, only accumulate energy. As a consequence, using fountain codes can reduce the total energy required for transmitting data from the source to the destination. We first analyze the scenario where the source stops transmitting and the relay nodes start transmitting afterLrelay nodes have successfully decoded the source data. We optimizeL, and also give closed-form equations for the energy savings that can be achieved by the use of mutual-information-collection at the receiver instead of traditional energy-collection methods. We then analyze an alternate scenario where each relay node starts its transmission to the destination as soon as it has decoded the source data, and helps the other relay nodes that are still in reception mode. Doing so further reduces the total transmission time and energy consumption.

We describe the architecture for our distributed video coding (DVC) system. Some key differences between our work and previous systems include a new method of enabling decoder motion compensation, and the use of serially concatenated accumulate syndrome codes for distributed source coding. To evaluate performance, we compare our system to the H.263+ and H.264/AVC video codecs. Experiments show that our system is comparable to DVC systems from Stanford and Berkeley in the sense that our system performs better than H.263+Intra, but worse than H.263+Inter and H.264/AVC

Reduced latency versions of iterative decoders of low-density parity-check codes are analyzed in this paper. The proposed schemes converge faster than standard approaches. Two methods, density evolution and EXIT charts, are used to analyze the performance of the proposed algorithms. Both theoretical analysis and simulations show that the new schedules offer good performance versus complexity and latency trade-offs.

Replica shuffled versions of iterative decoders of turbo codes, low-density parity-check codes and turbo product codes are presented. The proposed schemes converge faster than standard and previously proposed "shuffled" approaches. Simulations show that the new schedules offer good performance versus complexity/latency trade-offs.

Important inference problems in statistical physics, computer vision, error-correcting coding theory, and artificial intelligence can all be reformulated as the computation of marginal probabilities on factor graphs. The belief propagation (BP) algorithm is an efficient way to solve these problems that is exact when the factor graph is a tree, but only approximate when the factor graph has cycles. We show that BP fixed points correspond to the stationary points of the Bethe approximation of the free energy for a factor graph. We explain how to obtain region-based free energy approximations that improve the Bethe approximation, and corresponding generalized belief propagation (GBP) algorithms. We emphasize the conditions a free energy approximation must satisfy in order to be a "valid" or "maxent-normal" approximation. We describe the relationship between four different methods that can be used to generate valid approximations: the "Bethe method", the "junction graph method", the "cluster variation method", and the "region graph method". Finally, we explain how to tell whether a region-based approximation, and its corresponding GBP algorithm, is likely to be accurate, and describe empirical results showing that GBP can significantly outperform BP.

This paper studies the performance of two classes of rateless codes (LTand Raptor codes) on noisy channels such as the BSC and the AWGNC. We find that Raptor codes outperform LT codes, and have good performance on a wide variety of noisy channels.

We present sparse factor graph representations of Reed-Solomon codes based on a fast Fourier transform. These representations can be used to create encoders, or message-passing decoders that use soft input information. We discuss various simplifications and transformations of the factor graph representations that may be useful. Finally, we show that other interesting codes can be represented using sparse fast transform factor graphs.

We describe a practical method for distributed compression of q-ary sources using multi-level serially concatenated-accumulate codes. Our approach works well at high compression rates, and allows for graceful and incremental rate-adaptivity. Simulations show that the compression efficiency is near the information-theoretic limits for correlations between sources that obey a Gaussian or Laplacian distribution.

Belief propagation (BP) was only supposed to work for tree-like networks but works surprisingly well in many applications involving networks with loops, including turbo codes. However, there has been little understanding of the algorithm or the nature of the solutions it finds for general graphs. We show that BP can only converge to a stationary point of an approximate free energy, known as the Bethe free energy in statis(cid:173) tical physics. This result characterizes BP fixed-points and makes connections with variational approaches to approximate inference. More importantly, our analysis lets us build on the progress made in statistical physics since Bethe's approximation was introduced in 1935. Kikuchi and others have shown how to construct more ac(cid:173) curate free energy approximations, of which Bethe's approximation is the simplest. Exploiting the insights from our analysis, we de(cid:173) rive generalized belief propagation (GBP) versions ofthese Kikuchi approximations. These new message passing algorithms can be significantly more accurate than ordinary BP, at an adjustable in(cid:173) crease in complexity. We illustrate such a new GBP algorithm on a grid Markov network and show that it gives much more accurate marginal probabilities than those found using ordinary BP.

Construction toys are a superb medium for geometric models. We argue that such toys, suitably instrumented or sensed, could be the inspiration for a new generation of easy-to-use, tangible modeling systems—especially if the tangible modeling is combined with graphical-interpretation techniques for enhancing nascent models automatically. The three key technologies needed to realize this idea are embedded computation, vision-based acquisition, and graphical interpretation. We sample these technologies in the context of two novel modeling systems: physical building blocks that self-describe, interpret, and decorate the structures into which they are assembled; and a system for scanning, interpreting, and animating clay figures.

We describe a tangible interface for building virtual structures using physical building blocks. We demonstrate two applications of our system. In one version, the blocks are used to construct geometric models of objects and structures for a popular game, Quake II™. In another version, buildings created with our blocks are rendered in different styles, using intelligent decoration of the building model.