Tanium Reveal is a federated search engine deployed on large-scale enterprise networks that is capable of executing data queries across billions of private data files within 60 seconds. Data resides at the edge of networks, potentially distributed on hundreds of thousands of endpoints. The anatomy of the search engine consists of local inverse indexes on each endpoint and a global communication platform called Tanium for issuing search queries to all endpoints. Reveal enables asynchronous parsing and indexing on endpoints without noticeable impact to the endpoints' primary functionality. The engine harnesses the Tanium platform, which is based on a self-organizing, fault-tolerant, scalable, linear chain communication scheme. We demonstrate a multi-tier workflow for executing search queries across a network and for viewing matching snippets of text on any endpoint. We analyze metrics for federated indexing and searching in multiple environments including a production network with 1.05 billion searchable files distributed across 4236 endpoints. While primarily focusing on Boolean, phrase, and similarity query types, Reveal is compatible with further automation (e.g., semantic classification based on machine learning). Lastly, we discuss safeguards for sensitive information within Reveal including cryptographic hashing of private text and role-based access control (RBAC).

Each memoryless binary-input channel (BIC) can be uniquely described by its Blackwell measure, which is a probability distribution on the unit interval [0, 1] with mean 1/2. Conversely, any such probability distribution defines a BIC. The evolution of the Blackwell measure under Arıkan's polar transform is derived for general BICs, and is analogous to density evolution as cited in the literature. The present analysis emphasizes functional equations. Consequently, the evolution of a variety of channel functionals is characterized, including the symmetric capacity, Bhattacharyya parameter, moments of information density, Hellinger affinity, Gallager's reliability function, the Hirschfeld-Gebelein-Rényi maximal correlation, and the Bayesian information gain. The evolution of measure is specialized for symmetric BICs according to their decomposition into binary symmetric (sub)-channels (BSCs), which simplifies iterative computations and the construction of polar codes. It is verified that, as a consequence of the Blackwell-Sherman-Stein theorem, all channel functionals Ifthat can be expressed as an expectation of a convex function f with respect to the Blackwell measure of a channel polarize in each iteration due to the polar transformation on the class of symmetric BICs. Moreover, for f either convex or non-convex, a necessary and sufficient condition is established to determine whether the random process associated with each Ifis a martingale, submartingale, or supermartingale. Represented via functional inequalities in terms of f, this condition is numerically verifiable for all If, and can generate analytical proofs. To exhibit one such proof, it is shown that the random process associated with the squared maximal correlation parameter is a supermartingale, and converges almost surely on the unit interval [0, 1].

Multilayer switch networks are proposed as artificial generators of high-dimensional discrete data (e.g., binary vectors, categorical data, natural language, network log files, and discrete-valued time series). Unlike deconvolution networks which generate continuous-valued data and which consist of upsampling filters and reverse pooling layers, multilayer switch networks are composed of adaptive switches which model conditional distributions of discrete random variables. An interpretable, statistical framework is introduced for training these nonlinear networks based on a maximum-likelihood objective function. To learn network parameters, stochastic gradient descent is applied to the objective, and is stable until convergence. This direct optimization does not involve back-propagation over separate encoder and decoder networks, or adversarial training of dueling networks. While training remains tractable for moderately sized networks, Markov-chain Monte Carlo (MCMC) approximations of gradients are derived for deep networks which contain latent variables. The statistical framework is evaluated on synthetic data, high-dimensional binary data of handwritten digits, and web-crawled natural language data. Aspects of the model’s framework such as interpretability, computational complexity, and generalization ability are discussed.

Inductive matrix completion (IMC) is a model for incorporating side information in form of “features” of the row and column entities of an unknown matrix in the matrix completion problem. As side information, features can substantially reduce the number of observed entries required for reconstructing an unknown matrix from its given entries. The IMC problem can be formulated as a low-rank matrix recovery problem where the observed entries are seen as measurements of a smaller matrix that models the interaction between the column and row features. We take advantage of this property to study the optimization landscape of the factorized IMC problem. In particular, we show that the critical points of the objective function of this problem are either global minima that correspond to the true solution or are “escapable” saddle points. This result implies that any minimization algorithm with guaranteed convergence to a local minimum can be used for solving the factorized IMC problem.

An infrastructure-less indoor localization system is proposed based on fingerprints of light signals acquired at high frequencies. In contrast to other systems that modulate lights, the proposed system distinguishes lights by learning from training samples. Due to slight differences in the electronic components used in the construction of compact fluorescent light (CFL) and light emitting diode (LED) bulbs, the optical signals emitted by each light bulb have slight differences with other light bulbs even within the same brand and model. Light signals are digitized with a fast and accurate analog-to-digital converter (ADC) at up to 1 mega-samples/second, segmented, and mapped into the frequency domain using the Fast Fourier Transform (FFT). Spectral features based on the FFT are filtered, normalized, and used as training data for supervised machine learning algorithms. Results are provided for two classifiers of varying complexity: (1) A k-Nearest Neighbor (KNN) classifier; (2) A Convolutional Neural Net (CNN) classifier. A hardware system for indoor localization was designed to analyze the performance of the classifiers. Under certain restrictions, results show that light bulbs may be identified with high accuracy without special infrastructure for modulation. Identifying a light bulb is meant to be synonymous with identifying its associated location.

The classical problem in a network coding theory considers communication over multicast networks. Multiple transmitters send independent messages to multiple receivers that decode the same set of messages. In this paper, computation over multicast networks is considered: each receiver decodes an identical function of the original messages. For a countably infinite class of two-transmitter two-receiver single-hop linear deterministic networks, the computation capacity is characterized for a linear function (modulo-2 sum) of Bernoulli sources. A new upper bound is derived that is tighter than cut-set-based and genie-aided bounds. A matching inner bound is established via the development of a network decomposition theorem, which identifies elementary parallel subnetworks that can constitute an original network without loss of optimality. The decomposition theorem provides a conceptually simple proof of achievability that generalizes to L-transmitter L-receiver networks.

Polar codes are introduced for discrete memoryless broadcast channels. For m-user deterministic broadcast channels, polarization is applied to map uniformly random message bits from m-independent messages to one codeword while satisfying broadcast constraints. The polarization-based codes achieve rates on the boundary of the private-message capacity region. For two-user noisy broadcast channels, polar implementations are presented for two information-theoretic schemes: 1) Cover's superposition codes and 2) Marton's codes. Due to the structure of polarization, constraints on the auxiliary and channel-input distributions are identified to ensure proper alignment of polarization indices in the multiuser setting. The codes achieve rates on the capacity boundary of a few classes of broadcast channels (e.g., binary-input stochastically degraded). The complexity of encoding and decoding is O(n log n), where n is the block length. In addition, polar code sequences obtain a stretched-exponential decay of O(2-nβ) of the average block error probability where 0 <; β <; 1/2. Reproducible experiments for finite block lengths n = 512, 1024, 2048 corroborate the theory.

New theorems are established regarding polarized Bernoulli random variables: (i) The maximal correlations between polarized Bernoulli variables converge to zero or one as do the conditional entropy and Bhattacharyya parameters; (ii) The graphical model of polarized Bernoulli variables provides a way to compute pair-wise and higher-order correlations; (iii) The Wyner common information between two sequences of correlated random variables may be extracted using Arikan's polar transform which leads to a low-complexity solution to the Wyner network. In addition, a joint polarization theorem is provided involving common information.

Building on polar code constructions proposed by the authors for deterministic broadcast channels, two theorems are introduced in the present paper for noisy two-user broadcast channels. The theorems establish polar code constructions for two important information-theoretic broadcast strategies: (1) Cover's superposition strategy; (2) Marton's construction. One aspect of the polar code constructions is the alignment of polarization indices via constraints placed on the auxiliary and channel-input distributions. The codes achieve capacity-optimal rates for several classes of broadcast channels (e.g., binary-input stochastically degraded channels). Applying Arıkan's original matrix kernel for polarization, it is shown that the average probability of error in decoding two private messages at the broadcast receivers decays as O(2(-nβ)) where 0 <; β <; 1/2 and n is the code length. The encoding and decoding complexities remain O(n log n). The error analysis is made possible by defining new polar code ensembles for broadcast channels.

We investigate how the network connectivity can affect the degrees of freedom (DoF) of wireless networks. We consider a network of n source-destination (SD) pairs and assume that any two nodes are connected with a positive probability p, independent of other node pairs. We show that, for any arbitrarily small p, a constant DoF is achievable for every SD pair with probability approaching one as n tends to infinity. The achievability is based on the two-hop transmission with decode-and-forward relaying and over each-hop we adopt interference alignment. Considering that an achievable per-user DoF for direct or one-hop transmission can be arbitrarily small as the connectivity probability p decreases, our result shows that, somewhat surprisingly, two-hop transmission is enough to guarantee non-vanishing per-user DoF for any p showing that sparsely connected networks can still provide non-vanishing per-user DoF.

We characterize the capacity region to within log {2(M - 1)} bits/s/Hz for the M-transmitter K-receiver Gaussian multicast channel with feedback where each receiver wishes to decode every message from the M transmitters. Extending Cover-Leung's achievable scheme intended for (M, K) = (2, 1), we show that this generalized scheme achieves the cutset-based outer bound within log {2(M - 1)} bits per transmitter for all channel parameters. In contrast to the capacity in the nonfeedback case, the feedback capacity improves upon the naive intersection of the feedback capacities of K individual multiple access channels. We find that feedback provides unbounded multiplicative gain at high signal-to-noise ratios as was shown in the Gaussian interference channel. To complement the results, we establish the exact feedback capacity of the Avestimehr-Diggavi-Tse deterministic model, from which we make the observation that feedback can also be beneficial for function computation.

Determining the capacity of multi-receiver networks with arbitrary message demands is an open problem in the network coding literature. In this paper, we consider a multi-source, multi-receiver symmetric deterministic network model parameterized by channel coefficients (inspired by wireless network flow) in which the receivers compute a sum of the symbols generated at the sources. Scalar and vector linear coding strategies are analyzed. It is shown that computation alignment over finite field vector spaces is necessary to achieve the computation capacities in the network. To aid in the construction of coding strategies, network equivalence theorems are established for the decomposition of deterministic models into elementary sub-networks. The linear coding capacity for computation is characterized for all channel parameters considered in the model for a countably infinite class of networks. The constructive coding schemes introduced herein for a specific class of networks provide an optimistic viewpoint for the application of structured codes in network communication.

A multiplicative Gaussian wire-tap channel inspired by compressed sensing is studied. Lower and upper bounds on the secrecy capacity are derived, and shown to be relatively tight in the large system limit for a large class of compressed sensing matrices. Surprisingly, it is shown that the secrecy capacity of this channel is nearly equal to the capacity without any secrecy constraint provided that the channel of the eavesdropper is strictly worse than the channel of the intended receiver. In other words, the eavesdropper can see almost everything and yet learn almost nothing. This behavior, which contrasts sharply with that of many commonly studied wiretap channels, is made possible by the fact that a small number of linear projections can make a crucial difference in the ability to estimate sparse vectors.

Polar codes are the first codes to provably achieve capacity on the symmetric binary-input discrete memoryless channel (B-DMC) with low encoding and decoding complexity. The parity check matrix of polar codes is high-density and we show that linear program (LP) decoding fails on the fundamental polytope of the parity check matrix. The recursive structure of the code permits a sparse factor graph representation. We define a new polytope based on the fundamental polytope of the sparse graph representation. This new polytope P is defined in a space of dimension O(N logN) where N is the block length. We prove that the projection of P in the original space is tighter than the fundamental polytope based on the parity check matrix. The LP decoder over P obtains the ML-certificate property. In the case of the binary erasure channel (BEC), the new LP decoder is equivalent to the belief propagation (BP) decoder operating on the sparse factor graph representation, and hence achieves capacity. Simulation results of SC (successive cancellation) decoding, LP decoding over tightened polytopes, and (ML) maximum likelihood decoding are provided. For channels other than the BEC, we discuss why LP decoding over P with a linear objective function is insufficient.

A network in which sensors observe a common Gaussian source is analyzed. Using a fixed linear transform, each sensor compresses its high-dimensional observation into a low-dimensional representation. The latter is provided to a central decoder that reconstructs the source according to a mean squared error (MSE) distortion metric. The distributed Karhunen-Loeve Transform (d-KLT) has been shown to provide a (locally) optimal linear solution for compression at each sensor. While the d-KLT achieves the lowest distortion linear reconstruction known, it does not maintain a nested subspace structure. In the case of ideal links to the decoder, this paper presents transforms that maintain nested subspaces, allowing the decoder to approximate a delay-limited source in an online fashion according to a desired sensor schedule. A distortion envelope for one distributed transform with nested subspace properties (d-nested-KLT) is provided. In the case of i.i.d. noise to the decoder, under assumptions of power allocation over subspaces, it is also possible to achieve nested subspaces utilizing correlations between sensors' observations. Results are applicable for data access over networks, and online information processing in sensor networks.

A linear compressive network (LCN) is defined as a graph of sensors in which each encoding sensor compresses incoming jointly Gaussian random signals and transmits (potentially) low-dimensional linear projections to neighbors over a noisy uncoded channel. Each sensor has a maximum power to allocate over signal subspaces. The networks of focus are acyclic, directed graphs with multiple sources and multiple destinations. LCN pathways lead to decoding leaf nodes that estimate linear functions of the original high dimensional sources by minimizing a mean squared error (MSE) distortion cost function. An iterative optimization of local compressive matrices for all graph nodes is developed using an optimal quadratically constrained quadratic program (QCQP) step. The performance of the optimization is marked by power-compression-distortion spectra, with converse bounds based on cut-set arguments. Examples include single layer and multi-layer (e.g. p-layer tree cascades, butterfly) networks. The LCN is a generalization of the Karhunen-Loeve Transform to noisy multi-layer networks, and extends previous approaches for point-to-point and distributed compression-estimation of Gaussian signals. The framework relates to network coding in the noiseless case, and uncoded transmission in the noisy case.

Many important early vision techniques, such as active contours (ACs), can be formulated as energy minimization. However, finding global optimum (minimum energy) configurations is often computationally intractable. Approximate solutions obtained using iterative numerical methods may be ill-conditioned, and exhibit poor convergence and inaccuracy due to noise and discretization errors. We formulate AC as a statistical estimation problem and solve it using (Gaussian) message passing on factor graphs of linear models. The resulting algorithm exhibits faster convergence and the solutions possess higher numerical stability, robustness and accuracy.

We present a novel genre-independent SVM framework for detecting scene changes in broadcast video. Our framework works on content from a diverse range of genres by allowing sets of features, extracted from both audio and video streams, to be combined and compared automatically without the use of explicit thresholds. For ground truth, we use hand-labeled video scene boundaries from a wide variety of broadcast genres to generate positive and negative samples for the SVM. Our experiments include high-and low-level audio features such as semantic histograms and distances between Gaussian models, as well as video features such as shot cut positions. We evaluate the importance of these measures in a structured frame-work, with performance comparisons obtained via ROC curves. We achieve over 70% detection rate for 10% false positive rate on our corpus of over 7.5 hours of data collected from news, talk shows, sitcoms, dramas, music videos, and how-to shows.