Incomplete multiview clustering is of high recent interest, fueled by the advancement of common information-based deep multiview learning. The practical scenarios where unpaired multiview data with missing values have wide applications in generative learning, cross-modal retrieval, and wireless device identification problems. Following the perspective that the shared information between the incomplete multiview data aligns with the cluster targets, recent works have generalized the well-known common information frameworks in information theory multiview learning problems, with improved performance reported. Different from previous works, we extend the frameworks to incomplete multiview clustering problems and propose an efficient solver: Wyner Incomplete MultiView Clustering (WyIMVC). Interestingly, the common randomness in WyIMVC allows for joint clustering and missing value inference in contrast to the compared methods in the literature. Moreover, leveraging the difference-of-convex structure of the formulated problems, we propose an efficient solver with a convergence guarantee independent of initialization. Empirically, our solver outperforms the state-of-the-art solvers in a range of incomplete multiview datasets with varying numbers of views and dimensions.

In this work, we propose computationally efficient solvers for novel extensions of Wyner common information. By separating information sources into bipartite, the proposed Bipartite common information framework has difference-of-convex structure for efficient non-convex optimization. In known joint distribution cases, our difference-of-convex algorithm(DCA)-based solver has a provable convergence guarantee to local stationary points. As for unknown distribution settings, the insights from DCA combined with the exponential family of distributions for parameterization allows for closed-form expressions for efficient estimation. Furthermore, we show that the Bipartite common information applies to multi-modal clustering without employing ad-hoc clustering algorithms. Empirically, our solvers outperform state-of-the-art methods in clustering accuracy and running time over a range of non-trivial multi-modal clustering datasets with different number of data modalities.

Wireless fingerprinting is a device identification approach which leverages hardware imperfections and wireless channel variations as unique user-centric signatures. Recent studies have also demonstrated that user behavior can be used as a signature by collecting network traffic data, e.g., packet length, without the need to decode/decrypt the payload. Inspired by these results, we propose a multi-layer fingerprinting framework that jointly combines the multi-layer signatures for improved identification performance. In contrast to previous works in the area, our multi-view learning approach is rooted in the common information framework developed by Wyner [1] and is able to exploit data with multiple forms to enable the extraction of the user-centric signatures shared among the multi-layer features without the need for labels (i.e., unsupervised learning setup). We further use variational inference to obtain a computationally efficient algorithm based on a tight surrogate bound on the loss function. Our evaluation framework is based on a dataset obtained by combining real-world video traffic with simulated physical layer characteristics. Finally, our empirical results show that our Wyner Variational Autoencoder significantly outper-forms the state-of-the-art baseline in the unsupervised wireless fingerprinting setting.

In this work, we propose a deep learning approach for jointly compressing and denoising the CSI feedback in massive MIMO systems. We consider a practical scenario where only noisy CSI is available for training and inference. To jointly denoise and compress the CSI feedback for improved reconstruction quality without having access to true CSI, we propose a novel generic loss function based on the Stein's unbiased risk estimator (SURE) for unsupervised denoising, and the evidence lower bound (ELBO) for CSI compression. This is in contrast to most existing supervised denoising methods that either require knowledge of the true CSI or are limited to high SNR regimes. Empirically, we show that the proposed approach improves the reconstruction quality of the state-of-the-art method. Moreover, the proposed approach is independent of the choice of the encoder-decoder architecture and can be easily extended to the existing volume of work on this topic.

In this work, we adopt Wyner common information framework for unsupervised multi-view representation learning. Within this framework, we propose two novel formulations that enable the development of computational efficient solvers based on the alternating minimization principle. The first formulation, referred to as the variational form, enjoys a linearly growing complexity with the number of views and is based on a variational-inference tight surrogate bound coupled with a Lagrangian optimization objective function. The second formulation, i.e., the representational form, is shown to include known results as special cases. Here, we develop a tailored version from the alternating direction method of multipliers (ADMM) algorithm for solving the resulting non-convex optimization problem. In the two cases, the convergence of the proposed solvers is established in certain relevant regimes. Furthermore, our empirical results demonstrate the effectiveness of the proposed methods as compared with the state-of-the-art solvers. In a nutshell, the proposed solvers offer computational efficiency, theoretical convergence guarantees (local minima), scalable complexity with the number of views, and exceptional accuracy as compared with the state-of-the-art techniques. Our focus here is devoted to the discrete case and our results for continuous distributions are reported elsewhere.

In this work, we propose solving the Information Bottleneck (IB) and Privacy Funnel (PF) problems with Douglas-Rachford Splitting methods (DRS). We study a general Markovian information-theoretic Lagrangian that includes IB and PF into a unified framework. We prove the linear convergence of the proposed solvers using the Kurdyka- ojasiewicz inequality. Moreover, our analysis is beyond IB and PF and applies to any convex-weakly convex pair objectives. Based on the results, we develop two types of linearly convergent IB solvers, with one improves the performance of convergence over existing solvers while the other can be independent to the relevance-compression trade-off. Moreover, our results apply to PF, yielding a new class of linearly convergent PF solvers. Empirically, the proposed IB solvers IB obtain solutions that are comparable to the Blahut-Arimoto-based benchmark and is convergent for a wider range of the penalty coefficients than existing solvers. For PF, our non-greedy solvers can characterize the privacy-utility trade-off better than the clustering-based greedy solvers.

In this work, we generalize the information bottleneck (IB) approach to the multi-view learning context. The exponentially growing complexity of the optimal representation motivates the development of two novel formulations with more favorable performance-complexity tradeoffs. The first approach is based on forming a stochastic consensus and is suited for scenarios with significant representation overlap between the different views. The second method, relying on incremental updates, is tailored for the other extreme scenario with minimal representation overlap. In both cases, we extend our earlier work on the alternating directional methods of multiplier (ADMM) solver and establish its convergence and scalability. Empirically, we find that the proposed methods outperform state-of-the-art approaches in multi-view classification problems under a broad range of modelling parameters.

The Information bottleneck (IB) method enables optimizing over the trade-off between compression of data and prediction accuracy of learned representations, and has successfully and robustly been applied to both supervised and unsupervised representation learning problems. However, IB has several limitations. First, the IB problem is hard to optimize. The IB Lagrangian$\mathcal{L}_{IB}: =I(X;Z)-\beta I(Y;Z)$is non-convex and existing solutions guarantee only local convergence. As a result, the obtained solutions depend on initialization. Second, the evaluation of a solution is also a challenging task. Conventionally, it resorts to characterizing the information plane, that is, plotting$I(Y;Z)$versus$I(X;Z)$for all solutions obtained from different initial points. Furthermore, the IB Lagrangian has phase transitions while varying the multiplier$\beta$. At phase transitions, both$I(X;Z)$and$I(Y;Z)$increase abruptly and the rate of convergence becomes significantly slow for existing solutions. Recent works with IB adopt variational surrogate bounds to the IB Lagrangian. Although allowing efficient optimization, how close are these surrogates to the IB Lagrangian is not clear. In this work, we solve the IB Lagrangian using augmented Lagrangian methods. With augmented variables, we show that the IB objective can be solved with the alternating direction method of multipliers (ADMM). Different from prior works, we prove that the proposed algorithm is consistently convergent, regardless of the value of$\beta$. Empirically, our gradient-descent-based method results in information plane points that are comparable to those obtained through the conventional Blahut-Arimoto-based solvers, and is convergent for a wider range of the penalty coefficient than previous ADMM-based solvers.