Deep InfoMax (DIM) is a well-established method for self-supervised representation learning (SSRL) based on maximization of the mutual information between the input and the output of a deep neural network encoder. Despite the DIM and contrastive SSRL in general being well-explored, the task of learning representations conforming to a specific distribution (i.e., distribution matching, DM) is still under-addressed. Motivated by the importance of DM to several downstream tasks (including generative modeling, disentanglement, outliers detection and other), we enhance DIM to enable automatic matching of learned representations to a selected prior distribution. To achieve this, we propose injecting an independent noise into the normalized outputs of the encoder, while keeping the same InfoMax training objective. We show that such modification allows for learning uniformly and normally distributed representations, as well as representations of other absolutely continuous distributions. Our approach is tested on various downstream tasks. The results indicate a moderate trade-off between the performance on the downstream tasks and quality of DM.

The Information Bottleneck (IB) principle offers an information-theoretic framework for analyzing the training process of deep neural networks (DNNs). Its essence lies in tracking the dynamics of two mutual information (MI) values: between the hidden layer output and the DNN input/target. According to the hypothesis put forth by Shwartz-Ziv & Tishby (2017), the training process consists of two distinct phases: fitting and compression. The latter phase is believed to account for the good generalization performance exhibited by DNNs. Due to the challenging nature of estimating MI between high-dimensional random vectors, this hypothesis was only partially verified for NNs of tiny sizes or specific types, such as quantized NNs. In this paper, we introduce a framework for conducting IB analysis of general NNs. Our approach leverages the stochastic NN method proposed by Goldfeld et al. (2019) and incorporates a compression step to overcome the obstacles associated with high dimensionality. In other words, we estimate the MI between the compressed representations of high-dimensional random vectors. The proposed method is supported by both theoretical and practical justifications. Notably, we demonstrate the accuracy of our estimator through synthetic experiments featuring predefined MI values and comparison with MINE (Belghazi et al., 2018). Finally, we perform IB analysis on a close-to-real-scale convolutional DNN, which reveals new features of the MI dynamics.

We propose a novel approach to the problem of mutual information (MI) estimation via introducing a family of estimators based on normalizing flows. The estimator maps original data to the target distribution, for which MI is easier to estimate. We additionally explore the target distributions with known closed-form expressions for MI. Theoretical guarantees are provided to demonstrate that our approach yields MI estimates for the original data. Experiments with high-dimensional data are conducted to highlight the practical advantages of the proposed method.

In modern communication systems, multiple types of error-correcting codes can be utilized for different transmission scenarios. Therefore, the receiver should include the decoder compatible with multiple codes used in the transmission scheme, which results in the increase of the resources required for its implementation. In this paper, we investigate the possibility of training a single syndrome-based DNN decoder to solve the problem of unified decoding. We observe, that the syndrome-based approach allows to extend the unified decoding capabilities to the codes with considerably larger lengths in comparison to the initially described by Wang et al. (2018) method. Through numerical experiments, we show that the model trained for decoding a pair of moderate lengths codes (BCH and CRC-Aided Polar) achieves performance results comparable with classical decoding solutions, while sharing the same architecture and the set of trainable weights. We note, that the unification of a syndrome-based DNN decoder does not lead to large performance degradation, in comparison to the decoder trained on a single code. The approach described in the paper is promising in terms of reducing the hardware resources required to implement the decoder.

We consider a coded compressed sensing approach for the unsourced random access and replace the outer tree code proposed by Amalladinne et al. (2020) with the list recoverable code capable of correcting$t$errors. A finite-length random coding bound for such codes is derived. The numerical experiments in the single-antenna quasi-static Rayleigh fading channel show that transition to list recoverable codes correcting$t$errors improves the performance of the coded compressed sensing scheme by 7–10 dB compared to the tree code-based scheme. We propose two practical constructions of outer codes. The first is a modification of the tree code called$t$-tree code. It utilizes the same code structure, and a key difference is a decoder capable of correcting up to$t$errors. The second is based on the Reed–Solomon codes and Guruswami–Sudan list decoding algorithm. The first scheme provides energy efficiency very close to the random coding bound when the decoding complexity (number of decoding paths) is unbounded. But when we restrict the number of decoding paths with a practical value, the second scheme outperforms the first one. Both schemes improve the performance of a tree code-based scheme for a small and moderate number of active users.

We consider the unsourced random access based on a coded compressed sensing approach. The main idea is to replace the outer tree code proposed by Amalladinne et al. with the code capable of correcting t errors. We derive a finite-length random coding bound for such codes and suggest a practical code construction. We have conducted numerical experiments in the single antenna quasi-static Rayleigh fading MAC. The results show that transition to list-recoverable codes correcting t errors allows performance improvement of coded compressed sensing scheme by 7–10 dB compared to the tree code-based scheme.

A distributed storage system must support efficient access to stored data while ensuring recovery of temporally unavailable nodes. Another important aspect of a distributed storage system is security. In this paper, we bring these features together and investigate the problem of efficient access to stored data in presence of a passive eavesdropper with access to limited number of nodes. The access efficiency is measured in two different terms, namely, the number of accessed nodes and the volume of generated network traffic. These quantities possess a natural connection to locality and repair bandwidth in distributed storage system. For each of them we derive bounds on parameters and provide explicit constructions based on maximum distance separable codes. Motivated by practical perspectives we propose the techniques to ensure the same workload on each node as well as constructions over small fields based on subfield subcodes, Euclidean geometry codes and Reed-Muller codes. Finally, we derive an asymptotic random coding bound on parameters of a secure distributed storage system and propose further research directions.

A distributed storage system (DSS) needs to be efficiently accessible and repairable. Recently, considerable effort has been made towards the latter, while the former is usually not considered, since a trivial solution exists in the form of systematic encoding. However, this is not a viable option when considering storage that has to be secure against eavesdroppers. This work investigates the problem of efficient access to data stored on a DSS under such security constraints. Further, we establish methods to balance the access load, i.e., ensure that each node is accessed equally often. We establish the capacity for the alphabet independent case and give an explicit code construction. For the alphabet-dependent case we give existence results based on a random coding argument.

Flanking traditional metrics such as throughput and reliability, age of information (AoI) is emerging as a fundamental tool to capture the performance of IoT systems. In this context, we focus on a setup in which a large number of nodes attempt delivery of time-stamped updates to a common destination over a shared channel, and investigate the ability of different grant-free access strategies to maintain fresh information at the receiver. Specifically, we derive for the first time an exact closed-form expression of the average AoI achieved by irregular repetition slotted ALOHA (IRSA), and compare its performance to that of a slotted ALOHA approach. Our analysis reveals the potential of modern random access schemes, and pinpoints some fundamental trade-offs, providing useful hints for proper system design.

We consider a problem of unsourced random access in the quasi-static Rayleigh fading channel. In the previous work, the authors have proposed LDPC code based solutions based on joint and treat interference as noise in combination with successive interference cancellation (TIN-SIC) decoder architectures. The authors showed that TIN-SIC decoding significantly outperforms the joint decoding approach and much simpler from the implementation point of view. In this paper, we continue the analysis of TIN-SIC decoding. We derive a finite length achievability bound for TIN-SIC decoder using random coding and propose a practical polar code based TIN-SIC scheme. The latter's performance becomes significantly better in comparison to LDPC code based solutions and close to the finite length achievability bound.

We discuss the problem of designing channel access architectures for enabling fast, low-latency, grant-free, and uncoordinated uplink for densely packed wireless nodes. Specifically, we study random-access codes, previously introduced for the AWGN MAC, in the practically more relevant case of Rayleigh fading, when channel gains are unknown to the decoder. We propose a random coding achievability bound, which we analyze both non-asymptotically and asymptotically. As a candidate practical solution, we propose an explicit iterative coding scheme. The performance of such a solution is surprisingly close to the finite blocklength bounds. Our main findings are twofold. First, just like in the AWGN MAC, we see that jointly decoding a large number of users leads to a surprising phase transition effect, where, at spectral efficiencies below a critical threshold, a perfect multi-user interference cancellation is possible. Second, while the presence of Rayleigh fading significantly increases the minimal required energy-per-bit, the inherent randomization introduced by the channel makes it much easier to attain the optimal performance via iterative schemes. We hope that a principled definition of the random-access model, together with their information-theoretic analysis, will open the road towards unified benchmarking and performance comparison of various random-access solutions for the 5G/6G.

We discuss the problem of designing channel access architectures for enabling fast, low-latency, grant-free and uncoordinated uplink for densely packed wireless nodes. Specifically, we extend the concept of random-access code introduced at ISIT'2017 by one of the authors to the practically more relevant case of the AWGN multiple-access channel (MAC) subject to Rayleigh fading, unknown to the decoder. We derive bounds on the fundamental limits of random-access coding and propose an alternating belief-propagation scheme as a candidate practical solution. The latter's performance was found to be surprisingly close to the information-theoretic bounds. It is curious, thus, that while fading significantly increases the minimal required energy-per-bit Eb/N0(from about 0-2 dB to about 8-11 dB), it appears that it is much easier to attain the optimal performance over the fading channel with a practical scheme by leveraging the inherent randomization introduced by the channel. Finally, we mention that while a number of candidate solutions (MUSA, SCMA, RSMA, etc.) are being discussed for the 5G, the information-theoretic analysis and benchmarking has not been attempted before (in part due to lack of common random-access model). Our work may be seen as a step towards unifying performance comparisons of these methods.

We investigate the problem of uncoordinated massive random access in the quasi-static asynchronous Rayleigh fading channel. In the previous work [1], the authors assumed a completely synchronous scenario which is impossible in any practical implementation. This paper extends the previous work to the asynchronous case. As energy efficiency is of critical importance for massive machine-type communication (mMTC), our main goal is to minimize the energy-per-bit required to achieve the target probability of error. Another issue required for mMTC is a transmitter simplicity. As in the synchronous case, we focus on grant-free transmission and do not use preambles and other synchronization sequences. We propose a practical implementation of a transmission scheme based on synchronization error estimation and cancellation in the frequency domain. The simulation shows that the proposed transmission scheme's performance is very close to the synchronous case. The only source of E_b/N_0 loss is the need for an additional cyclic prefix that helps to solve the synchronization error cancellation problem in the frequency domain.

In this paper, we extend the notion of locally recoverable codes with availability to secret sharing schemes. The main problem that we considered is how to store information using locally recoverable codes with all symbol locality and availability in such way that useful information can be recovered using an only small subset of coordinates while a user who observes less than a certain number of coordinates does not get any information. In other words, we have to protect locally recoverable codes with availability over passive eavesdropper that can observe only limited number of coordinates. Upper bounds on number of bits that can be securely stored in such systems together with explicit constructions of codes with such a property are proposed.

Massive machine-type communications (mMTC) is one of the key application scenarios for future 5G networks. In the literature, this problem is known as unsourced random access. We propose a polar code based scheme for the unsourced random access Gaussian channel. This scheme is based on T-fold irregular repetition slotted ALOHA (IRSA). We use polar codes as slot codes and investigate their practical performance in T-user MAC. We compare two possible decoding techniques: joint successive cancellation algorithm and joint iterative algorithm. In order to optimize the codes (choose frozen bits), we propose a specialized and efficient design algorithm. Finally, we investigate the performance of the resulting scheme by means of simulations and conclude that replacing of LDPC codes with polar codes in IRSA scheme leads to a significant performance gain.

In this paper, a low complexity scheme for unsourced random multiple access over the Gaussian channel is proposed. Following the literature by the word "unsourced" we mean the fact that the users use the same codebook codes. The proposed scheme is based on T-fold ALOHA with successive interference cancellation (SIC) procedure. In each slot, the same codebook concatenated code construction with outer non-binary (NB) low- density parity-check (LDPC) code and inner linear binary code is decoded with iterative joint decoding algorithm. Outer NB-LDPC code is decoded with low-complexity iterative q-ary sum-product algorithm (QSPA) and short inner binary code is decoded with maximum likelihood. Finally, the numerical results and comparison with theoretical bounds are represented.

We address the problem of uncoordinated massive random-access in the Gaussian multiple access channel (MAC). The performance of low-complexity T-fold irregular repetition slotted ALOHA (IRSA) scheme is investigated and achievability bounds are derived. The main difference of this scheme in comparison to IRSA is as follows: any collisions of order up to T can be resolved with some probability of error introduced by noise. In order to optimize the parameters of the scheme we combine the density evolution method (DE) proposed by G. Liva and a finite length random coding bound for the Gaussian MAC proposed by Y. Polyanskiy. As energy efficiency is of critical importance for massive machine-type communication (mMTC), then our main goal is to minimize the energy-per-bit required to achieve the target packet loss ratio (PLR). We consider two scenarios: (a) the number of active users is fixed; (b) the number of active users is a Poisson random variable.

We investigate the distance properties of linear locally recoverable codes (LRC codes) with all-symbol locality and availability. New upper and lower bounds on the minimum distance of such codes are derived. The upper bound is based on the shortening method and generalized Hamming weights that are fundamental parameters of any linear codes with many useful applications. This bound improves existing upper bounds. To reduce the gap in between upper and lower bounds, we do not restrict the alphabet size and propose explicit constructions of codes with locality and availability via rank-metric codes. The first construction relies on expander graphs and is better in low rate region. The second construction utilizes the LRC codes developed by Wang et al. as inner codes and is better in high rate region. We also suggest one possible generalization of LRC codes in which the recovering sets can intersect in a small number of coordinates. This feature allows us to increase the achievable code rate and still meet load balancing requirements. We derive upper and lower bounds on the parameters of such codes and present explicit constructions of codes with such a property.

We continue our investigation of one possible generalization of locally recoverable codes (LRC) with all-symbol locality and availability when recovering sets can intersect in a small number of coordinates. This feature allows us to increase the achievable code rate and still meet load balancing requirements. In this paper we derive upper and lower bounds on the minimum distance of such codes. The upper bound is based on generalized Hamming weights (GHWs) that are fundamental parameters of any linear codes with many useful applications. In order to derive a lower bound we propose an explicit construction of (r, t, x), -LRC via rank-metric codes and previously developed high rate (r, t, x) -LRC codes.

In this paper we consider the low complexity majority-logic decoding algorithm for irregular non-binary low-density parity-check (LDPC) codes. The decoding algorithm is a generalization of the bit-flipping algorithm for binary LDPC codes. The lower estimate on the decoding radius realized by this algorithm is derived for the first time for irregular non-binary LDPC codes. We present the numerical results for the derived lower bound.

In this paper, we proposed a discrete sorting optimization approach for uplink channel of Massive Multiple Input, Multiple Output (MIMO) system. The algorithm includes users (UEs) sorting before QR decomposition (QRD) and sorting-reduced (SR) K-best detector for 48x64 MIMO uncoded systems. Simulation results show that detector losses is about 1dB to a Maximum Likelihood (ML) detection in low detector complexity.The UEs sorting is required to sort diagonal elements of the R matrix in ascending order to avoid error propagation in multi-user (MU) scenario. Fast and low complexity method of online discrete optimization is used to find the loss function minimum. Sorting tracking is proposed, so that the pre-sorted interpolated R matrix is used for further sorting optimization, resulting in low sorting complexity. The proposed sorting demonstrates huge performance gain compare to a common power-based one. Simulation results in 5G QuaDRiGa channel are presented.SR-K-best detector is a variant of K-best detector. The SR-K-best with (K,S,p) parameters results in significant losses in scenarios with high correlated users, therefore we proposed a new structure (K,S,p,v,q) of the SR-K-best algorithm and a new discrete optimization method to increase performance. Discrete stochastic optimization was done offline in QuaDRiGa channel to find optimal (K,S,p,v,q) parameters for fixed detector structure.

We investigate the distance properties of linear locally recoverable codes (LRC codes) with all-symbol locality and availability. New upper and lower bounds on the minimum distance of such codes are derived. The upper bound is based on the shortening method and improves existing shortening bounds. To reduce the gap in between upper and lower bounds we do not restrict the alphabet size and propose explicit constructions of codes with locality and availability via rank-metric codes. The first construction relies on expander graphs and is better in low rate region, the second construction utilizes LRC codes developed by Wang et al. as inner codes and better in high rate region.

We investigate one possible generalization of locally recoverable codes (LRC) with all-symbol locality and availability when recovering sets can intersect in a small number of coordinates. This feature allows us to increase the achievable code rate and still meet load balancing requirements. In this paper we derive an upper bound for the rate of such codes and give explicit constructions of codes with such a property. These constructions utilize LRC codes developed by Wang et al.

A locally recoverable code (LRC code) is a code over a finite alphabet, such that every symbol in the encoding is a function of a small number of other symbols that form a recovering set. In this paper, we derive new finite-length and asymptotic bounds on the parameters of LRC codes. For LRC codes with a single recovering set for every coordinate, we derive an asymptotic Gilbert-Varshamov type bound for LRC codes and find the maximum attainable relative distance of asymptotically good LRC codes. Similar results are established for LRC codes with two disjoint recovering sets for every coordinate. For the case of multiple recovering sets (the availability problem), we derive a lower bound on the parameters using expander graph arguments. Finally, we also derive finite-length upper bounds on the rate and the distance of LRC codes with multiple recovering sets.

In [1] a syndrome counting based upper bound on the minimum distance of regular binary LDPC codes is given. In this paper we extend the bound to the case of irregular and generalized LDPC codes over GF(q). The comparison to the lower bound for LDPC codes over GF(q) and to the upper bound for non-binary codes is done. The new bound is shown to lie under the Gilbert-Varshamov bound at high rates.

We consider the decoding of LDPC codes over GF(q) with the low-complexity majority algorithm from [1]. A modification of this algorithm with multiple thresholds is suggested. A lower estimate on the decoding radius realized by the new algorithm is derived. The estimate is shown to be better than the estimate for a single threshold majority decoder. At the same time the transition to multiple thresholds does not affect the order of complexity.

The paper addresses the problem of constructing an asynchronous multiple access system for a multi-user Q-frequency channel with additive white Gaussian noise (AWGN). We propose a coding scheme for the channel which allows a large number of users to work simultaneously in the system. The scheme combines the ideas of Frequency Shift Keying (FSK) and Frequency Hopping Spread Spectrum (FHSS). The major component of the scheme is a non-binary low-density parity-check (LDPC) code. The efficiency of the resulting multiple-access system is shown by simulations.

We address the problem of increasing the sum rate in a multiple-access system from [1] for small number of users. We suggest an improved signal-code construction in which in case of a small number of users we give more resources to them. For the resulting multiple-access system a lower bound on the relative sum rate is derived. It is shown to be very close to the maximal value of relative sum rate in [1] even for small number of users. The bound is obtained for the case of decoding by exhaustive search. We also suggest reduced-complexity decoding and compare the maximal number of users in this case and in case of decoding by exhaustive search.

The minimum distance of expander codes over GF(q) is studied. A new upper bound on the minimum distance of expander codes is derived. The bound is shown to lie under the Varshamov-Gilbert (VG) bound while q ≥ 32. Lower bounds on the minimum distance of some families of expander codes are obtained. A lower bound on the minimum distance of low-density parity-check (LDPC) codes with a Reed-Solomon constituent code over GF(q) is obtained. The bound is shown to be very close to the VG bound and to lie above the upper bound for expander codes.