The Information Bottleneck (IB) principle offers an information-theoretic framework for analyzing the training process of deep neural networks (DNNs). Its essence lies in tracking the dynamics of two mutual information (MI) values: between the hidden layer output and the DNN input/target. According to the hypothesis put forth by Shwartz-Ziv & Tishby (2017), the training process consists of two distinct phases: fitting and compression. The latter phase is believed to account for the good generalization performance exhibited by DNNs. Due to the challenging nature of estimating MI between high-dimensional random vectors, this hypothesis was only partially verified for NNs of tiny sizes or specific types, such as quantized NNs. In this paper, we introduce a framework for conducting IB analysis of general NNs. Our approach leverages the stochastic NN method proposed by Goldfeld et al. (2019) and incorporates a compression step to overcome the obstacles associated with high dimensionality. In other words, we estimate the MI between the compressed representations of high-dimensional random vectors. The proposed method is supported by both theoretical and practical justifications. Notably, we demonstrate the accuracy of our estimator through synthetic experiments featuring predefined MI values and comparison with MINE (Belghazi et al., 2018). Finally, we perform IB analysis on a close-to-real-scale convolutional DNN, which reveals new features of the MI dynamics.

In modern communication systems, multiple types of error-correcting codes can be utilized for different transmission scenarios. Therefore, the receiver should include the decoder compatible with multiple codes used in the transmission scheme, which results in the increase of the resources required for its implementation. In this paper, we investigate the possibility of training a single syndrome-based DNN decoder to solve the problem of unified decoding. We observe, that the syndrome-based approach allows to extend the unified decoding capabilities to the codes with considerably larger lengths in comparison to the initially described by Wang et al. (2018) method. Through numerical experiments, we show that the model trained for decoding a pair of moderate lengths codes (BCH and CRC-Aided Polar) achieves performance results comparable with classical decoding solutions, while sharing the same architecture and the set of trainable weights. We note, that the unification of a syndrome-based DNN decoder does not lead to large performance degradation, in comparison to the decoder trained on a single code. The approach described in the paper is promising in terms of reducing the hardware resources required to implement the decoder.

We consider a coded compressed sensing approach for the unsourced random access and replace the outer tree code proposed by Amalladinne et al. (2020) with the list recoverable code capable of correcting$t$errors. A finite-length random coding bound for such codes is derived. The numerical experiments in the single-antenna quasi-static Rayleigh fading channel show that transition to list recoverable codes correcting$t$errors improves the performance of the coded compressed sensing scheme by 7–10 dB compared to the tree code-based scheme. We propose two practical constructions of outer codes. The first is a modification of the tree code called$t$-tree code. It utilizes the same code structure, and a key difference is a decoder capable of correcting up to$t$errors. The second is based on the Reed–Solomon codes and Guruswami–Sudan list decoding algorithm. The first scheme provides energy efficiency very close to the random coding bound when the decoding complexity (number of decoding paths) is unbounded. But when we restrict the number of decoding paths with a practical value, the second scheme outperforms the first one. Both schemes improve the performance of a tree code-based scheme for a small and moderate number of active users.

We consider the unsourced random access based on a coded compressed sensing approach. The main idea is to replace the outer tree code proposed by Amalladinne et al. with the code capable of correcting t errors. We derive a finite-length random coding bound for such codes and suggest a practical code construction. We have conducted numerical experiments in the single antenna quasi-static Rayleigh fading MAC. The results show that transition to list-recoverable codes correcting t errors allows performance improvement of coded compressed sensing scheme by 7–10 dB compared to the tree code-based scheme.

We consider a problem of unsourced random access in the quasi-static Rayleigh fading channel. In the previous work, the authors have proposed LDPC code based solutions based on joint and treat interference as noise in combination with successive interference cancellation (TIN-SIC) decoder architectures. The authors showed that TIN-SIC decoding significantly outperforms the joint decoding approach and much simpler from the implementation point of view. In this paper, we continue the analysis of TIN-SIC decoding. We derive a finite length achievability bound for TIN-SIC decoder using random coding and propose a practical polar code based TIN-SIC scheme. The latter's performance becomes significantly better in comparison to LDPC code based solutions and close to the finite length achievability bound.

We discuss the problem of designing channel access architectures for enabling fast, low-latency, grant-free, and uncoordinated uplink for densely packed wireless nodes. Specifically, we study random-access codes, previously introduced for the AWGN MAC, in the practically more relevant case of Rayleigh fading, when channel gains are unknown to the decoder. We propose a random coding achievability bound, which we analyze both non-asymptotically and asymptotically. As a candidate practical solution, we propose an explicit iterative coding scheme. The performance of such a solution is surprisingly close to the finite blocklength bounds. Our main findings are twofold. First, just like in the AWGN MAC, we see that jointly decoding a large number of users leads to a surprising phase transition effect, where, at spectral efficiencies below a critical threshold, a perfect multi-user interference cancellation is possible. Second, while the presence of Rayleigh fading significantly increases the minimal required energy-per-bit, the inherent randomization introduced by the channel makes it much easier to attain the optimal performance via iterative schemes. We hope that a principled definition of the random-access model, together with their information-theoretic analysis, will open the road towards unified benchmarking and performance comparison of various random-access solutions for the 5G/6G.

We discuss the problem of designing channel access architectures for enabling fast, low-latency, grant-free and uncoordinated uplink for densely packed wireless nodes. Specifically, we extend the concept of random-access code introduced at ISIT'2017 by one of the authors to the practically more relevant case of the AWGN multiple-access channel (MAC) subject to Rayleigh fading, unknown to the decoder. We derive bounds on the fundamental limits of random-access coding and propose an alternating belief-propagation scheme as a candidate practical solution. The latter's performance was found to be surprisingly close to the information-theoretic bounds. It is curious, thus, that while fading significantly increases the minimal required energy-per-bit Eb/N0(from about 0-2 dB to about 8-11 dB), it appears that it is much easier to attain the optimal performance over the fading channel with a practical scheme by leveraging the inherent randomization introduced by the channel. Finally, we mention that while a number of candidate solutions (MUSA, SCMA, RSMA, etc.) are being discussed for the 5G, the information-theoretic analysis and benchmarking has not been attempted before (in part due to lack of common random-access model). Our work may be seen as a step towards unifying performance comparisons of these methods.

We investigate the problem of uncoordinated massive random access in the quasi-static asynchronous Rayleigh fading channel. In the previous work [1], the authors assumed a completely synchronous scenario which is impossible in any practical implementation. This paper extends the previous work to the asynchronous case. As energy efficiency is of critical importance for massive machine-type communication (mMTC), our main goal is to minimize the energy-per-bit required to achieve the target probability of error. Another issue required for mMTC is a transmitter simplicity. As in the synchronous case, we focus on grant-free transmission and do not use preambles and other synchronization sequences. We propose a practical implementation of a transmission scheme based on synchronization error estimation and cancellation in the frequency domain. The simulation shows that the proposed transmission scheme's performance is very close to the synchronous case. The only source of E_b/N_0 loss is the need for an additional cyclic prefix that helps to solve the synchronization error cancellation problem in the frequency domain.

Massive machine-type communications (mMTC) is one of the key application scenarios for future 5G networks. In the literature, this problem is known as unsourced random access. We propose a polar code based scheme for the unsourced random access Gaussian channel. This scheme is based on T-fold irregular repetition slotted ALOHA (IRSA). We use polar codes as slot codes and investigate their practical performance in T-user MAC. We compare two possible decoding techniques: joint successive cancellation algorithm and joint iterative algorithm. In order to optimize the codes (choose frozen bits), we propose a specialized and efficient design algorithm. Finally, we investigate the performance of the resulting scheme by means of simulations and conclude that replacing of LDPC codes with polar codes in IRSA scheme leads to a significant performance gain.

We address the problem of uncoordinated massive random-access in the Gaussian multiple access channel (MAC). The performance of low-complexity T-fold irregular repetition slotted ALOHA (IRSA) scheme is investigated and achievability bounds are derived. The main difference of this scheme in comparison to IRSA is as follows: any collisions of order up to T can be resolved with some probability of error introduced by noise. In order to optimize the parameters of the scheme we combine the density evolution method (DE) proposed by G. Liva and a finite length random coding bound for the Gaussian MAC proposed by Y. Polyanskiy. As energy efficiency is of critical importance for massive machine-type communication (mMTC), then our main goal is to minimize the energy-per-bit required to achieve the target packet loss ratio (PLR). We consider two scenarios: (a) the number of active users is fixed; (b) the number of active users is a Poisson random variable.