Since sparse neural networks usually contain many zero weights, these unnecessary network connections can potentially be eliminated without degrading network performance. Therefore, well-designed sparse neural networks have the potential to significantly reduce the number of floating-point operations (FLOPs) and computational resources. In this work, we propose a new automatic pruning method-sparse connectivity learning (SCL). Specifically, a weight is reparameterized as an elementwise multiplication of a trainable weight variable and a binary mask. Thus, network connectivity is fully described by the binary mask, which is modulated by a unit step function. We theoretically prove the fundamental principle of using a straight-through estimator (STE) for network pruning. This principle is that the proxy gradients of STE should be positive, ensuring that mask variables converge at their minima. After finding Leaky ReLU, Softplus, and identity STEs can satisfy this principle, we propose to adopt identity STE in SCL for discrete mask relaxation. We find that mask gradients of different features are very unbalanced; hence, we propose to normalize mask gradients of each feature to optimize mask variable training. In order to automatically train sparse masks, we include the total number of network connections as a regularization term in our objective function. As SCL does not require pruning criteria or hyperparameters defined by designers for network layers, the network is explored in a larger hypothesis space to achieve optimized sparse connectivity for the best performance. SCL overcomes the limitations of existing automatic pruning methods. Experimental results demonstrate that SCL can automatically learn and select important network connections for various baseline network structures. Deep learning models trained by SCL outperform the state-of-the-art human-designed and automatic pruning methods in sparsity, accuracy, and FLOPs reduction.

Consider a memoryless degraded broadcast channel (DBC) in which the channel output is a single-letter function of the channel input and the channel noise. As examples, for the Gaussian broadcast channel (BC), this single-letter function is real scalar addition and for the binary-symmetric BC, this single-letter function is modulo-two addition. This paper identifies several classes of discrete memoryless DBCs for which a relatively simple encoding scheme, which we call natural encoding, achieves capacity. Natural encoding (NE) combines symbols from independent codebooks (one for each receiver) using the same single-letter function that adds distortion to the channel. The alphabet size of each NE codebook is bounded by that of the channel input. This paper also defines the input-symmetric DBC, introduces permutation encoding for the input-symmetric DBC, and proves its optimality. Because it is a special case of permutation encoding, NE is capacity achieving for the two-receiver group-operation DBC. Combining the broadcast Z channel and group-operation DBC results yields a proof that NE is also optimal for the discrete multiplication DBC. Along the way, the paper also provides explicit parametric expressions for the two-receiver binary-symmetric DBC and broadcast Z channel.

Certain degraded broadcast channels (DBCs) have the property that the boundary of the capacity region can be achieved by an encoder that combines independent codebooks (one for each receiver) using the same single-letter function that adds distortion to the channel. We call this the natural encoder for the DBC. Natural encoders are known to achieve the capacity region boundary of the broadcast Gaussian channel, and the broadcast binary-symmetric channel. Recently, they have also been shown to achieve the capacity region of the broadcast Z channel. This paper shows that natural encoding achieves the capacity region boundary for discrete multiplicative DBCs. The optimality of the natural encoder also leads to a relatively simple expression for the capacity region for discrete multiplicative DBCs.

This paper presents serially concatenated trellis coded modulations (SCTCMs) that perform consistently close to the available mutual information for periodic erasure channel (PEC), periodic fading channel (PFC) and the 2 times 2 compound matrix channel. We use both the maximum-likelihood decoding criteria and iterative decoding criteria to design universal SCTCMs for the PEC and the PFC. For the space-time channel, by demultiplexing the symbols across the antennas, the proposed universal SCTCMs for the period-2 PFC deliver consistent performance over the eigenvalue skew of the matrix channel. Within the family of channels having the same eigenvalue skew, a time-varying linear transformation (TVLT) is used to mitigate the performance variation over different eigenvectors. The proposed space-time SCTCMs of 1.0, 2.0 and 3.0 bits per transmission require excess mutual information in the ranges 0.11-0.15, 0.23- 0.26 and 0.35-0.53 bits per antenna, respectively. Because of their consistent performance over all channels, the proposed codes will have good frame-error-rate (FER) performance over any quasi-static fading distribution. In particular, the codes provide competitive FER performance in quasi-static Rayleigh fading.

This paper provides an explicit expression for the capacity region of the two-user broadcast Z channel and proves that the optimal boundary can be achieved by independent encoding of each user. Specifically, the information messages corresponding to each user are encoded independently and the OR of these two encoded streams is transmitted. Nonlinear turbo codes that provide a controlled distribution of ones and zeros are used to demonstrate a low-complexity scheme that operates close to the optimal boundary.

In this paper, we propose serially concatenated trellis coded modulations (SCTCMs) that perform consistently close to the available mutual information for the 2-by-2 compound matrix channel. The proposed SCTCMs use universal SCTCMs for the period-2 periodic fading channel in order to deliver consistent performance over eigenvalue skew. Within the family of channels having the same eigenvalue skew, a time-varying linear transformation (TVLT) is used to mitigate the performance variation over different eigenvectors. The proposed SCTCMs of 1, 2 and 3 bits per transmission require excess mutual information in the ranges 0.11- 0.15, 0.23-0.26 and 0.35-0.53 bits per antenna, respectively. Because of their consistent performance over all channels, the proposed codes will have good frame-error-rate (FER) performance over any quasi- static fading distribution. In particular, the codes provide competitive FER performance in quasi-static Rayleigh fading.