As a neural network's depth increases, it can improve generalization performance. However, training deep networks is challenging due to gradient and signal propagation issues. To address these challenges, extensive theoretical research and various methods have been introduced. Despite these advances, effective weight initialization methods for tanh neural networks remain insufficiently investigated. This paper presents a novel weight initialization method for neural networks with tanh activation function. Based on an analysis of the fixed points of the function $\tanh(ax)$, the proposed method aims to determine values of $a$ that mitigate activation saturation. A series of experiments on various classification datasets and physics-informed neural networks demonstrates that the proposed method outperforms Xavier initialization methods (with or without normalization) in terms of robustness across different network sizes, data efficiency, and convergence speed. Code is available at https://github.com/1HyunwooLee/Tanh-Init.

Appropriate weight initialization settings, along with the ReLU activation function, have become cornerstones of modern deep learning, enabling the training and deployment of highly effective and efficient neural network models across diverse areas of artificial intelligence. The problem of "dying ReLU," where ReLU neurons become inactive and yield zero output, presents a significant challenge in the training of deep neural networks with ReLU activation function. Theoretical research and various methods have been introduced to address the problem. However, even with these methods and research, training remains challenging for extremely deep and narrow feedforward networks with ReLU activation function. In this paper, we propose a novel weight initialization method to address this issue. We establish several properties of our initial weight matrix and demonstrate how these properties enable the effective propagation of signal vectors. Through a series of experiments and comparisons with existing methods, we demonstrate the effectiveness of the novel initialization method.

Pliable index coding (PICOD) has recently been regarded as a promising solution that exploits the coding advantage to improve communication efficiency of content-type systems (e.g., recommendation system), where clients are pliable and are interested in receiving any new message that they do not have. PICOD aims to find an effective coding strategy that satisfies the demands of all clients with the minimum number of transmissions. However, most of the previous works mainly provided theoretical understanding on PICOD in special instances based on greedy algorithms. In contrast, in this paper, we present a flexible sparse and low-rank matrix modeling approach to minimize the number of transmissions for the general PICOD problems. This is achieved by establishing generalized pliable alignment conditions to guarantee the requirements of all clients. As the resulting non-convex problem is highly intractable, we further develop an alternating pursuit framework to detect the rank of the matrix to be recovered by using the rank-increasing strategy. To address the feasibility-detection issues in the existing methods, we propose an alternating projection algorithm, which admits closed-form expressions and avoids excessive sparsity inducing. Moreover, we establish the global convergence of the alternating projection algorithm with random initial points. Simulation results demonstrate that the proposed alternating pursuit algorithm significantly reduces the number of transmissions compared to the state-of-the-art methods.

Massive access is a critical design challenge of Internet of Things (IoT) networks. In this paper, we consider the grant-free uplink transmission of an IoT network with a multiple-antenna base station (BS) and a large number of single-antenna IoT devices. Taking into account the sporadic nature of IoT devices, we formulate the joint activity detection and channel estimation (JADCE) problem as a group-sparse matrix estimation problem. This problem can be solved by applying the existing compressed sensing techniques, which however either suffer from high computational complexities or lack of algorithm robustness. To this end, we propose a novel algorithm unrolling framework based on the deep neural network to simultaneously achieve low computational complexity and high robustness for solving the JADCE problem. Specifically, we map the original iterative shrinkage thresholding algorithm (ISTA) into an unrolled recurrent neural network (RNN), thereby improving the convergence rate and computational efficiency through end-to-end training. Moreover, the proposed algorithm unrolling approach inherits the structure and domain knowledge of the ISTA, thereby maintaining the algorithm robustness, which can handle non-Gaussian preamble sequence matrix in massive access. With rigorous theoretical analysis, we further simplify the unrolled network structure by reducing the redundant training parameters. Furthermore, we prove that the simplified unrolled deep neural network structures enjoy a linear convergence rate. Extensive simulations based on various preamble signatures show that the proposed unrolled networks outperform the existing methods in terms of the convergence rate, robustness and estimation accuracy.

Linear regression without correspondences is the problem of performing a linear regression fit to a dataset for which the correspondences between the independent samples and the observations are unknown. Such a problem naturally arises in diverse domains such as computer vision, data mining, communications and biology. In its simplest form, it is tantamount to solving a linear system of equations, for which the entries of the right hand side vector have been permuted. This type of data corruption renders the linear regression task considerably harder, even in the absence of other corruptions, such as noise, outliers or missing entries. Existing methods are either applicable only to noiseless data or they are very sensitive to initialization or they work only for partially shuffled data. In this paper we address these issues via an algebraic geometric approach, which uses symmetric polynomials to extract permutation-invariant constraints that the parameters ξ* ∈ Rnof the linear regression model must satisfy. This naturally leads to a polynomial system of n equations in n unknowns, which contains ξ* in its root locus. Using the machinery of algebraic geometry we prove that as long as the independent samples are generic, this polynomial system is always consistent with at most n! complex roots, regardless of any type of corruption inflicted on the observations. The algorithmic implication of this fact is that one can always solve this polynomial system and use its most suitable root as initialization to the Expectation Maximization algorithm. To the best of our knowledge, the resulting method is the first working solution for small values of n able to handle thousands of fully shuffled noisy observations in milliseconds.

Massive MIMO system yields significant improvements in spectral and energy efficiency for future wireless communication systems. The regularized zero-forcing (RZF) beamforming is able to provide good performance with the capability of achieving numerical stability and robustness to the channel uncertainty. However, in massive MIMO systems, the matrix inversion operation in RZF beamforming becomes computationally expensive. To address this computational issue, we shall propose a novel randomized sketching based RZF beamforming approach with low computational latency. This is achieved by solving a linear system via randomized sketching based on the preconditioned Richard iteration, which guarantees high quality approximations to the optimal solution. We theoretically prove that the sequence of approximations obtained iteratively converges to the exact RZF beamforming matrix linearly fast as the number of iterations increases. Also, it turns out that the system sum-rate for such sequence of approximations converges to the exact one at a linear convergence rate. Our simulation results verify our theoretical findings.

Towards low-latency communication for short-packet transmission, this paper tackles the problem of shuffled linear regression for large-scale wireless sensor networks with header-free communication by using results from algebraic geometry as well as an alternating optimization scheme. The shuffled linear regression problem is to solve a linear system with shuffled entries of the right hand side vector. However, solving the shuffled linear system requires high computational cost. The key idea of our approach is to eliminate the shuffled structure via symmetric polynomials, which leads to a system of polynomial equations. Considering one of the solutions of the resulting polynomial system as an initialization to the Expectation Maximization algorithm, we propose the Algebraically-Initialized Expectation Maximization algorithm. Computational experiments with synthetic data show that our proposed algorithm is extensively efficient, and it performs well even with noise.

High-radix topologies in large-scale networks provide low network diameter and high path diversity, but the idle power from high-speed links results in energy inefficiency, especially at low traffic load. In this work, we exploit the high path diversity and non-minimal adaptive routing in high-radix topologies to consolidate traffic to a smaller number of links to enable more network channels to be power-gated. In particular, we propose TCEP (Traffic Consolidation for Energy-Proportional high-radix networks), a distributed, proactive power management mechanism for large-scale networks that achieves energy-proportionality by proactively power-gating network channels through traffic consolidation. Instead of naively power-gating the least utilized link, TCEP differentiates links with the type of traffic (i.e., minimally vs. non-minimally routed traffic) on them since the performance impact of power-gating on minimal traffic is greater than non-minimal traffic. The performance degradation from the reduced number of channels is minimized by concentrating available links to a small number of routers, instead of distributing them across the network, to maximize path diversity. TCEP introduces a shadow link to quickly reactivate an inactive link and Power-Aware progressive Load-balanced (PAL) routing algorithm that incorporates the link power states in load-balancing the network. Our evaluations show that TCEP achieves significantly higher throughput across various traffic patterns while providing comparable energy savings for real workloads, compared to a prior approach proposed for the flattened butterfly topology.

In this paper, a novel similarity measure for comparing massive complex networks based on moment matrices is proposed. We consider the corresponding adjacency matrix of a graph as a real random variable of the algebraic probability space with a state. It is shown that the spectral distribution of the matrix can be expressed as a unique discrete probability measure. Then we use the geodesic distance between positive definite moment matrices for comparing massive networks. It is proved that this distance is graph invariant and sub-structure invariant. Numerical simulations demonstrate that the proposed method outperforms state-of-art method in collaboration network classification and its computational cost is extremely cheap.

We present a linear model with an unknown permutation matrix for header-free communication in massive Internet-of-Things (IoT) networks, thereby supporting low-latency communication. Besides the header-free communication application, the permuted linear model also has many applications in matching and correspondence estimation problems. To solve this permuted linear system, the key idea is to convert it into a system of polynomial equations via power sum symmetric polynomials. In our proposed method, specific assumptions (e.g., low-rankness and sparsity) about the permuted linear model is not needed. The closed form of the solution is established for a special case. Computational results show that the proposed method achieves good performance.