Two key tasks in high-dimensional regularized regression are tuning the regularization strength for accurate predictions and estimating the out-of-sample risk. It is known that the standard approach — $k$-fold cross-validation — is inconsistent in modern high-dimensional settings. While leave-one-out and generalized cross-validation remain consistent in some high-dimensional cases, they become inconsistent when samples are dependent or contain heavy-tailed covariates. As a first step towards modeling structured sample dependence and heavy tails, we use right-rotationally invariant covariate distributions — a crucial concept from compressed sensing. In the proportional asymptotics regime where the number of features and samples grow comparably, which is known to better reflect the empirical behavior in moderately sized datasets, we introduce a new framework, ROTI-GCV, for reliably performing cross-validation under these challenging conditions. Along the way, we propose new estimators for the signal-to-noise ratio and noise variance. We conduct experiments that demonstrate the accuracy of our approach in a variety of synthetic and semi-synthetic settings.

The existence of “lottery tickets” (Frankle & Carbin, 2018) at or near initialization raises the tantalizing question of whether large models are necessary in deep learning, or whether sparse networks can be quickly identified and trained without ever training the dense models that contain them. However, efforts to find these sparse subnetworks without training the dense model (“pruning at initialization”) have been broadly unsuccessful (Frankle et al., 2020b). We put forward a theoretical explanation for this, based on the model’s effective parameter count, $p_\text{eff}$, given by the sum of the number of non-zero weights in the final network and the mutual information between the sparsity mask and the data. We show the Law of Robustness of (Bubeck & Sellke, 2023) extends to sparse networks with the usual parameter count replaced by $p_\text{eff}$, meaning a sparse neural network which robustly interpolates noisy data requires a heavily data-dependent mask. We posit that pruning during and after training outputs masks with higher mutual information than those produced by pruning at initialization. Thus two networks may have the same sparsities, but differ in effective parameter count based on how they were trained. This suggests that pruning near initialization may be infeasible and explains why lottery tickets exist, but cannot be found fast (i.e. without training the full network). Experiments on neural networks confirm that information gained during training may indeed affect model capacity.

The controversy and argument on the usefulness of the physical layer (PHY) academic research for wireless communications are long-standing since the cellular communication paradigm gets to its maturity. In particular, researchers suspect that the performance improvement in cellular communications is primarily attributable to the increases in telecommunication infrastructure and radio spectrum instead of the PHY academic research, whereas concrete evidence is lacking. To respond to this controversy from an objective perspective, we employ econometric approaches to quantify the contributions of the PHY academic research and other performance determinants. Through empirical analysis and the quantitative evidence obtained, albeit preliminary, we shed light on the following issues: 1) what determines the cross-national differences in cellular network performance; 2) to what extent the PHY academic research and other factors affect cellular network performance; 3) what suggestions we can obtain from the data analysis for the stakeholders of the PHY research.

In this paper, a forward decoding procedure is developed for the compress-and-forward (CF) relaying scheme. This procedure uses a layered framework and is based on exploiting a feature of the N -to-1 mapping inherent in the underlying Wyner-Ziv binning. It is shown that exploiting this mapping enables the relaxation of the constraint on the rate of the relay codewords representing the bin indices. For the cooperative multimessage network, the proposed procedure achieves the same rate region as the short-message noisy network coding (SNNC) scheme. However, this procedure is more advantageous for other networks including the two networks presented herein. The first network is a relay chain one with two destinations, whereas the second network is a partially cooperative multimessage one with three destinations. In both networks, side information is available to a subset of the decoding nodes, but not to the rest of the nodes, and in both cases, the network benefits from the relaxation of the rate of the CF bin indices. This relaxation results in rate regions larger than those achieved by the conventional CF and SNNC.

We consider a three-node quasi-static communication system with a full-duplex relay. The goal is to determine the relaying mode that enables rate-efficient communication under given channel conditions. To achieve this goal, we consider a generalized scheme that subsumes the decode-and-forward (DF) and compress-and-forward (CF) schemes as special cases. The generalized scheme is considered when the source and relay signals are synthesized from commonly-used Gaussian codebooks, which are shown to be capacity achieving in two asymptotic cases: perfect relay-destination link and broken source-destination link. Studying the generalized DF-CF scheme, it is shown that, for two non-asymptotic cases in which the signal-to-noise ratios (SNRs) of the links satisfy certain conditions, this scheme reduces to either DF or CF. For another set of non-asymptotic SNRs, the generalized scheme is shown to yield strictly higher rates than both DF and CF. Despite the complexity of the generalized scheme, its rate advantage over DF and CF is shown to be upper bounded by 0.5 bits per channel use. This indicates that the practical benefit of the analysis of this scheme is to enable selecting the relaying mode that suits a given channel realization. Numerical results show that, under Rayleigh fading conditions, this selection yields significant gains over fixed DF and CF.

In this paper, the generalization of the decode-and-forward (DF) and compress-and-forward (CF) relaying schemes is studied for the case in which Gaussian codebooks are used for signalling over scalar Gaussian memoryless channels. Three SNR regions are identified wherein the generalized DF-CF scheme reduces to either the DF or the CF scheme. In addition, it is shown that there is an SNR region in which the generalized DF-CF scheme can be more advantageous than both schemes.