Parities have become a standard benchmark for evaluating learning algorithms. Recent works show that regular neural networks trained by gradient descent can efficiently learn degree $k$ parities on uniform inputs for constant $k$, but fail to do so when $k$ and $d-k$ grow with $d$ (here $d$ is the ambient dimension). However, the case where $k=d-O_d(1)$, including the degree $d$ parity (the full parity), has remained unsettled. This paper shows that for gradient descent on regular neural networks, learnability depends on the initial weight distribution. On one hand, the discrete Rademacher initialization enables efficient learning of almost-full parities, while on the other hand, its Gaussian perturbation with large enough constant standard deviation $\sigma$ prevents it. The positive result for almost-full parities is shown to hold up to $\sigma=O(d^{-1})$, pointing to questions about a sharper threshold phenomenon. Unlike statistical query (SQ) learning, where a singleton function class like the full parity is trivially learnable, our negative result applies to a fixed function and relies on an initial gradient alignment}measure of potential broader relevance to neural networks learning.

This paper introduces a quantitative generalization of the “more capable” comparison of broadcast channels, which is termed “more capable with advantage”. Some basic properties are demonstrated (including tensorization on product channels), and a characterisation is given for the cases of Binary Symmetric Channel (BSC) and Binary Erasure Channel (BEC). It is then applied to two problems. First, a list decoding bound on the BSC is given that applies to transitive codes that achieve capacity on the BEC. Second, new lower bounds on entropy rates of binary hidden Markov processes are derived.