We investigate the reasons for the performance degradation incurred with batch-independent normalization. We find that the prototypical techniques of layer normalization and instance normalization both induce the appearance of failure modes in the neural network's pre-activations: (i) layer normalization induces a collapse towards channel-wise constant functions; (ii) instance normalization induces a lack of variability in instance statistics, symptomatic of an alteration of the expressivity. To alleviate failure mode (i) without aggravating failure mode (ii), we introduce the technique "Proxy Normalization" that normalizes post-activations using a proxy distribution. When combined with layer normalization or group normalization, this batch-independent normalization emulates batch normalization's behavior and consistently matches or exceeds its performance.

Stochastic Gradient Descent (SGD) has proven to be remarkably effective in optimizing deep neural networks that employ ever-larger numbers of parameters. Yet, improving the efficiency of large-scale optimization remains a vital and highly active area of research. Recent work has shown that deep neural networks can be optimized in randomly-projected subspaces of much smaller dimensionality than their native parameter space. While such training is promising for more efficient and scalable optimization schemes, its practical application is limited by inferior optimization performance. Here, we improve on recent random subspace approaches as follows. We show that keeping the random projection fixed throughout training is detrimental to optimization. We propose re-drawing the random subspace at each step, which yields significantly better performance. We realize further improvements by applying independent projections to different parts of the network, making the approximation more efficient as network dimensionality grows. To implement these experiments, we leverage hardware-accelerated pseudo-random number generation to construct the random projections on-demand at every optimization step, allowing us to distribute the computation of independent random directions across multiple workers with shared random seeds. This yields significant reductions in memory and is up to 10x faster for the workloads in question.

Diffusion MRI is a frequently-used imaging modality that can infer microstructural properties of tissue, down to the scale of microns. For single-compartment models, such as the diffusion tensor (DT), the model interpretation depends on voxels having homogeneous composition. This limitation makes it difficult to measure diffusion parameters for small structures such as the fornix in the brain, because of partial volume. In this work, we use a segmentation from a structural scan to calculate the tissue composition for each diffusion voxel. We model the measured diffusion signal as a linear combination of signals from each of the tissues present in the voxel, and fit parameters on a per-region basis by optimising over all diffusion data simultaneously. We test the proposed method by using diffusion data from the Human Connectome Project (HCP). We downsample the HCP data, and show that our method returns parameter estimates that are closer to the high-resolution ground truths than for classical methods. We show that our method allows accurate estimation of diffusion parameters for regions with partial volume. Finally, we apply the method to compare diffusion in the fornix for adults born extremely preterm and matched controls. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.