We investigate the phenomenon of grokking -- delayed generalization accompanied by non-monotonic test loss behavior -- in a simple binary logistic classification task, for which "memorizing" and "generalizing" solutions can be strictly defined. Surprisingly, we find that grokking arises naturally even in this minimal model when the parameters of the problem are close to a critical point, and provide both empirical and analytical insights into its mechanism. Concretely, by appealing to the implicit bias of gradient descent, we show that logistic regression can exhibit grokking when the training dataset is nearly linearly separable from the origin and there is strong noise in the perpendicular directions. The underlying reason is that near the critical point, "flat" directions in the loss landscape with nearly zero gradient cause training dynamics to linger for arbitrarily long times near quasi-stable solutions before eventually reaching the global minimum. Finally, we highlight similarities between our findings and the recent literature, strengthening the conjecture that grokking generally occurs in proximity to the interpolation threshold, reminiscent of critical phenomena often observed in physical systems.

Grokking is the intriguing phenomenon where a model learns to generalize long after it has fit the training data. We show both analytically and numerically that grokking can surprisingly occur in linear networks performing linear tasks in a simple teacher-student setup. In this setting, the full training dynamics is derived in terms of the expected training and generalization data covariance matrix. We present exact predictions on how the grokking time depends on input and output dimensionality, train sample size, regularization, and network parameters initialization. The key findings are that late generalization increase may not imply a transition from "memorization" to "understanding", but can simply be an artifact of the accuracy measure. We provide empirical verification for these propositions, along with preliminary results indicating that some predictions also hold for deeper networks, with non-linear activations.

Differentiation into well-defined patterns and tissue growth are recognized as key processes in organismal development. However, it is unclear whether patterns are passively, homogeneously dilated by growth or whether they remodel during tissue expansion. Leaf vascular networks are well-fitted to investigate this issue, since leaves are approximately two-dimensional and grow manyfold in size. Here we study experimentally and computationally how vein patterns affect growth. We first model the growing vasculature as a network of viscoelastic rods and consider its response to external mechanical stress. We use the so-called texture tensor to quantify the local network geometry and reveal that growth is heterogeneous, resembling non-affine deformations in composite materials. We then apply mechanical forces to growing leaves after veins have differentiated, which respond by anisotropic growth and reorientation of the network in the direction of external stress. External mechanical stress appears to make growth more homogeneous, in contrast with the model with viscoelastic rods. However, we reconcile the model with experimental data by incorporating randomness in rod thickness and a threshold in the rod growth law, making the rods viscoelastoplastic. Altogether, we show that the higher stiffness of veins leads to their reorientation along external forces, along with a reduction in growth heterogeneity. This process may lead to the reinforcement of leaves against mechanical stress. More generally, our work contributes to a framework whereby growth and patterns are coordinated through the differences in mechanical properties between cell types.