Abstract The classical universal approximation (UA) theorem for neural networks establishes mild conditions under which a feedforward neural network can approximate a continuous function f with arbitrary accuracy. A recent result shows that neural networks also enjoy a more general interval universal approximation (IUA) theorem, in the sense that the abstract interpretation semantics of the network using the interval domain can approximate the direct image map of f (i.e., the result of applying f to a set of inputs) with arbitrary accuracy. These theorems, however, rest on the unrealistic assumption that the neural network computes over infinitely precise real numbers, whereas their software implementations in practice compute over finite-precision floating-point numbers. An open question is whether the IUA theorem still holds in the floating-point setting. This paper introduces the first IUA theorem for floating-point neural networks that proves their remarkable ability to perfectly capture the direct image map of any rounded target function f , showing no limits exist on their expressiveness. Our IUA theorem in the floating-point setting exhibits material differences from the real-valued setting, which reflects the fundamental distinctions between these two computational models. This theorem also implies surprising corollaries, which include (i) the existence of provably robust floating-point neural networks; and (ii) the computational completeness of the class of straight-line programs that use only floating-point additions and multiplications for the class of all floating-point programs that halt.

Existing works on the expressive power of neural networks typically assume real-valued parameters and exact mathematical operations during the evaluation of networks. However, neural networks run on actual computers can take parameters only from a small subset of the reals and perform inexact mathematical operations with round-off errors and overflows. In this work, we study the expressive power of floating-point neural networks, i.e., networks with floating-point parameters and operations. We first observe that for floating-point neural networks to represent all functions from floating-point vectors to floating-point vectors, it is necessary to distinguish different inputs: the first layer of a network should be able to generate different outputs for different inputs. We also prove that such distinguishability is sufficient, along with mild conditions on activation functions. Our result shows that with practical activation functions, floating-point neural networks can represent floating-point functions from a wide domain to all finite or infinite floats. For example, the domain is all finite floats for Sigmoid and tanh, and it is all finite floats of magnitude less than 1/8 times the largest float for ReLU, ELU, SeLU, GELU, Swish, Mish, and sin.

Cognitive development studies how information processing in the brain changes over the course of development. A key part of this question is how information is represented and stored in memory. This study examined allocentric (world-based) spatial memory, an important cognitive tool for planning routes and interacting with the space around us. This is typically theorized to use multiple landmarks all at once whenever it operates. In contrast, here we show that allocentric spatial memory frequently operates over a limited spatial window, much less than the full proximal scene, for children between 3.5 and 8.5 years old. The use of multiple landmarks increases gradually with age. Participants were asked to point to a remembered target location after a change of view in immersive virtual reality. A k-fold cross-validation model-comparison selected a model where young children usually use the target location's vector to the single nearest landmark and rarely take advantage of the vectors to other nearby landmarks. The comparison models, which attempt to explain the errors as generic forms of noise rather than encoding to a single spatial cue, did not capture the distribution of responses as well. Parameter fits of this new single- versus multi-cue model are also easily interpretable and related to other variables of interest in development (age, executive function). Based on this, we theorize that spatial memory in humans develops through three advancing levels (but not strict stages): most likely to encode locations egocentrically (relative to the self), then allocentrically (relative to the world) but using only one landmark, and finally, most likely to encode locations relative to multiple parts of the scene.