We present an extension to masked autoencoders (MAE) which improves on the representations learnt by the model by explicitly encouraging the learning of higher scene-level features. We do this by: (i) the introduction of a perceptual similarity term between generated and real images (ii) incorporating several techniques from the adversarial training literature including multi-scale training and adaptive discriminator augmentation. The combination of these results in not only better pixel reconstruction but also representations which appear to capture better higher-level details within images. More consequentially, we show how our method, Perceptual MAE, leads to better performance when used for downstream tasks outperforming previous methods. We achieve 78.1% top-1 accuracy linear probing on ImageNet-1K and up to 88.1% when fine-tuning, with similar results for other downstream tasks, all without use of additional pre-trained models or data.

Two generalizations of the Varshamov-Gilbert bound for error-correcting and error-detecting codes are developed. Sufficient intrinsic conditions are given for classes of linear codes over GF(q)to include "good" codes, and these classes are related to other known classes. A lower bound on the maximal order of subspaces contained in subsets of certain finite vector spaces is given and related to a bound on error-detecting codes.