In supervised learning, obtaining a large set of fully-labeled training data is expensive. We show that we do not always need full label information on every single training example to train a competent classifier. Specifically, inspired by the principle of sufficiency in statistics, we present a statistic (a summary) of the fully-labeled training set that captures almost all the relevant information for classification but at the same time is easier to obtain directly. We call this statistic "sufficiently-labeled data" and prove its sufficiency and efficiency for finding the optimal hidden representations, on which competent classifier heads can be trained using as few as a single randomly-chosen fully-labeled example per class. Sufficiently-labeled data can be obtained from annotators directly without collecting the fully-labeled data first. And we prove that it is easier to directly obtain sufficiently-labeled data than obtaining fully-labeled data. Furthermore, sufficiently-labeled data is naturally more secure since it stores relative, instead of absolute, information. Extensive experimental results are provided to support our theory.

Recently, Information Theoretic Learning (ITL) has helped explain the learning dynamics for deep learning models such as multilayer perceptrons (MLP), convolutional neural networks (CNN), and stacked autoencoders (SAE). It is understood that for MLPs and CNNs, where the desired signal and input are independent of each other, the set of consecutive layers in the primary and adjoint networks represent individual Markov Chains (MC) that effect two data processing inequalities (DPI) in their respective directions. For the SAE, the desired signal is the input, so the DPI is only confirmed until the bottleneck layer. In this paper, we propose using the adjoint network to compute conditional mutual information with the backpropagated errors to demonstrate the DPI until the SAE's last layer. Also, we present an ITL-based analysis of the residual network (ResNet) architecture and propose an explanation for why the identity mapping is the optimal shortcut connection.