We study an online and streaming unsupervised classification system. Our setting consists of a collection of classifiers (with unknown confusion matrices) each of which can classify one sample per unit time, and which are accessed by a stream of unlabeled samples. Each sample is dispatched to one or more classifiers, and depending on the labels collected from these classifiers, may be sent to other classifiers to collect additional labels. The labels are continually aggregated. Once the aggregated label has high enough accuracy (a pre-specified threshold for accuracy) or the sample is sent to all the classifiers, the now labeled sample is ejected from the system. For any given pre-specified threshold for accuracy, the objective is to sustain the maximum possible rate of arrival of new samples, such that the number of samples in memory does not grow unbounded. In this paper, we characterize the Pareto-optimal region of accuracy and arrival rate, and develop an algorithm that can operate at any point within this region. Our algorithm uses queueing-based routing and scheduling approaches combined with novel online tensor decomposition method to learn the hidden parameters, to Pareto-optimality guarantees. We finally verify our theoretical results through simulations on two ensembles formed using AlexNet, VGG, and ResNet deep image classifiers.

State of the art visual classification continues to improve, particularly with the use of deep learning and millions of labeled images. However, the effort required to label training sets of this size has led to semi-supervised approaches that collect partially noisy labeled data with less effort. Label noise has been shown to degrade supervised learning, but these analyses focus on noise from erroneous label assignment of data instances. Group-based labeling reduces workload by assigning a single label to a group of images simultaneously, which introduces label noise with structure dependent on all training instances. This work investigates the impact of group-based label noise on classifier learning, and discusses how and why this differs from instance-based label noise. We also discuss label noise modeling designed to provide more robust classification given noisy training instances, and evaluate the generalization of these techniques to group-based noise.

The output encodings of neural nets determine the structure of the space in which inference occurs. Yet, they are generally given very little thought. It is common practice for neural nets to use 1-Hot encoding when training to discriminate among many classes. The primary exceptions to this are error correcting output codes, and semantic output encodings. Output encodings based upon semantic descriptors cause a net to learn responses for classes to which it has not been exposed, provided those classes may be characterized by the same semantic descriptors. This raises a number of questions, such as “Can a net implicitly learn encodings for unobserved classes in the absence of a semantic encoding, or any encoding that requires some form of prior knowledge and hand crafting?”. Also, are some output encodings better than others for learning these implicit encodings? In this paper, we will compare how effectively different non-semantic encodings are at causing a neural net to implicitly learn encodings for unobserved classes. Also, while evaluating the efficacy of these implicit encodings, we will look for evidence of a phenomenon akin to over-training. Specifically, as training on the observed classes occurs, we initially see improvement in how well the implicitly learned encodings can be used to differentiate among the classes which are unobserved during the net's training. However, as training continues to improve discrimination among the observed classes, the efficacy of the implicit codes either remains steady, or undergoes a degradation. This degradation is akin to the overtraining that one generally tries to guard against when training a neural net.

Historically, neural nets have learned new things at the cost of forgetting what they already know. This problem is known as 'catastrophic forgetting'. Here, we examine how training a neural net in accordance with latently learned [1] output encodings drastically reduces catastrophic forgetting. Previous approaches to dealing with catastrophic forgetting have tended either to add extra samples to new training sets, modify the training of hidden nodes or model the interaction between short term and long term memory. Our approach is unique in that it both uses transfer learning to mitigate catastrophic forgetting and focuses upon the output nodes of a neural network. This results in a technique that makes it easier rather than harder to learn new tasks while retaining existing knowledge; is architecture independent and trivial to implement on any existing net. Additionally, we examine the use of ternary output codes. Binary codes assign a value to each output bit that may be thought of as either affirmative or negative. Ternary codes allow for the possibility that not every output bit has a meaningful response to every given input. By not forcing each output bit to train for a specific response for each new class, we hope to lessen catastrophic forgetting.

Best Fit Least-Squares (BFLS) is a required technique for many STEM subjects. It is a method to compute a linear model for a set of data points. Due to its utility, BFSL is frequently taught to STEM students before they have sufficient mathematical experience to follow the mechanics of a derivation. Not only does this fail to produce procedural and conceptual understandings, but also it encourages students to view formulae and algorithms as things to be looked up, rather than derived. This is discourages students from developing productive dispositions. In this paper, we describe the “Close Fit Rod-Spring” (CFRS) approach to the problem. This approach computes the resting position of a rigid rod, which is connected by vertically oriented springs to a set of data points. This method of derivation results in two linear equations that may be solved for the slope and intercept of the best fit line. The result is equivalent to BFSL. However, it is achieved in an intuitively understandable and mathematically accessible way for high school students.

A neural net can learn to discriminate among a set of classes without explicitly training to do so. It does not even need exposure to any instances of those classes. The learning occurs while the net is being trained to discriminate among a set of related classes. This form of transfer learning is referred to as `Latent Learning' by psychologists, because until specifically elicited, the knowledge remains latent. Evidence that latent learning has occurred lies in the existence of consistent, unique responses to the unseen classes. Standard supervised learning can improve the accuracy of those responses with exceedingly small sets of labeled images. In this paper, we use a convolutional neural net (CNN) to demonstrate not only a method of determining a net's latent responses, but also simple ways to optimize latent learning. Additionally, we take advantage of the fact that CNN's are deep nets in order to show how the latently learned accuracy of the CNN may be greatly improved by allowing only its output layer to train. We compare our results both to those obtained with standard backpropagation training of the CNN on small datasets without any transfer learning and to a related set of current published results.

Psychologists define latent learning as learning that occurs without task-specific reinforcement and is not demonstrated until needed. Since this knowledge is acquired while mastering some other task(s), it is a form of transfer learning. We utilize latent learning to enable a deep neural net to distinguish among a set of handwritten numerals. The accuracies obtained are compared to those achievable with a simplistic `group-mean' classification technique, which is explained later in this paper. The deep neural net architecture used was a Le-Net 5 convolutional neural net with only minor differences in the output layer.