Knowledge distillation (KD) has been a popular and effective method for model compression. One important assumption of KD is that the teacher’s original dataset will also be available when training the student. However, in situations such as continual learning and distilling large models trained on company-withheld datasets, having access to the original data may not always be possible. This leads practitioners towards utilizing other sources of supplemental data, which could yield mixed results. One must then ask: "what makes a good dataset for transferring knowledge from teacher to studentƒ" Many would assume that only real in-domain imagery is viable, but is that the only optionƒ In this work, we explore multiple possible surrogate distillation datasets and demonstrate that many different datasets, even unnatural synthetic imagery, can serve as a suitable alternative in KD. From examining these alternative datasets, we identify and present various criteria describing what makes a good dataset for distillation. Source code is available at https://github.com/osu-cvl/good-kd-dataset.

Abstract Existing Reject Option Classification formulations typically learn a function (e.g., softmax threshold) to select/reject uncertain classifier predictions. Such approaches leverage a user-defined cost of rejection or constraints on the accuracy or coverage of the selected predictions. We formulate a new objective for applications that have no such costs/constraints by using a natural constraint on the rejected predictions. Our proposed Reject Option Classification formulation eliminates regions of random chance classification in the decision space of any neural classifier and dataset. The goal is to maximize accuracy in the selected region while permitting a reasonable degree of prediction randomness in the rejected region. Optimally, the hope would be to reject more incorrect than correct predictions. We employ a novel selection/rejection function and learn per-class softmax thresholds using a validation set. Results demonstrate the advantages of our proposed method compared to naïvely thresholding calibrated/uncalibrated softmax scores. We evaluate 2-D points, imagery, and text classification datasets using state-of-the-art pretrained and learned models. Source code is available at https://github.com/osu-cvl/learning-idk .

Semi-supervised learning approaches train on small sets of labeled data in addition to large sets of unlabeled data. Self-training is a semi-supervised teacher-student approach that often suffers from "confirmation bias" that occurs when the student model repeatedly overfits to incorrect pseudo-labels given by the teacher model for the unlabeled data. This bias impedes improvements in pseudo-label accuracy across self-training iterations, leading to unwanted saturation in model performance after just a few iterations. In this work, we study multiple design choices to improve the Noisy Student self-training pipeline and reduce confirmation bias. We showed that our proposed Weighted SplitBatch Sampler and Dataset-Adaptive Techniques for Model Calibration and Entropy-Based Pseudo-Label Selection provided performance gains over existing design choices across multiple datasets. Finally, we also study the extendability of our enhanced approach to Open Set unlabeled data (containing classes not seen in labeled data). The source code can be licensed for use via email.

There is a recently discovered and intriguing phenomenon called Neural Collapse: at the terminal phase of training a deep neural network for classification, the within-class penultimate feature means and the associated classifier vectors of all flat classes collapse to the vertices of a simplex Equiangular Tight Frame (ETF). Recent work has tried to exploit this phenomenon by fixing the related classifier weights to a pre-computed ETF to induce neural collapse and maximize the separation of the learned features when training with imbalanced data. In this work, we propose to fix the linear classifier of a deep neural network to a Hierarchy-Aware Frame (HAFrame), instead of an ETF, and use a cosine similarity-based auxiliary loss to learn hierarchy-aware penultimate features that collapse to the HAFrame. We demonstrate that our approach reduces the mistake severity of the model's predictions while maintaining its top-1 accuracy on several datasets of varying scales with hierarchies of heights ranging from 3 to 12. Code: https://github.com/ltong1130ztr/HAFrame.

Real-world datasets are growing ever larger and more dynamic, motivating the need for efficient online data exploration algorithms. In this work, we present S-FINCH, a streaming domain optimization of the recent FINCH clustering method. The original FINCH approach demonstrated state-of-the-art offline performance while avoiding sensitive hyperparameters but was not designed for the online data streaming domain. The proposed S-FINCH method is an exact, incremental update to the FINCH approach which generates high-quality clusters while avoiding the potential sensitivities of other online clustering methods. We also provide alternative cluster tree representatives for faster empirical cluster times. Experiments are shown comparing the original FINCH approach to the proposed S-FINCH method in a streaming domain with multiple benchmark synthetic and real datasets. The S-FINCH method leverages the ability to make local changes for efficient, real time updates and can be applied to multiple data streaming scenarios.

The application of convolutional neural networks (CNNs) and deep learning to different domains has become increasingly popular in the last several years. In particular, such models have been used in the agriculture domain to identify plant species, identify plant stresses, and estimate crop yields. Although there has been much success in applying these techniques to the agriculture domain, these works contain many shortcomings that are hindering their chance for adoption in practice (e.g., lack of domain knowledge, predicting only specific stress types, etc.). We address issues of previous works for the task of plant stress identification by applying a hierarchical classification approach employing confidence as a means to determine the specificity of a classification. This work is a collaboration between computer science and agricultural engineering experts.

We present a bottom-up approach to hierarchical classification based on posteriors conditioned with logits. Beginning with the output logits for a set of terminal labels from a base classifier, an initial hypothesis is repeatedly generalized (softened) to a weaker label until a particular confidence measure is achieved. As conditioning the probabilistic model with the full set of terminal logits quickly becomes intractable for large label sets, we propose an alternative approach employing “generalized logits” spanning relevant hypotheses within the label hierarchy. Experimental results are compared with related methods on multiple datasets and base classifiers. The proposed approach provides an efficient and effective hierarchical classification framework with monotonic, non-decreasing inference behavior.

This paper describes a concept called Streaming Workflows that can collect data from sensors connected to edge devices and pass on the heavy load of computation to on-demand cloud services including Microsoft Azure, Amazon Web Services and Google Cloud Platform. The data streaming is done on the edge device while contextualization and modeling of the data are done with on-demand cloud resources. Many workflows using this edge-cloud architecture will be deployed on the cloud-based Smart Manufacturing (SM) PlatformTM developed by the Clean Energy Smart Manufacturing Innovation Institute (CESMII) at UCLA. Kepler workflows are used to orchestrate and manage the deployment of compute resources, the data transfer, data contextualization, modeling, and termination of the compute resources on a cloud platform. Test data in this study were from an aluminum rolling mill. The objective was to use operating data to predict exit temperature using an edge and Microsoft Azure architecture. This work addresses how to implement a run-time model-based control and optimization approach using Streaming Workflows for similar projects.

Article Free Access Share on Flexible specification of workflow compensation scopes Authors: Weimin Du Hewlett-Packard Laboratories, 1501 Page Mill Road, Palo Alto, CA Hewlett-Packard Laboratories, 1501 Page Mill Road, Palo Alto, CAView Profile , Jim Davis Hewlett-Packard Laboratories, 1501 Page Mill Road, Palo Alto, CA Hewlett-Packard Laboratories, 1501 Page Mill Road, Palo Alto, CAView Profile , Ming-Chien Shan Hewlett-Packard Laboratories, 1501 Page Mill Road, Palo Alto, CA Hewlett-Packard Laboratories, 1501 Page Mill Road, Palo Alto, CAView Profile Authors Info & Claims GROUP '97: Proceedings of the 1997 ACM International Conference on Supporting Group WorkNovember 1997 Pages 309–316https://doi.org/10.1145/266838.267342Published:16 November 1997Publication History 24citation445DownloadsMetricsTotal Citations24Total Downloads445Last 12 Months4Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF