Unsupervised visual defect detection is essential across various industrial applications. Typically, this involves learning a representation space that captures only the features of normal data, and subsequently identifying defects by measuring deviations from this norm. However, balancing the expressiveness and compactness of this space is challenging. The space must be comprehensive enough to encapsulate all regular patterns of normal data, yet without becoming overly expressive, which leads to wasted computational and storage resources and may cause mode collapse-blurring the distinction between normal and defect data embeddings and impairing detection accuracy. To overcome these issues, we introduce a novel approach using an extended VQ-VAE framework optimized for unsupervised defect detection. Unlike traditional methods that apply a constant and uniform representation capacity across an image, our model employs a patch-aware dynamic code assignment scheme. This approach trains the model to allocate codes of varying resolutions based on the context richness of different image regions, aiming to optimize code usage spatially for each sample in a learnable fashion. We also leverage the learned strategy for code allocation in inference, to enlarge the discrepancy between normal and defective samples, thereby improving detection capabilities. Our extensive testing on the MVTecAD, BTAD, and MTSD datasets demonstrates that our model achieves state-of-the-art performance.

As audio-visual systems are being deployed for safety-critical tasks such as surveillance and malicious content filtering, their robustness remains an under-studied area. Existing published work on robustness either does not scale to large-scale dataset, or does not deal with multiple modalities. This work aims to study several key questions related to multi-modal learning through the lens of robustness: 1) Are multi-modal models necessarily more robust than uni-modal models? 2) How to efficiently measure the robustness of multi-modal learning? 3) How to fuse different modalities to achieve a more robust multi-modal model? To understand the robustness of the multi-modal model in a large-scale setting, we propose a density-based metric, and a convexity metric to efficiently measure the distribution of each modality in high-dimensional latent space. Our work provides a theoretical intuition together with empirical evidence showing how multi-modal fusion affects adversarial robustness through these metrics. We further devise a mix-up strategy based on our metrics to improve the robustness of the trained model. Our experiments on AudioSet [1] and Kinetics-Sounds [2] verify our hypothesis that multi-modal models are not necessarily more robust than their uni-modal counterparts in the face of adversarial examples. We also observe our mix-up trained method could achieve as much protection as traditional adversarial training, offering a computationally cheap alternative.

After its sweeping success in vision and language tasks, pure attention-based neural architectures (e.g.DeiT) [1] are emerging to the top of audio tagging (AT) leaderboards [2], which seemingly obsoletes traditional convolutional neural networks (CNNs), feed-forward networks or recurrent networks.However, taking a closer look, there is great variability in published research, for instance, performances of models initialized with pretrained weights differ drastically from without pretraining [2], training time for a model varies from hours to weeks, and often, essences are hidden in seemingly trivial details.This urgently calls for a comprehensive study since our 1st comparison [3] is half-decade old.In this work, we perform extensive experiments on AudioSet [4] which is the largest weakly-labeled sound event dataset available, we also did analysis based on the data quality and efficiency.We compare a few state-of-the-art baselines on the AT task, and study the performance and efficiency of 2 major categories of neural architectures: CNN variants and attention-based variants.We also closely examine their optimization procedures.Our opensourced experimental results 1 provide insights to trade off between performance, efficiency, optimization process, for both practitioners and researchers. 2

As audio/visual classification models are widely deployed for sensitive tasks like content filtering at scale, it is critical to understand their robustness along with improving the accuracy. This work aims to study several key questions related to multimodal learning through the lens of adversarial noises: 1) The trade-off between early/middle/late fusion affecting its robustness and accuracy 2) How does different frequency/time domain features contribute to the robustness? 3) How does different neural modules contribute against the adversarial noise? In our experiment, we construct adversarial examples to attack state-of-the-art neural models trained on Google AudioSet.[1]1We compare how much attack potency in terms of adversarial perturbation of size using different Lpnorms we would need to "deactivate" the victim model. Using adversarial noise to ablate multimodal models, we are able to provide insights into what is the best potential fusion strategy to balance the model parameters/accuracy and robustness trade-off, and distinguish the robust features versus the non-robust features that various neural networks model tend to learn.

Multi-modal machine learning has been a prominent multi-disciplinary research area since its success in complex real-world problems. Empirically, multi-branch fusion models tend to generate better results when there is a high diversity among each branch of the model. However, such experience alone does not guarantee the fusion model's best performance nor have sufficient theoretical support. We present the theoretical estimation of the fusion models' performance by measuring each branch model's performance and the distance between branches based on the analysis of several most popular fusion methods. The theorem is validated empirically by numerical experiments. We further present a branch model selection framework to identify the candidate branches for fusion models to achieve the optimal multi-modal performance by using the theorem. The framework's effectiveness is demonstrated on various datasets by showing how effectively selecting the combination of branch models to attain superior performance.

Optimally scheduling jobs in processing networks to meet multiple objectives for economic considerations and operational efficiencies has been a hot topic. However, most data-driven methods are rarely applied. One of the critical reasons is that the scheduling policy generated by these methods tends to bias toward the specific environment. In order to better deal with the discrepancy, this work in progress paper presents a context-aware dynamic scheduling method (CADS) that can adaptively select a specific policy based on the on-demand context. The CADS has two components: 1) The evaluation module that evaluates the performances of the policies that are learned from each operational context; 2) The decision-making module that maintains the knowledge of each policy’s performance under each context and constructs the weighted best-fit policy based on the identified context. The promising preliminary result using numerical simulation that demonstrates the effectiveness of CADS is presented. CADS outperforms traditional scheduling methods in various kinds of processing network environments.

In the environment of modern processing systems, one topic of great interest is how to optimally schedule (i.e., allocate) jobs with different requirements for the systems to meet various objectives. Methods using distributed reinforcement learning (DRL) have recently achieved great success with large-scale dynamic scheduling problems. However, most DRL methods require a huge amount of computational time and a large amount of data for the DRL agents for a control policy. Meanwhile, various scheduling experts have already developed several scheduling policies (i.e., dispatching rules) that can successfully fulfill different objectives with acceptable performance for a processing system based on their understandings of the processing systems characteristics. In this paper, we propose to learn from experts to reduce the learning and searching costs of a good policy in large-scale dynamic scheduling problems. In the learning process, our DRL agents select the experts who have better performance in the scheduling environment, observe the experts' actions, and learn a scheduling policy guided by these experts' demonstrations. Our realistic simulations results demonstrate that this expert-guided DRL (EGDRL) approach outperforms DRL methods without expert guidance, as well as some other reinforcement learning from demonstration (RLfD) methods in several systems. To the best of our knowledge, our research is one of the first works that incorporates existing expert policies to guide the learning of optimal policies for large-scale dynamic scheduling problems.

Voice Assistants (VAs) such as Amazon Alexa or Google Assistant rely on wake-word detection to respond to people's commands, which could potentially be vulnerable to audio adversarial examples. In this work, we target our attack on the wake-word detection system. Our goal is to jam the model with some inconspicuous background music to deactivate the VAs while our audio adversary is present. We implemented an emulated wake-word detection system of Amazon Alexa based on recent publications. We validated our models against the real Alexa in terms of wake-word detection accuracy. Then we computed our audio adversaries with consideration of expectation over transform and we implemented our audio adversary with a differentiable synthesizer. Next we verified our audio adversaries digitally on hundreds of samples of utterances collected from the real world. Our experiments show that we can effectively reduce the recognition F1 score of our emulated model from 93.4% to 11.0%. Finally, we tested our audio adversary over the air, and verified it works effectively against Alexa, reducing its F1 score from 92.5% to 11.0%. To the best of our knowledge, this is the first real-world adversarial attack against a commercial grade VA wake-word detection system. Our demo video is included in the supplementary material.

In the area of demand-driven manufacturing systems, one critical problem currently is how to dynamically and optimally schedule, i.e., allocate, actual jobs with different customer requirements in large-scale manufacturing systems in order to meet the various objectives. Scheduling experts have proposed several scheduling methods with acceptable performance for manufacturing systems based on their understanding of the systems' characteristics. However, the problem remains challenging due to the complicated composition of the multiple objectives of the system, the complex system dynamics, constraints, and the extremely high computational cost for large-scale manufacturing systems. In this paper, we apply a stochastic processing network that can capture the stochasticity and dynamics of discrete manufacturing systems. We then propose a data-driven, distributed reinforcement learning (DRL) method so that little information about the system dynamics is required, and the learning and search costs of a scheduling policy with high production performance in a processing system can be reduced, thus this method is capable of scaling to large-scale processing systems. In particular, we first use a stochastic processing network, i.e., a queueing model, to represent the production processes in a typical discrete manufacturing system so that it can be simulated. We then decompose the reinforcement learning into local processes. Each local processs agent can make decisions locally by assigning indices to jobs based on each job's real-time information (index policy). Because of this distributed learning characteristic and index policy, our approach is much more scalable and efficient than either centralized methods or traditional decentralized reinforcement learning methods. Based on our simulation, we find our approach can achieve higher production performance than other heuristics, past decentralized reinforcement learning methods, or centralized methods in the stochastic processing networks with different scales.

Learning acoustic models directly from the raw waveform data with minimal processing is challenging. Current waveform-based models have generally used very few (~2) convolutional layers, which might be insufficient for building high-level discriminative features. In this work, we propose very deep convolutional neural networks (CNNs) that directly use time-domain waveforms as inputs. Our CNNs, with up to 34 weight layers, are efficient to optimize over very long sequences (e.g., vector of size 32000), necessary for processing acoustic waveforms. This is achieved through batch normalization, residual learning, and a careful design of down-sampling in the initial layers. Our networks are fully convolutional, without the use of fully connected layers and dropout, to maximize representation learning. We use a large receptive field in the first convolutional layer to mimic bandpass filters, but very small receptive fields subsequently to control the model capacity. We demonstrate the performance gains with the deeper models. Our evaluation shows that the CNN with 18 weight layers outperforms the CNN with 3 weight layers by over 15% in absolute accuracy for an environmental sound recognition task and is competitive with the performance of models using log-mel features.

Environmental sound detection is a challenging application of machine learning because of the noisy nature of the signal, and the small amount of (labeled) data that is typically available. This work thus presents a comparison of several state-of-the-art Deep Learning models on the IEEE challenge on Detection and Classification of Acoustic Scenes and Events (DCASE) 2016 challenge task and data, classifying sounds into one of fifteen common indoor and outdoor acoustic scenes, such as bus, cafe, car, city center, forest path, library, train, etc. In total, 13 hours of stereo audio recordings are available, making this one of the largest datasets available.

Advanced manufacturing, which employs the latest information and communication technologies to facilitate interconnected, efficient, and adaptive manufacturing systems, has become a prominent research topic in both academics and industry in recent years. One critical aspect of advanced manufacturing is how to match various complex job requirements with a manufacturer's real-time processing capabilities and its other resources within a job shop to optimally schedule manufacturing processes with multiple objectives. In general, a manufacturing scheduling problem is a non-deterministic, polynomial-time (NP)-hard problem. Due to its complexity, scheduling problems present a number of challenges to find the best possible solutions. In order to deal with a dynamic job shop scheduling problem-particularly, a dispatching problem-for a manufacturing system that is able to handle multiple product types through multi-stages and multi-machines with dynamic orders, stochastic processing time and setup time. Based on a generic framework, this research develops a solution by using a reinforcement learning-based scheduling approach that can adaptively update the production schedule by utilizing the real-time product and process events information during executions. More specifically, we first propose a framework that describes the process of dealing with a complex scheduling problem. Next, to learn the dispatching pattern, we formally define the scheduling problem through the construction of objective functions and related constraints for the manufacturing tasks. We then apply a reinforcement learning approach that incorporates the real-time production environment to generate optimal policies under various manufacturing-process conditions. When tested under different objectives and constraint conditions, our results demonstrate that the proposed learning-based method provides better performance than most common dispatching rules.

Due to rapid development of information and communications technology (ICT) and the impetus for more effective, efficient and adaptive manufacturing, the concept of ICT based advanced manufacturing has increasingly become a prominent research topic across academia and industry during recent years. One critical aspect of advanced manufacturing is how to incorporate real time information and then optimally schedule manufacturing processes with multiple objectives. Due to its complexity and the need for adaptation, the manufacturing scheduling problem presents challenges for utilizing advanced ICT and thus calls for new approaches. The paper proposes a centralized reinforcement learning approach for optimally scheduling of a manufacturing system of multi-stage processes and multiple machines for multiple types of products. The approach, which employs learning and control algorithms to enable real time cooperation of each processing unit inside the system, is able to adaptively respond to dynamic scheduling changes. More specifically, we first formally define the scheduling problem through the construction of an objective function and related heuristic constraints for the underlying manufacturing tasks. Next, to effectively deal with the problem we defined, we maintain a distributed weighted vector to capture the cooperative pattern of massive action space and apply the reinforcement-learning approach to achieve the optimal policies for a set of processing machines according to a real time production environment, including dynamic requests for various products. Numerical experiments demonstrate that compared to different heuristic methods and multi-agent algorithms, the proposed centralized reinforcement learning method can provide more reliable solutions for the scheduling problem.