With the emergence of big data and advanced technologies, memory devices play an important role. However, there is a lack of efficient approaches for waveform anomaly detection to validate the reliability of memory devices under various operations. In this paper, we propose an efficient segment-level waveform anomaly detection method for memory devices that includes the stages of waveform segmentation and anomaly analysis. Being compatible with general waveform sequences, the proposed method features lightweight computational complexity. During the waveform segmentation stage, we execute downsampled segment collection and downsampled segment pair matching to obtain downsampled segments that are associated with individual operations. During the anomaly analysis stage, we execute ensemble denoising, entrywise alignment, anomaly presence examination through excess kurtosis, and anomaly detection with density-based spatial clustering of applications with noise (DBSCAN) to predict anomaly locations. Simulation results confirm the outstanding performance of the proposed method in terms of matching accuracy, excess kurtosis measurements, and segment-level anomaly detection.

Pre-trained machine learning (ML) models have shown great performance for a wide range of applications, in particular in natural language processing (NLP) and computer vision (CV). Here, we study how pre-training could be used for scientific machine learning (SciML) applications, specifically in the context of transfer learning. We study the transfer behavior of these models as (i) the pretrained model size is scaled, (ii) the downstream training dataset size is scaled, (iii) the physics parameters are systematically pushed out of distribution, and (iv) how a single model pre-trained on a mixture of different physics problems can be adapted to various downstream applications. We find that—when fine-tuned appropriately—transfer learning can help reach desired accuracy levels with orders of magnitude fewer downstream examples (across different tasks that can even be out-of-distribution) than training from scratch, with consistent behaviour across a wide range of downstream examples. We also find that fine-tuning these models yields more performance gains as model size increases, compared to training from scratch on new downstream tasks. These results hold for a broad range of PDE learning tasks. All in all, our results demonstrate the potential of the “pre-train and fine-tune” paradigm for SciML problems, demonstrating a path towards building SciML foundation models. Our code is available as open-source.

We present scalable hybrid-parallel algorithms for training large-scale 3D convolutional neural networks. Deep learning-based emerging scientific workflows often require model training with large, high-dimensional samples, which can make training much more costly and even infeasible due to excessive memory usage. We solve these challenges by extensively applying hybrid parallelism throughout the end-to-end training pipeline, including both computations and I/O. Our hybrid-parallel algorithm extends the standard data parallelism with spatial parallelism, which partitions a single sample in the spatial domain, realizing strong scaling beyond the mini-batch dimension with a larger aggregated memory capacity. We evaluate our proposed training algorithms with two challenging 3D CNNs, CosmoFlow and 3D U-Net. Our comprehensive performance studies show that good weak and strong scaling can be achieved for both networks using up to 2K GPUs. More importantly, we enable training of CosmoFlow with much larger samples than previously possible, realizing an order-of-magnitude improvement in prediction accuracy.