Current approaches to resilience for highly distributed, heterogeneous, large-scale scientific workflows are limited. Most existing workflow and resource management systems have a single point of failure and resilience strategies are often static, depend on a centralized control, and require considerable design effort from experts. The increasing scale and complexity of workflows coupled with limited resilience capabilities in centralized systems necessitates a fully decentralized, adaptive resource management approach. This paper addresses a very important slice of the overall problem by leveraging the advances in multi-agent systems (MAS). In particular, we explore the suitability of a MAS consisting of globally distributed agents to perform distributed job selection from a dynamic job pool in a truly decentralized, performant, and resilient manner. We present a novel consensus formulation of the distributed job selection problem. By introducing a cost function encapsulating the requirements and constraints of the job and resource loads, we design a novel, greedy consensus algorithm leveraging the Practical Byzantine Fault Tolerance (PBFT)-based consensus method, allowing agents to collectively select jobs in a resilient manner. We compared our algorithms with other state of the art approaches by deploying them in a network testbed infrastructure to emulate distributed job selection. Our evaluation results demonstrated that our greedy consensus algorithm employing the cost-function and PBFT-based consensus method outperforms the ones using the vanilla PBFT-based consensus method - improving scheduling latency by as much as 63.5 % and reducing resource idle time by as much as 63.8 %, with benefits increasing with higher numbers of agents emulated.

Data rebalancing techniques, including undersampling and oversampling, are common approaches to the problem of classifying imbalanced data. However, indiscriminate oversampling increases the size of the dataset with no regard to "how these additional samples effect the classification in imbalanced datasets." Such behavior often leads to less than desirable accuracy in the presence of noise and introduces underfitting. To prevent these learning obstacles, we developed a bilevel optimization framework where we optimize over samples of majority training data after typical optimization over model parameters takes place. Our framework allows us to dynamically assess a sample’s impact on loss and accepts only those datapoints that improve loss, thereby, finding an optimal subset of majority training data for classification. Experimental results show our proposed technique with F1 scores up to 10% higher than state-of-the-art methods.

This paper discusses the key challenges and future research directions for privacy-preserving federated learning (PPFL), with a focus on its application to large-scale scientific artificial intelligence models, in particular, foundation models (FMs). PPFL enables collaborative model training across distributed datasets while preserving privacy—an important collaborative approach for science. We discuss the need for efficient and scalable algorithms to address the increasing complexity of FMs, particularly when dealing with heterogeneous clients. In addition, we underscore the need for developing advance privacy-preserving techniques, such as differential privacy, to balance privacy and utility in large FMs emphasizing fairness and incentive mechanisms to ensure equitable participation among heterogeneous clients. Finally, we emphasize the need for a robust software stack supporting scalable and secure PPFL deployments across multiple high-performance computing facilities. We envision that PPFL would play a crucial role to advance scientific discovery and enable large-scale, privacy-aware collaborations across science domains.

In contemporary scientific research, data acquisition and analysis platforms have grown increasingly complex, often spanning multiple facilities with diverse internal structures. Efficiently managing the interactions between job scheduling, resource allocation, and networking across these distributed systems requires a robust simulation framework. However, existing simulators fall short in capturing the detailed interactions necessary for comprehensive analysis of large-scale distributed environments. To address this gap, we introduce DISTRI, a versatile framework specifically designed for the development and testing of distributed multi-facility workflows. DISTRI allows for customizable facility configurations and includes built-in support for distributed, resilient scheduling and resource management, alongside detailed network simulation for data communication. Key features of DISTRI encompass inter- and intra-facility resource management, agent-based distributed scheduling, and extensive performance metrics logging for both resource and network management. By providing these essential tools, DISTRI enables thorough analysis and optimization, thereby advancing research in the resilience and efficiency of multi-facility systems.

Anomaly detection in computational workflows is critical for ensuring system reliability and security. However, traditional rule-based methods struggle to detect novel anomalies. This paper leverages large language models (LLMs) for workflow anomaly detection by exploiting their ability to learn complex data patterns. Two approaches are investigated: (1) supervised fine-tuning (SFT), where pretrained LLMs are fine-tuned on labeled data for sentence classification to identify anomalies, and (2) in-context learning (ICL), where prompts containing task descriptions and examples guide LLMs in few-shot anomaly detection without fine-tuning. The paper evaluates the performance, efficiency, and generalization of SFT models and explores zeroshot and few-shot ICL prompts and interpretability enhancement via chain-of-thought prompting. Experiments across multiple workflow datasets demonstrate the promising potential of LLMs for effective anomaly detection in complex executions.

In this article, we address two key challenges in deep reinforcement learning (DRL) setting, sample inefficiency and slow learning, with a dual-neural network (NN)-driven learning approach. In the proposed approach, we use two deep NNs with independent initialization to robustly approximate the action-value function in the presence of image inputs. In particular, we develop a temporal difference (TD) error-driven learning (EDL) approach, where we introduce a set of linear transformations of the TD error to directly update the parameters of each layer in the deep NN. We demonstrate theoretically that the cost minimized by the EDL regime is an approximation of the empirical cost, and the approximation error reduces as learning progresses, irrespective of the size of the network. Using simulation analysis, we show that the proposed methods enable faster learning and convergence and require reduced buffer size (thereby increasing the sample efficiency).

Deep neural networks are powerful predictors for a variety of tasks. However, they do not capture uncertainty directly. Using neural network ensembles to quantify uncertainty is competitive with approaches based on Bayesian neural networks while benefiting from better computational scalability. However, building ensembles of neural networks is a challenging task because, in addition to choosing the right neural architecture or hyperparameters for each member of the ensemble, there is an added cost of training each model. To address this issue, we propose AutoDEUQ, an automated approach for generating an ensemble of deep neural networks. Our approach leverages joint neural architecture and hyperparameter search to generate ensembles. We use the law of total variance to decompose the predictive variance of deep ensembles into aleatoric (data) and epistemic (model) uncertainties. We show that AutoDEUQ outperforms probabilistic backpropagation, Monte Carlo dropout, deep ensemble, distribution-free ensembles, and hyper ensemble methods on a number of regression benchmarks.

In this paper, a novel approach is presented to mitigate the issue of domain shift observed in big-data classification. Since little information is available about the shift, we introduce a "distortion model", and obtain additional data-samples to represent the shift. Next, a deep neural network (DNN), referred as "classifier," is used to compensate for the shift by learning through these additional samples while maintaining performance on training samples. As the exact magnitude of domain shift is uncertain, we compensate for the optimal expected shift by formulating a zero-sum game. In the proposed game, the distortion model is viewed as the maximizing player which increases the domain shift while the classifier becomes the minimizing player that reduces the impact of domain shift on learning. The Nash solution of the game, which is demonstrated mathematically, provides the domain shift and its optimal adaptation through the classifier. To solve the proposed game for the Nash solution, a direct error-driven learning scheme is introduced where a cost function is derived and solved for each layer in the DNN and the distortion model. Comprehensive mathematical and simulation study is presented to demonstrate the efficacy of the approach.

We formulate the continual learning (CL) problem via dynamic programming and model the trade-off between catastrophic forgetting and generalization as a two-player sequential game. In this approach, player 1 maximizes the cost due to lack of generalization whereas player 2 minimizes the cost due to catastrophic forgetting. We show theoretically that a balance point between the two players exists for each task and that this point is stable (once the balance is achieved, the two players stay at the balance point). Next, we introduce balanced continual learning (BCL), which is designed to attain balance between generalization and forgetting and empirically demonstrate that BCL is comparable to or better than the state of the art.

In this article, a novel learning methodology is introduced for the problem of classification in the context of high-dimensional data. In particular, the challenges introduced by high-dimensional data sets are addressed by formulating a$L_{1}$regularized zero-sum game where optimal sparsity is estimated through a two-player game between the penalty coefficients/sparsity parameters and the deep neural network weights. In order to solve this game, a distributed learning methodology is proposed where additional variables are utilized to derive layerwise cost functions. Finally, an alternating minimization approach developed to solve the problem where the Nash solution provides optimal sparsity and compensation through the classifier. The proposed learning approach is implemented in a parallel and distributed environment through a novel computational algorithm. The efficiency of the approach is demonstrated both theoretically and empirically with nine data sets.

In this paper, a multi-layer neural network (MNN) based online optimal adaptive regulation of a class of nonlinear discrete-time systems in affine form with uncertain internal dynamics is introduced. The multi-layer neural networks (MNN)-based actor-critic framework is utilized to estimate the optimal control input and cost function. The temporal difference (TD) error is derived from the difference between actual and estimated cost function. The MNN weights of both critic and actor are tuned at every sampling instant as a function of the instantaneous temporal difference and control policy errors. The proposed approach does not require the selection of any basis function and its derivatives. The boundedness of the system state vector and actor and critic NN weights are shown through Lyapunov theory. Extension of the proposed approach to MNNs with more hidden layers is discussed. Simulation results are provided to illustrate the effectiveness of the proposed approach.

In this brief, heterogeneity and noise in big data are shown to increase the generalization error for a traditional learning regime utilized for deep neural networks (deep NNs). To reduce this error, while overcoming the issue of vanishing gradients, a direct error-driven learning (EDL) scheme is proposed. First, to reduce the impact of heterogeneity and data noise, the concept of a neighborhood is introduced. Using this neighborhood, an approximation of generalization error is obtained and an overall error, comprised of learning and the approximate generalization errors, is defined. A novel NN weight-tuning law is obtained through a layer-wise performance measure enabling the direct use of overall error for learning. Additional constraints are introduced into the layer-wise performance measure to guide and improve the learning process in the presence of noisy dimensions. The proposed direct EDL scheme effectively addresses the issue of heterogeneity and noise while mitigating vanishing gradients and noisy dimensions. A comprehensive simulation study is presented where the proposed approach is shown to mitigate the vanishing gradient problem while improving generalization by 6%.

In this paper, a novel dimension-reduction approach is presented to overcome challenges such as nonlinear relationships, heterogeneity, and noisy dimensions. Initially, the p attributes in the data are first organized into random groups. Next, to systematically remove redundant and noisy dimensions from the data, each group is independently mapped into a low dimensional space via a parametric mapping. The group-wise transformation parameters are estimated using a low-rank approximation of distance covariance. The transformed attributes are reorganized into groups based on the magnitude of their respective eigenvalues. The group-wise organization and reduction process is performed until a user-defined criterion on eigenvalues is satisfied. In addition, novel procedures are introduced to aggregate the transformation parameters when the data is available in batches. Overall performance is demonstrated with extensive simulation analysis on classification by employing 10 data-sets.

While analyzing high dimensional data-sets using deep neural network (NN), increased sparsity is desirable but requires careful selection of "sparsity parameters." In this paper, a novel distributed learning methodology is proposed to optimize the NN while addressing this challenge. To address this challenge, the optimal sparsity in the NN is estimated via a two player zero-sum game in the paper. In the proposed game, sparsity parameter is the first player with the aim of increasing sparsity in the NN while NN weights is the second player with the goal of improving its performance in the presence of increased sparsity. To solve the game, additional variables are introduced into the optimization problem such that the output at every layer in the NN depends on this variable instead of the previous layer. Using these additional variables, layer wise cost-functions are derived that are then independently optimized to learn the additional variables, NN weights and the sparsity parameters. To implement the proposed learning procedure in a parallelized and distributed environment, a novel computational algorithm is also proposed. The efficiency of the proposed approach is demonstrated using a total of six data-sets.

In this paper, a novel methodology to reduce the generalization errors occurring due to domain shift in big data classification is presented. This reduction is achieved by introducing a suitably selected domain shift to the training data via what is referred to as "distortion model". These distortions are introduced through an affine transformation and additional data-samples are obtained. Next, a deep neural network (NN), referred as "classifier", is used to classify both the original and the additional data samples. By learning from both the original and additional data-samples, the classifier compensates for the domain shift while maintaining its performance on original data. However, as the exact magnitude of the shift one would encounter in real applications is unknown a priori and difficult to predict. The objective is to compensate for the optimal shift that can be introduced by the distortion model without significantly degrading the performance of the model. A two-player zero-sum game is thus designed where the first player is the distortion model with the aim of increasing the domain shift. The classifier then becomes the second player whose aim is to minimize the impact of domain shift. Finally, a direct error-driven learning scheme is utilized to minimize the impact of the classifier while maximizing the domain shift. A comprehensive simulation study is presented where a 12% improvement in the presence of domain shift is demonstrated. The proposed approach is also shown to improve generalization by 6%.

In this paper, the relevance of deep neural network (DNN) is studied in big data scenarios, specifically for prognostics applications. It is observed that fault predictions can be performed more efficiently when DNN is used with a pre-processing step. A novel hierarchical dimension reduction (HDR) approach is therefore proposed as a pre-processing step to DNN. This two-step approach is shown to be effective in extracting value from complex and uncertain big data. It is shown that use of HDR prior to DNN improves convergence and allows for the possibility of reduction in model size without any drop in accuracy. A comprehensive methodology is developed to facilitate prognostics using DNN. Simulation results are included to demonstrate the overall methodology using big data-sets.