Large Language Models (LLMs) have shown remarkable success in supporting a wide range of knowledge-intensive tasks. In specialized domains, there is growing interest in leveraging LLMs to assist subject matter experts with domain-specific challenges. However, deploying LLMs as SaaS solutions raises data privacy concerns, while many open-source models demand significant computational resources for effective domain adaptation and deployment. A promising alternative is to develop smaller, domain-specialized LLMs, though this approach is often constrained by the lack of high-quality domain-specific training data. In this work, we address these limitations by presenting a cost-efficient and scalable training pipeline that combines guided synthetic data generation from a small seed corpus with bottom-up domain data curation. Our pipeline integrates Domain-Adaptive Pretraining (DAPT), Domain-specific Supervised Fine-tuning (DSFT), and Direct Preference Optimization (DPO) to train effective small-scale models for specialized use cases. We demonstrate this approach through DiagnosticSLM, a 3B-parameter domain-specific model tailored for fault diagnosis, root cause analysis, and repair recommendation in industrial settings. To evaluate model performance, we introduce four domain-specific benchmarks: multiple-choice questions (DiagnosticMCQ), question answering (DiagnosticQA), sentence completion (DiagnosticComp), and summarization (DiagnosticSum). DiagnosticSLM achieves up to 25% accuracy improvement over open-source models of comparable or larger size (2B-9B) on the MCQ task, while also outperforming or matching them in other tasks, demonstrating effective domain-specific reasoning and generalization capabilities.

Time series data have grown at an explosive rate in numerous domains and have stimulated a surge of time series modeling research. A comprehensive comparison of different time series models, for a considered data analytics task, provides useful guidance on model selection for data analytics practitioners. Data scarcity is a universal issue that occurs in a vast range of data analytics problems, due to the high costs associated with collecting, generating, and labeling data as well as some data quality issues such as missing data. In this paper, we focus on the temporal classification/regression problem that attempts to build a mathematical mapping from multivariate time series inputs to a discrete class label or a real-valued response variable. For this specific problem, we identify two types of scarce data: scarce data with small samples and scarce data with sparsely and irregularly observed time series covariates. Observing that all existing works are incapable of utilizing the sparse time series inputs for proper modeling building, we propose a model called sparse functional multilayer perceptron (SFMLP) for handling the sparsity in the time series covariates. The effectiveness of the proposed SFMLP under each of the two types of data scarcity, in comparison with the conventional deep sequential learning models (e.g., Recurrent Neural Network, and Long Short-Term Memory), is investigated through mathematical arguments and numerical experiments.

Industrial AI is concerned with the application of Artificial Intelligence (AI), Machine Learning (ML) and related technologies towards addressing real-world use cases in industrial and societal domains. These uses cases can be broadly categorized into the horizontal areas of maintenance and repair, operations and supply chain, quality, safety, design, and end-to-end optimization - with applications in a variety of verticals. In the last few years, we have witnessed a growing interest in applying Deep Learning (DL) techniques to Industrial AI problems, ranging from using sequence models such as Long Short-Term Memory (LSTM) for predicting failures in equipment, to using Deep Reinforcement Learning (Deep RL) for scheduling and dispatching. Applying deep learning techniques to industrial applications imposes a set of unique challenges, which include, but are not limited to, (1) limited data, highly skewed class distribution and occurrence of rare classes such as failures, (2) multi-modal data (sensors, events, images, text, etc.) indexed over space and time (3) the need for explainable decisions, (4) a need to attain consistency between different but "related" models and between multiple generations of the same model, and (5) decision making to optimize business outcomes where the cost of a mistake could be very high. This tutorial presents an overview of these challenges, along with new methods and best practices to address them. Examples of these methods include using sequence DL models and Functional Neural Networks (FNNs) for modeling sensor and spatiotemporal measurements; using multi-task learning, graph models and ensemble learning for improving consistency of DL models; using deep RL for health indicator learning and dynamic dispatching; cost-based decision making for prognostics; and using GANs for generating senor data for prognostics. Finally, we will present some open problems in Industrial AI and how the research community can shape the future of the next industrial and societal revolution.

Deep learning classifiers are assisting humans in making decisions and hence the user's trust in these models is of paramount importance. Trust is often a function of constant behavior. From an AI model perspective it means given the same input the user would expect the same output, especially for correct outputs, or in other words consistently correct outputs. This paper studies a model behavior in the context of periodic retraining of deployed models where the outputs from successive generations of the models might not agree on the correct labels assigned to the same input. We formally define consistency and correct-consistency of a learning model. We prove that consistency and correct-consistency of an ensemble learner is not less than the average consistency and correct-consistency of individual learners and correct-consistency can be improved with a probability by combining learners with accuracy not less than the average accuracy of ensemble component learners. To validate the theory using three datasets and two state-of-the-art deep learning classifiers we also propose an efficient dynamic snapshot ensemble method and demonstrate its value. Code for our algorithm is available at https://github.com/christa60/dynens.

Monitoring and understanding the performance of equipment and operations is a real and challenging problem in industrial and commercial settings. Although many operations have a set of agreed-upon Key Performance Indicators (KPIs) that have been defined by domain experts, most of these KPIs suffer from many issues including but not limited to sensitivity to seasonality and load conditions, approximate and inaccurate calculations, and measurement noise. This makes further analysis of these measurements a real challenge. This paper focuses on answering a simple yet crucial question in regards to KPI monitoring: "Have a given event made an impact on the performance of an operation?". The paper presents an approach to answer this question in a systematic and statistically-sound way while alleviating the aforementioned problems with the provided KPIs. The key idea behind the proposed approach is to predict the would-be performance as if the event has not happened. The approach then applies a three-way hypothesis testing on the difference between the would-be and actual performance to determine whether the event resulted in improvement, degradation or no-change in performance. Results from simulation study and experiments on real use cases demonstrate the effectiveness and usability of the proposed approach.

Equipment-as-a-service is gaining popularity in industries as it provides the customers with high flexibility and scalability and allows the service provider to optimize operations so that both the service provider and the customers are benefited. However, pricing modeling is a critical challenge in EaaS due to the change of business objectives from selling products to providing services. Particularly, EaaS faces severe high-cost issues as unexpected equipment failures can result in expensive downtime costs. In this paper, we take advantage of Failure Prediction (FP) in Predictive Maintenance (PdM) domain and Dynamic Pricing to build a hybrid model that balances cost and benefit for EaaS. First, we build a Gradient Boosting based FP model to predict failure probabilities. Then a Risk-based Dynamic Pricing (RBDP) model uses the confusion matrix of FP to estimate the cost effects of predicted failures with a constrained objective function. Finally, the FP model and RBDP model are trained simultaneously to minimize the objective function so that it compensates for the failure cost and maximize the overall profit gain. The proposed hybrid modeling approach is compared with traditional FP without pricing policy approach, and FP with a static pricing policy approach to show significant improvements in profit increases.

In this paper, we propose a novel algorithm for generalizing Multilayer Perceptron (MLP) to handle sparse functional data, wherein for a given subject there are multiple observations available over time and these observations are sparsely and irregularly distributed within the considered time range. The algorithm uses pooled observations across all the subjects to estimate a set of basis functions for the underlying correlation between time steps and then use these basis functions to build a sparse functional neuron that extracts features for each subject. We justify the validity of our algorithm through theoretical arguments. We also demonstrate the use of the proposed algorithm in solving three data challenges: the classification of synthetic curves, the prediction of patient's long-term survival, and the estimation of the remaining time to critical failures for turbofan engines. To show the superiority of our algorithm under sparse functional data scenarios, we compare the performance of our model with two alternative common practices, and demonstrate that our method outperforms the baseline methods in all numerical studies.

One of the key challenges in predictive maintenance is to predict the impending downtime of an equipment with a reasonable prediction horizon so that countermeasures can be put in place. Classically, this problem has been posed in two different ways which are typically solved independently: (1) Remaining useful life (RUL) estimation as a long-term prediction task to estimate how much time is left in the useful life of the equipment and (2) Failure prediction (FP) as a short-term prediction task to assess the probability of a failure within a pre-specified time window. As these two tasks are related, performing them separately is sub-optimal and might results in inconsistent predictions for the same equipment. In order to alleviate these issues, we propose two methods: Deep Weibull model (DW-RNN) and multi-task learning (MTL-RNN). DW-RNN is able to learn the underlying failure dynamics by fitting Weibull distribution parameters using a deep neural network, learned with a survival likelihood, without training directly on each task. While DW-RNN makes an explicit assumption on the data distribution, MTL-RNN exploits the implicit relationship between the long-term RUL and short-term FP tasks to learn the underlying distribution. Additionally, both our methods can leverage the non-failed equipment data for RUL estimation. We demonstrate that our methods consistently outperform baseline RUL methods that can be used for FP while producing consistent results for RUL and FP. We also show that our methods perform at par with baselines trained on the objectives optimized for either of the two tasks.

For many industrial and commercial operations,maintenance accounts for a large part of operating costs (e.g., 15%-60% of the total production costs in manufacturing plants). Even with maintenance cost being such a substantial part of the overall cost, maintenance managers have little visibility into whether maintenance expenditure is money well spent or not. They also do not have standard quantitative methods to answer even simple questions such as, does it make any difference if one does preventive maintenance every six months on a piece of equipment, or whether a particular maintenance action is improving the performance of an equipment or not. In this paper, we formally define the problem of estimating the effectiveness of a single or a group of maintenance actions, and propose a systemic way of solving the problem. We also present a benchmark to evaluate the proposed methods and demonstrate how they can accurately identify effective maintenance actions.

The Kernel Mean Matching (KMM) is an elegant algorithm that produces density ratios between training and test data by minimizing their maximum mean discrepancy in a kernel space. The applicability of KMM to large-scale problems is however hindered by the quadratic complexity of calculating and storing the kernel matrices over training and test data. To address this problem, this paper proposes a novel ensemble algorithm for KMM, which divides test samples into smaller partitions, estimates a density ratio for each partition and then fuses these local estimates with a weighted sum. Our theoretical analysis shows that the ensemble KMM has a lower error bound than the centralized KMM, which uses all the test data at once to estimate the density ratio. Considering its suitability for distributed implementation, the proposed algorithm is also favorable in terms of time and space complexities. Experiments on benchmark datasets confirm the superiority of the proposed algorithm in terms of estimation accuracy and running time.

The kernel k-means is an effective method for data clustering which extends the commonly-used k-means algorithm to work on a similarity matrix over complex data structures. It is, however, computationally very complex as it requires the complete kernel matrix to be calculated and stored. Further, its kernelized nature hinders the parallelization of its computations on modern scalable infrastructures for distributed computing. In this paper, we are defining a family of kernelbased low-dimensional embeddings that allows for scaling kernel k-means on MapReduce via an efficient and unified parallelization strategy. Afterwards, we propose two practical methods for low-dimensional embedding that adhere to our definition of the embeddings family. Exploiting the proposed parallelization strategy, we present two scalable MapReduce algorithms for kernel k-means. We demonstrate the effectiveness and efficiency of the proposed algorithms through an empirical evaluation on benchmark datasets.

Covariate shift is a challenging problem in supervised learning that results from the discrepancy between the training and test distributions. An effective approach which recently drew a considerable attention in the research community is to reweight the training samples to minimize that discrepancy. In specific, many methods are based on developing Density-ratio (DR) estimation techniques that apply to both regression and classification problems. Although these methods work well for regression problems, their performance on classification problems is not satisfactory. This is due to a key observation that these methods focus on matching the sample marginal distributions without paying attention to preserving the separation between classes in the reweighted space. In this paper, we propose a novel method for Discriminative Density-ratio (DDR) estimation that addresses the aforementioned problem and aims at estimating the density ratio of joint distributions in a class-wise manner. The proposed algorithm is an iterative procedure that alternates between estimating the class information for the test data and estimating a new density ratio for each class. To incorporate the estimated class information of the test data, a soft matching technique is proposed. In addition, we employ an effective criterion which adopts mutual information as an indicator to stop the iterative procedure while resulting in a decision boundary that lies in a sparse region. Experiments on synthetic and benchmark datasets demonstrate the superiority of the proposed method.

Given a very large data set distributed over a cluster of several nodes, this paper addresses the problem of selecting a few data instances that best represent the entire data set. The solution to this problem is of a crucial importance in the big data era as it enables data analysts to understand the insights of the data and explore its hidden structure. The selected instances can also be used for data preprocessing tasks such as learning a low-dimensional embedding of the data points or computing a low-rank approximation of the corresponding matrix. The paper first formulates the problem as the selection of a few representative columns from a matrix whose columns are massively distributed, and it then proposes a MapReduce algorithm for selecting those representatives. The algorithm first learns a concise representation of all columns using random projection, and it then solves a generalized column subset selection problem at each machine in which a subset of columns are selected from the sub-matrix on that machine such that the reconstruction error of the concise representation is minimized. The paper then demonstrates the effectiveness and efficiency of the proposed algorithm through an empirical evaluation on benchmark data sets.

In data mining applications, data instances are typically described by a huge number of features. Most of these features are irrelevant or redundant, which negatively affects the efficiency and effectiveness of different learning algorithms. The selection of relevant features is a crucial task which can be used to allow a better understanding of data or improve the performance of other learning tasks. Although the selection of relevant features has been extensively studied in supervised learning, feature selection with the absence of class labels is still a challenging task. This paper proposes a novel method for unsupervised feature selection, which efficiently selects features in a greedy manner. The paper first defines an effective criterion for unsupervised feature selection which measures the reconstruction error of the data matrix based on the selected subset of features. The paper then presents a novel algorithm for greedily minimizing the reconstruction error based on the features selected so far. The greedy algorithm is based on an efficient recursive formula for calculating the reconstruction error. Experiments on real data sets demonstrate the effectiveness of the proposed algorithm in comparison to the state-of-the-art methods for unsupervised feature selection.