Selecting the right set of hyperparameters is crucial in time series forecasting. The classical temporal cross-validation framework for hyperparameter optimization (HPO) often leads to poor test performance because of a possible mismatch between validation and test periods. To address this test-validation mismatch, we propose a novel technique, H-Pro to drive HPO via test proxies by exploiting data hierarchies often associated with time series datasets. Since higher-level aggregated time series often show less irregularity and better predictability as compared to the lowest-level time series which can be sparse and intermittent, we optimize the hyperparameters of the lowest-level base-forecaster by leveraging the proxy forecasts for the test period generated from the forecasters at higher levels. H-Pro can be applied on any off-the-shelf machine learning model to perform HPO. We validate the efficacy of our technique with extensive empirical evaluation on five publicly available hierarchical forecasting datasets. Our approach outperforms existing state-of-the-art methods in Tourism, Wiki, and Traffic datasets, and achieves competitive result in Tourism-L dataset, without any model-specific enhancements. Moreover, our method outperforms the winning method of the M5 forecast accuracy competition.

Today’s highly instrumented systems generate large amounts of time series data from many different domains. In order to create meaningful insights from these data, techniques are needed to handle the collection, processing, and analysis at scale. The high frequency and volume of data that is generated introduces several challenges including data transformation, managing concept drift, the operational cost of model re-training and tracking, and scaling hyperparameter optimization.Incremental machine learning can provide a viable solution to handle these kinds of data. Further, distributed machine learning can be an efficient technique to improve performance, increase accuracy, and scale to larger input sizes.In this paper, we introduce a framework that combines the computational capabilities of Apache Spark and the workflow parallelization of Ray for distributed incremental learning. We conduct an empirical analysis of our framework for time series forecasting using the Walmart M5 dataset. The system can perform a parameter search on streaming data with concept drift producing a robust pipeline that fits high-volume data effectively. The results are encouraging and substantiate system proficiency over traditional big data analysis approaches that exclusively use either offline or online training.

In this paper, we describe an overarching ML system with a simple programming interface that leverages existing AI and ML frameworks to make the task of model exploration easier. The proposed system introduces a new programming construct namely pipeline graph (a directed acyclic graph) consisting of multiple machine learning operations provided by different ML repositories. End user uses the pipeline graph as a common interface for modeling different ML tasks such as classification, regression, and timeseries prediction, while enabling efficient execution on different environments (Spark, Celery and Cloud). We further annotated the pipeline graph with a hyper-parameter grid and an option to try-out a wide range of optimization strategies (i.e., Random, Bayesian, Bandit, AutoLearn, etc). Given a large pre-defined pipeline graph along with its hyper-parameters, we provided a general-purpose, scalable and efficient pipeline-graph exploration technique to provide the automated solutions to a variety of ML tasks. We compare our automated approach to several state-of-the-art automated AI systems and find that we achieve performance comparable to the best results, while often producing simpler pipelines using off the shelf components. Our evaluation suite consists of experiments on 60+ classifications and regressions datasets.

Fueled with growth in the fields of Internet of Things (IoT) and Big Data, data has become one of the most valuable assets in today's world. While we are leveraging this data for analyzing complex systems using machine learning and deep learning, a considerable amount of time and effort is spent on addressing data quality issues. If undetected, data quality issues can cause large deviations in the analysis, misleading data scientists. To ease the effort of identifying and addressing data quality challenges, we introduce DQA, a scalable, automated and interactive data quality advisor. In this paper, we describe the DQA framework, provide detailed description of its components and the benefits of integrating it in a data science process. We propose a programmatic approach for implementing the data quality framework which automatically generates dynamic executable graphs for performing data validations fine-tuned for a given dataset. We discuss the use of DQA to build a library of validation checks common to many applications. We provide insight into how DQA addresses many persistence and usability issues which currently make data cleaning a laborious task for data scientists. Finally, we provide a case study of how DQA is implemented in a realworld system and describe the benefits realized.