Today’s highly instrumented systems generate large amounts of time series data from many different domains. In order to create meaningful insights from these data, techniques are needed to handle the collection, processing, and analysis at scale. The high frequency and volume of data that is generated introduces several challenges including data transformation, managing concept drift, the operational cost of model re-training and tracking, and scaling hyperparameter optimization.Incremental machine learning can provide a viable solution to handle these kinds of data. Further, distributed machine learning can be an efficient technique to improve performance, increase accuracy, and scale to larger input sizes.In this paper, we introduce a framework that combines the computational capabilities of Apache Spark and the workflow parallelization of Ray for distributed incremental learning. We conduct an empirical analysis of our framework for time series forecasting using the Walmart M5 dataset. The system can perform a parameter search on streaming data with concept drift producing a robust pipeline that fits high-volume data effectively. The results are encouraging and substantiate system proficiency over traditional big data analysis approaches that exclusively use either offline or online training.

With the lowering costs of sensors, high-volume and high-velocity data are increasingly being generated and analyzed, especially in IoT domains like energy and smart homes. Consequently, applications that require accurate short-term forecasts and predictions are also steadily increasing. In this paper, we provide an overview of a novel end-to-end platform that provides efficient ingestion, compression, transfer, query processing, and machine learning-based analytics for high-frequency and high-volume time series from IoT. The performance of the platform is evaluated using real-world dataset from RES installations. The results show the importance of high-frequency analytics and the surprisingly positive impact of error bounded lossy compression on machine learning in the form of AutoML. For example, when detecting yaw misalignments in wind turbines, an improvement of 9% in accuracy was observed for AutoML models on lossy compressed data compared to the current industry standard of 10-minute aggregated data. Thus, these small-scale experiments show the potential of the platform, and larger pilots are planned.

Over the last few years, incremental machine learning for streaming data has gained significant attention due to the need to learn from a constantly evolving stream of data without the need to store it. The advent of big data has further fuelled research on developing systems that can cope with continuously changing data streams and tackle the challenges associated with historical data requirements. The problem of time series forecasting has been studied using varied approaches like neural networks, ensemble methods, decision trees and rules, support vector machines to name a few. However, neural network models have gained particular attention in dealing with changing data distribution due to their generalization abilities. In this paper, we propose a prequential framework named PS-PGSM which involves incrementally training the base models and online Regions of Competence (ROC) computation followed by selection of the best forecaster for the task of time series forecasting using saliency maps. We build upon the state-of-the-art approach named OS-PGSM (Online Model Selection using Performance Gradient based Saliency Maps) in which the model training and ROC computation is performed offline. Past research has demonstrated that a set of different models enables specialization for each model compared to a single forecasting model which is particularly useful when predicting for an evolving time series sequence. Our approach uses saliency maps for prequential calculation of ROC for each model to find the best forecaster based on the performance of each model. We evaluate the proposed approach against OS-PGSM, as well as against previous best performing model by first conducting preliminary experiments on 10 real-world time series datasets and then using 2 real-world big datasets to showcase its applicability to big data. Experimental results not only validate the effectiveness of our approach for big data but also demonstrate superior performance in terms of prediction accuracy and computational time efficiency while also handling concept drift.

Advances in Deep Neural Network (DNN) techniques have revolutionized video analytics and unlocked the potential for querying and mining video event patterns. This paper details GNOSIS, an event processing platform to perform near-real-time video event detection in a distributed setting. GNOSIS follows a serverless approach where its component acts as independent microservices and can be deployed at multiple nodes. GNOSIS uses a declarative query-driven approach where users can write customize queries for spatiotemporal video event reasoning. The system converts the incoming video streams into a continuous evolving graph stream using machine learning (ML) and DNN models pipeline and applies graph matching for video event pattern detection. GNOSIS can perform both stateful and stateless video event matching. To improve Quality of Service (QoS), recent work in GNOSIS incorporates optimization techniques like adaptive scheduling, energy efficiency, and content-driven windows. This paper demonstrates the Occupational Health and Safety query use cases to show the GNOSIS efficacy.