Continuously learning and serving from evolving streaming data and serving in real-time is a challenging problem. Traditionally, data is partitioned and processed in batches to train machine learning (ML) models. In industrial applications, static models’ performance drops over time (model degradation, concept drift), requiring new models to be trained with recent data and redeployed in production. The scientific community has been studying online and adaptive methods to address batch-learning limitations and continuously train AI tasks for industrial applications such as cyber-security, AIOps, anomaly scoring, and drift detection in stock markets. This paper deals with the MLOps aspects of deploying such online and dynamic models to address the requirements in the production systems for real-time applications. Our architectures - based on open-source tools such as Kafka and River - demonstrated how online learning methods could be scaled horizontally in production to meet the demands of a high-velocity streaming pipeline. We demonstrate an MLOps strategy to perform incremental learning from streaming data and continuously deploy the online learning model without pausing the inference pipeline. Indeed, the design satisfies requirements such as model versioning, monitoring, audibility and reproducibility of prediction in both a supervised and semi-supervised setting. Our experiments - for malicious URLs detection task - performed on high-dimensional and feature-evolving streaming data (more than 3 million features) establish the effectiveness and efficiency of online learning models compared to batch (static) machine learning regarding both time and space complexity. Finally, we provide some best practices on data engineering for deploying online models to process a real-time feature stream in production environments. Code is publicly available for reproducibility.

Knowledge Graphs (KG) are valuable information sources that store knowledge in a domain (healthcare, finance, e-commerce, cyber-security.). Most industrial KGs are dynamic by nature as they are updated regularly with streaming data (customer activity, network traffic, application logs, IT process). However, extracting insights from continuously updated data comes with major challenges, particularly in big data settings. In this paper, we address the following challenges: 1) ingesting heterogeneous data, 2) training and deployment of predictive models on continuously evolving data, and 3) implementation of data pipelines for updating and maintaining the KG in production. We cover multiple aspects of this process, from knowledge collection to its operationalization. We propose Stream2Graph, a stream-based system for building and updating the knowledge base dynamically in real time. Then we show how graph features can be used in downstream online machine learning models. The solution speeds up big data stream learning and knowledge extraction to enhance Graph-based AI applications. Experimental results show the effectiveness of our solution for knowledge base construction and improvement of big data learning capabilities. Using data from Stream2Graph resulted in speedups for training and inference time in the range from 547x to 2000x in downstream ML models. Finally, we provide the lessons learned from applying graph-based online learning on large-scale network processing high-velocity streaming data.

The growing need for predictive analytics over streaming data in the industry requires a flexible and continuously scalable big data system. In real-time big data applications (cybersecurity, AIOps, anomaly detection, predictive maintenance, IoT etc.), efficient machine learning models must be trained and industrialized within existing data processing plat-forms and industrial tools. This requires interoperability between various components: data collection, processing, summarization, modelling and analytics. Existing works focus on building AI models for big data, neglecting real-world challenges when integrating such models into an existing industrial production framework. In this paper, we propose StreamFlow, an operational data pipeline to address industrial challenges for continuous learning over big data streams. We also propose an online method using sliding windows to summarize high-velocity data. The final result of the framework is a feature vector that describes the underlying processes and is ready to use in machine learning tasks. Moreover, we showcase real-world applications such as automated feature engineering for real-time monitoring and online machine learning for event classification. The proposed system has been deployed within production in a banking system, processing billions of daily traffic operations. Our experiments demonstrate the effectiveness and performance of our approach by evaluating it at different levels: processing, summarization, improvement of machine learning performance and effectiveness in an industrial setting. In the case of downstream machine learning tasks, using summarized data generated by StreamFlow results in up to 2 orders of magnitude speedups in training time without compromising predictive performance.