Abstract This work addresses the challenges of learning from Evolving Streaming Time Series, a scenario defined in streaming contexts by the need for continuous learning, managing temporal dependence, reacting to concept drifts, and avoiding catastrophic forgetting. Streaming Continual Learning (SCL) is an emerging paradigm specifically aimed at tackling these issues. To this end, SCL combines elements of Streaming Machine Learning, Continual Learning, and Time Series Analysis. The core idea of SCL is to maintain a progressively enriched representation that reflects everything learned so far. When a drift occurs, the model integrates newly emerging information and also selectively draws on earlier knowledge. In this context, the present work proposes Dynamic Continuous Progressive Neural Networks (DYNcPNN), a comprehensive and pioneering SCL embodiment built on our previously introduced cPNN architecture. DYNcPNN is a dynamic variant designed to adapt more effectively to concept drifts. Its key contribution is a mechanism that dynamically determines when to expand the architecture to incorporate new knowledge, reducing unnecessary complexity while preserving high performance. DYNcPNN also introduces a strategy to prevent forgetting when, after a concept drift, the model continues learning without expanding, thereby risking overwriting the existing knowledge. The model integrates an automatic concept drift detection system that enables adaptation without manual intervention. Experimental results show that DYNcPNN consistently outperforms traditional SML models, cPNN, and a continuously trained LSTM. It adapts more quickly to concept drifts, effectively mitigates catastrophic forgetting, and optimizes memory usage. Moreover, the results highlight the limitations of SML models, which are unable to account for temporal dependence.

Concept drift, temporal dependence, and catastrophic forgetting represent major challenges when learning from data streams. While Streaming Machine Learning and Continual Learning (CL) address these issues separately, recent efforts in Streaming Continual Learning (SCL) aim to unify them. In this work, we introduce MAGIC Net, a novel SCL approach that integrates CL-inspired architectural strategies with recurrent neural networks to tame temporal dependence. MAGIC Net continuously learns, looks back at past knowledge by applying learnable masks over frozen weights, and expands its architecture when necessary. It performs all operations online, ensuring inference availability at all times. Experiments on synthetic and real-world streams show that it improves adaptation to new concepts, limits memory usage, and mitigates forgetting.

Internet of Things (IoT) Analytics often involves applying machine learning (ML) models on data streams. In such scenarios, traditional ML paradigms face obstacles related to continuous learning while dealing with concept drifts, temporal dependence, and avoiding forgetting. Moreover, in IoT, different edge devices build up a network. When learning models on those devices, connecting them could be useful in improving performance and reusing others’ knowledge. This work proposes Mutual Assisted Learning, a learning paradigm grounded on Vygotsky’s popular Sociocultural Theory of Cognitive Development. Each device is autonomous and does not need a central orchestrator. Whenever it degrades its performance due to a concept drift, it asks for assistance from others and decides whether their knowledge is useful for solving the new problem. This way, the number of connections is drastically reduced compared to the classical Federated Learning approaches, where the devices communicate at each training round. Every device is equipped with a Continuous Progressive Neural Network (cPNN) to handle the dynamic nature of data streams. We call this implementation Mutual Assisted cPNN (MAcPNN). To implement it, we allow cPNNs for single data point predictions and apply quantization to reduce the memory footprint. Experimental results prove the effectiveness of MAcPNN in boosting performance on synthetic and real data streams.

In batch learning, it is commonly assumed that samples are independent and identically distributed (i.i.d.). However, this assumption does not hold in dynamic environments where data streams are not identically distributed due to concept drifts. Furthermore, while most Streaming Machine Learning (SML) literature assumes independence among examples, data streams often have important temporal components that learning should adequately consider. Neglecting this temporal dependence can lead to significant misguidance in designing and evaluating SML models. To support our thesis, we propose Tenet, a novel benchmarking framework designed to evaluate data stream classifiers in non-i.i.d. scenarios comparatively. Tenet consists of a data stream generator and a baseline. The data stream generator introduces temporal dependence into the data streams commonly used for evaluating SML algorithms. The baseline is a continuous version of the Long Short-Term Memory algorithm called cLSTM. Extensive experiments using Tenet demonstrate that cLSTM consistently outperforms state-of-the-art SML classifiers when learning from data streams with temporal dependence. This result is a call to action for the SML and the Deep Learning communities to investigate classifiers in the time-dependent streaming scenario and makes Tenet the first publicly available benchmark to support this research.

International audience

Continuous data generation over time presents new challenges for Machine Learning systems, which must develop real-time models due to memory and latency limitations. Streaming Machine Learning algorithms analyze data streams one sample at a time, progressively updating their models. However, is it necessary to utilize all the data for model updates? This paper introduces the Online Ensemble SPaced Learning (OE-SPL) strategy, an ensemble meta-strategy that combines online ensemble learning and the Spaced Learning heuristic to rapidly learn underlying concepts without using all samples. We evaluated OE-SPL on synthetic and real data streams containing various concept drifts, providing statistical evidence that OE-SPL achieves comparable performance to state-of-the-art ensemble models while recovering from multiple concept drift occurrences more efficiently, using less time and RAM-Hours.

Sports analytics has grown significantly through continuous data streams from wearable tracking devices. This article explores the performance of traditional Machine Learning methods (ML) compared to Streaming Machine Learning approaches (SML) in the context of sports analytics, specifically in identifying passes during a soccer match. The study utilizes leg movement data from wearable sensors on players’ shoes to differentiate between pass and nonpass actions. Balanced, imbalanced and rebalanced datasets are created and analyzed. Several traditional and streaming algorithms are tested, and statistical analyses are performed to assess their performance. The findings indicate that Streaming Machine Learning can achieve comparable or better performance than traditional methods, especially on larger rebalanced datasets. This research highlights the potential of Streaming Machine Learning for online sports analytics and suggests future directions for exploring player tendencies and edge device adaptability.

Abstract Dealing with an unbounded data stream involves overcoming the assumption that data is identically distributed and independent. A data stream can, in fact, exhibit temporal dependencies (i.e., be a time series), and data can change distribution over time (concept drift). The two problems are deeply discussed, and existing solutions address them separately: a joint solution is absent. In addition, learning multiple concepts implies remembering the past (a.k.a. avoiding catastrophic forgetting in Neural Networks’ terminology). This work proposes Continuous Progressive Neural Networks (cPNN), a solution that tames concept drifts, handles temporal dependencies, and bypasses catastrophic forgetting. cPNN is a continuous version of Progressive Neural Networks, a methodology for remembering old concepts and transferring past knowledge to fit the new concepts quickly. We base our method on Recurrent Neural Networks and exploit the Stochastic Gradient Descent applied to data streams with temporal dependencies. Results of an ablation study show a quick adaptation of cPNN to new concepts and robustness to drifts.

The world is constantly changing, and so are the massive amount of data produced. However, only a few studies deal with online class imbalance learning that combines the challenges of class-imbalanced data streams and concept drift. In this paper, we propose the Synthetic Minority Oversampling TEchnique with Online Bagging (SMOTE-OB). It is a novel cost-sensitive ensemble strategy that uses Online Bagging and a new sketched version of SMOTE to over/undersample the minority and majority classes. We benchmarked SMOTE-OB on synthetic and real data streams containing different concept drifts, imbalance levels, and class distributions. We bring statistical evidence that the SMOTE-OB ensemble achieves minority class performance that are better than the state-of-the-art ones. Moreover, we perform a time/memory consumption analysis.

The continuous search for the k best results given a query (top-k) in a data stream is a problem that has gained a lot of attention in recent years, especially in social media and IoT contexts. In these contexts it is essential that the system is able to respond reactively. Therefore, in this study we address how to scale the monitoring of approximate continuous top-k queries to guarantee the reactiveness of the system.In this paper we introduce a scalable distributed version of the MinTopK+N algorithm that allows the system to remain reactive when the load increases. In addition, we present a scale- out/in algorithm that allows the system to add instances to cope with workload increments and remove instances to avoid over- provisioning. Furthermore, in a set of managed experiments, we demonstrated how the increase in the workload could affect the system performance and the impact of the instant of triggering scaling actions on the system performance. Finally, we present a study on the impact of the scaling action on the correctness of the result, showing that the scaling algorithm does not negatively impact the correctness of the results, and in certain conditions, it can even improve the quality of the approximation.

Processing data streams gained much importance in recent years. Standard machine learning algorithms do not cope well with non-stationary streaming data, where decision models evolve and generate so-called concept drift. Online adaptive algorithms emerged to solve these issues. They learn incrementally and generally require explicit forgetting mechanisms to adapt to concept drift. In this paper, we propose the application of Kalman filtering to handle evolving data streams. This novel approach addresses data stream mining and concept drift management challenges from a new perspective, directly modelling a representation suitable for the data streams. First, we study a Kalman filter based learning a pproach and investigate its integration into the Naïve Bayes algorithm, namely KalmanNB. Additionally, we propose the Hoeffding Kalman Tree, a combination of the Hoeffding Tree with KalmanNB. Empirical results demonstrate that the Kalman filter based approach inherently manages concept drifts, and it adapts to the emerging concept more rapidly than the state-of-the-art algorithms. Moreover, it is an accurate and robust approach and requires less storage while still being faster.

Abstract The world is constantly changing, and so are the massive amount of data produced. However, only a few studies deal with online class imbalance learning that combines the challenges of class-imbalanced data streams and concept drift. In this paper, we propose the very fast continuous synthetic minority oversampling technique ( VFC - SMOTE ). It is a novel meta-strategy to be prepended to any streaming machine learning classification algorithm aiming at oversampling the minority class using a new version of Smote and Borderline - Smote inspired by Data Sketching. We benchmarked VFC - SMOTE pipelines on synthetic and real data streams containing different concept drifts, imbalance levels, and class distributions. We bring statistical evidence that VFC - SMOTE pipelines learn models whose minority class performances are better than state-of-the-art. Moreover, we analyze the time/memory consumption and the concept drift recovery speed.

Streaming Machine Learning (SML) studies single-pass learning algorithms that update their models one data item at a time given an unbounded and often non-stationary flow of data (a.k.a., in presence of concept drift). Online class imbalance learning is a branch of SML that combines the challenges of both class imbalance and concept drift. In this paper, we investigate the binary classification problem of rebalancing an imbalanced stream of data in the presence of concept drift, accessing one sample at a time. We propose Continuous Synthetic Minority Oversampling Technique (C-SMOTE), a novel rebalancing meta-strategy to pipeline with SML classification algorithms. C-SMOTE is inspired by the popular SMOTE algorithm but operates continuously. We benchmark C-SMOTE pipelines on ten different groups of data streams. We bring empirical evidence that models learnt with C-SMOTE pipelines outperform models trained on imbalanced data stream without losing the ability to deal with concept drifts. Moreover, we show that they outperform other stream balancing techniques from the literature.

Streaming Analytics (SA) and Complex Event Recognition (CER) are of paramount importance in searching for an ultimate Big Data solution that can simultaneously address Data Velocity, Variety, and Volume. Indeed, the growing popularity of streaming data has pushed the boundaries of existing data systems, fostering the rise of Stream Processing Engines (SPE). However, Data Velocity never appears isolated. Streams are huge, heterogeneous, and noisy as they come from multiple sources. Horizontally-scalable SPEs like Flink and KSQL-DB allow continuous stream analytics using SQL-like languages. On the other hand, CER engines like OracleCEP and DroolFusion use regular languages for (parallel) pattern detection over heterogeneous streams. This paper takes a first step towards a unifying solution. To this extent, we present KELPr, an in-memory distributed CER engine designed extending the Dual Streaming Model and implemented on top of Kafka Streams.

Anomaly detection is valuable in many domains. Distance-based algorithms are one of the most well-known techniques for anomaly detection. These algorithms have been shown to produce good accuracy in a wide range of applications. However, distance-based algorithms are limited in terms of scalability. This makes it difficult for these algorithms to handle massive data sets. To overcome this limitation, in this paper we propose a parallel formulation of the popular KNN distance-based algorithm for unsupervised anomaly detection. Our formulation focused on improving scalability by analyzing in parallel different parts of the data. This approach is valuable whenever the analyzed data does not fit comfortably into memory, or when it is distributed across several locations. To reduce synchronization latency, the proposed algorithm computes approximate nearest neighborhoods. We show, both theoretically and empirically, how this approach greatly improves scalability without significantly penalizing detection accuracy. To validate our approach, we perform extensive experiments on both synthetic and real-world data sets.

Massive data integration technologies have been recently used to produce very large ontologies. However, knowledge in the world continuously evolves, and ontologies are largely incomplete for what concerns low-frequency data, belonging to the so-called long tail. Socially produced content is an excellent source for discovering emerging knowledge: it is huge, and immediately reflects the relevant changes which hide emerging entities. Thus, we propose a method for discovering emerging entities by extracting them from social content.

RDF and SPARQL are established standards for data interchange and querying on the Web. While they have been shown to be useful and applicable in many scenarios, they are not sufficiently adequate for dealing with streams of data and their intrinsic continuous nature. In the last years data and query languages have been proposed to extend both RDF and SPARQL for streams and continuous processing, under the name of RDF Stream Processing – RSP. These efforts resulted in several models and implementations that, at a first look, appear to propose alternative syntaxes but equivalent semantics. However, when asked to continuously answer the same queries on the same data streams, they provide different answers at disparate moments due to the heterogeneity of their operational semantics. These discrepancies render the process of understanding and comparing continuous query results complex and misleading. In this work, the authors propose RSP-QL, a comprehensive model that formally defines the semantics of an RSP system. RSP-QL makes explicit the hidden assumptions of currently available RSP systems, allows defining a formal notion of correctness for RSP query results and, thus, explains why available implementations provide different answers at disparate moments.

C-SPARQL is an extension of SPARQL to support continuous queries, registered and continuously executed over RDF data streams, considering windows of such streams. Supporting streams in RDF format guarantees interoperability and opens up important applications, in which reasoners can deal with knowledge that evolves over time. We present C-SPARQL by means of examples in Urban Computing.