Graphs model latent variable relationships in many real-world systems, and Message Passing Neural Networks (MPNNs) are widely used to learn such structures for downstream tasks. While edge-based MPNNs effectively capture local interactions, their expressive power is theoretically bounded, limiting the discovery of higher-order relationships. We introduce the Higher-Order Graph Attention (HoGA) module, which constructs a k-order attention matrix by sampling subgraphs to maximize diversity among feature vectors. Unlike existing higher-order attention methods that greedily resample similar k-order relationships, HoGA targets diverse modalities in higher-order topology, reducing redundancy and expanding the range of captured substructures. Applied to two single-hop attention models, HoGA achieves at least a 5% accuracy gain on all benchmark node classification datasets and outperforms recent baselines on six of eight datasets. Code is available at https://github.com/TB862/Higher_Order.

Abstract Anomaly detection in live trajectory data is a critical task for ensuring safety, security, and legality in global transport. Traditional anomaly detection methods often struggle with dynamic and evolving trajectory patterns, especially as systems must adapt to new scenarios over time due to increased traffic, geopolitical events, or global warming. We propose a continual learning approach to detect anomalous activity in moving vessels. Unlike conventional static models, our method leverages continual learning to enable the model to learn from new data continuously and recognise specific behaviours dependent on position and recent movements. We implement an adapter-based framework, Continual Learning for AIS Anomalies (CLAISA), that adapts to shifting behavioural environments in transportation, ensuring the system can identify novel and evolving patterns of anomalies, such as deviations from expected routes, irregular speed changes, or unusual local movements. Evaluations on synthetic maritime trajectory datasets spanning sparsely populated waters and heavily trafficked shipping lanes demonstrate that CLAISA achieves up to a $$32\%$$ decrease in error for trajectory forecasting and consistently outperforms benchmark methods in anomaly detection on synthetically generated datasets.

Regular school attendance is critical for young people, supporting academic achievement, social development, and the cultivation of lifelong habits. Existing research for analysing attendance patterns often relies on structured survey data targeted at their parents and teachers, which overlooks students' perspectives and experiences. To address this gap, our team developed and deployed the Our Journey platform, which enables young people to share their experiences through multimodal responses such as texts and images, offering unique insights into the factors influencing school attendance. The data is linked to official attendance records from the Ministry of Education, allowing the modelling of attendance outcomes based on students' input. To effectively analyse the data, we propose Thematic Bottleneck Models (TBMs) to enhance the understanding of subjective experiences behind data and the interpretability of attendance modelling. TBMs introduce qualitative concepts as intermediate labels, mapping multimodal data to qualitative insights from thematic analysis before the outcomes. The attendance modelling with TBMs outperforms existing multimodal methods in predicting attendance percentage and persistent absenteeism. Analysis of themes within TBMs reveals motivational and contextual factors associated with regular attendance and persistent absenteeism. The findings are used to inform education policy and guide strategies to support student engagement in New Zealand.

Thematic analysis (TA) is a widely used qualitative method for identifying underlying meanings within unstructured text. However, TA requires manual processes, which become increasingly labour-intensive and time-consuming as datasets grow. While large language models (LLMs) have been introduced to assist with TA on small-scale datasets, three key limitations hinder their effectiveness. First, current approaches often depend on interactions between an LLM agent and a human coder, a process that becomes challenging with larger datasets. Second, with feedback from the human coder, the LLM tends to mirror the human coder, which provides a narrower viewpoint of the data. Third, existing methods follow a sequential process, where codes are generated for individual samples without recalling previous codes and associated data, reducing the ability to analyse data holistically. To address these limitations, we propose Thematic-LM, an LLM-based multi-agent system for large-scale computational thematic analysis. Thematic-LM assigns specialised tasks to each agent, such as coding, aggregating codes, and maintaining and updating the codebook. We assign coder agents different identity perspectives to simulate the subjective nature of TA, fostering a more diverse interpretation of the data. We applied Thematic-LM to the Dreaddit dataset and the Reddit climate change dataset to analyse themes related to social media stress and online opinions on climate change. We evaluate the resulting themes based on trustworthiness principles in qualitative research. Our study reveals insights such as assigning different identities to coder agents promotes divergence in codes and themes.

Continual learning closely emulates human learning, which allows a model to learn from a stream of tasks sequentially without forgetting previously learned knowledge. Replay-based continual learning methods mitigate forgetting and improve performance by reintroducing data belonging to old tasks, however a replay method's performance may deteriorate when the reintroduced data does not effectively represent all experienced data. To address this concern, we propose the Sketch-based Replay Projection (SRP) method to capture and retain the original data stream's distribution within stored memory. SRP augments existing replay frameworks and introduces a two-fold approach. First, we develop a sketch-based sample selection technique to approximate feature distributions within distinct tasks, thereby capturing a wide distribution of examples for subsequent replay. Second, we propose a data compression method which projects examples into a reduced-dimensional space while preserving inter-example relationships and emphasizing inter-class disparities, encouraging diverse representations of each class while maintaining memory requirements similar to existing replay methodologies. Our experimental results demonstrate that SRP enhances replay diversity and improves the performance of existing replay models.

In the face of rising surface temperatures from climate change, impacting biodiversity, extreme weather events, and agricultural productivity, understanding the drivers behind temperature changes is imperative. Traditional global climate models (GCMs) are computationally expensive, limiting their applicability, while machine learning approaches, though promising, face interpretability challenges due to their "black box" nature, especially in a dynamic setting where the data is constantly evolving. We propose DUO, a framework to identify shifts in important features and feature combinations as the data distribution changes over time. Our model independently assesses the importance of features and their interactions while also evaluating their relevance when combined with additional features, contributing to the target class. As a case study, we apply DUO to assess the shifts in climate drivers for station-level temperatures in six locations across New Zealand from 1980 to 2020, we identify specific humidity, geopotential height, and air temperature at high atmospheric pressure levels as the most important features for describing temperature variability. By revealing how climate drivers change over time, DUO contributes to a deeper understanding of temperature change patterns, enabling practitioners to develop targeted and adaptive mitigation strategies.

A growing number of tasks require adaptive machine learning systems capable of learning continuously from incoming data and adapting to changes in their environment. In order to enable the widespread adoption of machine learning for streaming data, it is crucial that practitioners and researchers have the tools to efficiently build and evaluate adaptive learning systems. In this paper we demonstrate FALL, a Framework for Adaptive Life-long Learning, which we have developed to enable the full adaptive learning pipeline to be built using modular, reusable components, enabling users to easily and efficiently develop, implement, and evaluate state-of-the-art adaptive learning systems. Source code, documentation, and examples may be found at https://benhalstead.dev/FALL/.

Learning from streaming data is challenging as the distribution of incoming data may change over time, a phenomenon known as concept drift. The predictive patterns, or experience learned under one distribution may become irrelevant as conditions change under concept drift, but may become relevant once again when conditions reoccur. Adaptive learning methods adapt a classifier to concept drift by identifying which distribution, or concept , is currently present in order to determine which experience is relevant. Identifying a concept requires some representation to be stored for comparison, with the quality of the representation being key to accurate identification. Existing concept representations are based on meta-features, efficient univariate summaries of a concept. However, no single meta-feature can fully represent a concept, leading to severe accuracy loss when existing representations cannot describe concept drift. To avoid these failure cases, we propose the first general framework for combining a diverse range of meta-features into a single representation. We solve two main challenges, first presenting a method of efficiently computing, storing, and querying an arbitrary set of meta-features as a single representation, showing that a combination of meta-features may successfully avoid failure cases seen with existing methods. Second, we present the first method for dynamically learning which meta-features distinguish concepts in any given dataset, significantly improving performance. Our proposed approach enables state-of-the-art feature selection methods, such as mutual information, to be applied to concept representation meta-features for the first time. We investigate tradeoffs between memory budget and classification performance, observing accuracy increases of up to 16% by dynamically weighting the contribution of each meta-feature.

Pollution from wood burners has profound health implications for the general population. Typically, monitoring the level of airborne particulate matter, PM2.5, in these areas often requires making inferences about missing or corrupted readings. Air Quality inference in these cases often poses critical challenges. The factors can evolve over time, changing the distribution of data. Such changes in the distribution of data are known as concept drift. Moreover, air pollution inference for a location typically would require historical data to be collected for the location. We investigate five air quality studies in New Zealand rural towns. We explore two different research problems: (1) an adaptive recurrent drift algorithm to model recurrence patterns in PM2.5levels for a town with the ability to recover after accuracy deterioration after a concept drift using an adaptive recurrent drift algorithm, and (2) transfer learning for the data stream whereby we reuse a pre-trained air pollution inference model from a town as the starting point for an air pollution inference model on another town. We further investigate the relationship between the changes we detected and changes within the prediction horizon. We showed that the average accuracy of the air quality inference for the five towns is between 70% and 94% using the recurrent drift algorithm. We also show that transfer learning was advantageous between two of the five towns.

The distribution of streaming data often changes over time as conditions change, a phenomenon known as concept drift. Only a subset of previous experience, collected in similar conditions, is relevant to learning an accurate classifier for current data. Learning from irrelevant experience describing a different concept can degrade performance. A system learning from streaming data must identify which recent experience is irrelevant when conditions change and which past experience is relevant when concepts reoccur, e.g., when weather events or financial patterns repeat. Existing streaming approaches either do not consider experience to change in relevance over time and thus cannot handle concept drift, or only consider the recency of experience and thus cannot handle recurring concepts, or only sparsely evaluate relevance and thus fail when concept drift is missed. To enable learning in changing conditions, we propose SELeCT, a probabilistic method for continuously evaluating the relevance of past experience. SELeCT maintains a distinct internal state for each concept, representing relevant experience with a unique classifier. We propose a Bayesian algorithm for estimating state relevance, combining the likelihood of drawing recent observations from a given state with a transition pattern prior based on the system’s current state. The current state is continuously maintained using a Hoeffding bound based algorithm, which unlike existing methods, guarantees that every observation is classified using the state estimated as the most relevant, while also maintaining temporal stability. We find SELeCT is able to choose experience relevant to ground truth concepts with recall and precision above 0.9, significantly outperforming existing methods and close to a theoretical optimum, leading to significantly higher accuracy and enabling new opportunities for learning in complex changing conditions.

In the online data stream environment, a model is typically not effective until a sufficient number of data instances have been seen. One solution to alleviate this issue is using model transfer. A major gap in current research is that they do not address the cost-effectiveness of model transfer in the online context, where processing time is crucial as data instances continuously arrive at high-speed. Model transfer in data streams involves continuously adapting the model to new data in the target stream. Suppose the target stream contains hard-to-learn patterns not covered by the transferred model. In that case, adaptation may incur extra processing time with little accuracy performance gains compared to simply building a new model for the target stream from scratch. Therefore, considering the cost of model transfer and target model construction is essential to balance the accuracy performance gains against computation cost for cost-effective model transfer. To address this gap, we propose a framework called OPERA (Online Transfer using Phantom Tree for Real-Time Adaptation), that orchestrates transfer learning based on the accuracy gain and runtime tradeoffs between transferring and adapting a source model, versus no transfer and constructing a new model for the target stream. We conduct extensive empirical studies to show that our framework can balance accuracy performance gains and runtime for cost-effective transfer learning in the data stream environment.

Data collected over time often exhibit changes in distribution, or concept drift, caused by changes in hidden context relevant to the classification task, e.g. weather conditions. Adaptive learning methods are able to retain performance in changing conditions by explicitly detecting concept drift and changing the classifier used to make predictions. However, in realworld conditions, existing methods often select classifiers which poorly represent current data due to adaptation errors, where change in context is misidentified. We propose the AiRStream system, which uses a novel repair algorithm to identify and correct adaptation errors. We identify errors by periodically testing the performance of inactive classifiers. If an error is identified, a backtracking procedure repairs training done under the misidentified context. AiRStream achieves higher accuracy compared to baseline methods and selects classifiers which better match changes in context. A case study on a real-world air quality inference task shows that AiRStream is able to build a robust model of environmental conditions, allowing the adaptions made to concept drift to be analysed and related to changes in weather.

Streaming sources of data are becoming more common as the ability to collect data in real-time grows. A major concern in dealing with data streams is concept drift, a change in the distribution of data over time, for example, due to changes in environmental conditions. Representing concepts (stationary periods featuring similar behaviour) is a key idea in adapting to concept drift. By testing the similarity of a concept representation to a window of observations, we can detect concept drift to a new or previously seen recurring concept. Concept representations are constructed using meta-information features, values describing aspects of concept behaviour. We find that previously proposed concept representations rely on small numbers of meta-information features. These representations often cannot distinguish concepts, leaving systems vulnerable to concept drift. We propose FiCSUM, a general framework to represent both supervised and unsupervised behaviours of a concept in a fingerprint, a vector of many distinct meta-information features able to uniquely identify more concepts. Our dynamic weighting strategy learns which meta-information features describe concept drift in a given dataset, allowing a diverse set of meta-information features to be used at once. FiCSUM outperforms state-of-the-art methods over a range of 11 real world and synthetic datasets in both accuracy and modeling underlying concept drift.

Clinical narratives host vast accumulations of patient data pivotal for research and development of health related products. In order for this data to be utilized, the underlying protected health information needs to be de-identified to ensure medical confidentiality. Given the voluminous size of clinical texts, manual de-identification of such large datasets is both expensive and impractical. Therefore, the concept of automated de-identification is a highly appealing prospect. Machine learning or model based sequential labeling algorithms, such as the named entity recognition algorithms and rule-based algorithms are among the most effective approaches to automated de-identification. A natural question to ask is how we can combine them to have the best of both worlds. In this paper, we present an analytical and easy to interpret framework to dynamically combine a sequential labeling model and a soft-rule-based model in an incremental learning setup. This framework is applied to a case study, which is part of a project prototyping automated de-identification system for New Zealand clinical free text data. Evaluations show that our approach can accommodate changes in the incoming data through dynamic updating. The simplicity of the framework also allowed us to gain insights on behaviour e.g. change of importance between the machine learning and rule models.

Concept drift detection refers to the process of detecting changes in the underlying distribution of data. Interest in the data stream mining community has increased, because of their role in improving the performance of online learning algorithms. Over the years, a myriad of drift detection methods have been proposed. However, most of these methods are single detectors, which usually work well only with a single type of drift. In this research, we propose a semi-supervised locally-weighted ensemble detector (SLED), where the relative performance among its base detectors is characterized by a set of weights learned in a semi-supervised manner. The aim of this technique is to effectively deal with both abrupt and gradual concept drifts. In our experiments, SLED is configured with ten well-known drift detectors. To evaluate the performance of SLED, we compare it with single detectors as well as state-of-the-art ensemble methods on both synthetic and real-world datasets using different performance measures. The experimental results show that SLED has fewer false positives, higher precision, and higher Matthews correlation coefficient while maintaining reasonably good performance for other measures.

in their accompanying referral documents, which contain a mix of free text and structured data. By training a model to predict triage decisions from these referral documents, we can partially automate the triage process, resulting in more efficient and systematic triage decisions. One of the difficulties of this task is maintaining robustness against changes in triage priorities due to changes in policy, funding, staff, or other factors. This is reflected as changes in relationship between document features and triage labels, also known as concept drift. These changes must be detected so that the model can be retrained to reflect the new environment. We introduce a new concept drift detection algorithm for this domain called calibrated drift detection method (CDDM). We evaluated CDDM on benchmark and synthetic medical triage datasets, and find it competitive with state-of-the-art detectors, while also being less prone to false positives from feature drift.

Network embedding learns the vector representations of nodes. Most real world networks are heterogeneous and evolve over time. There are, however, no network embedding approaches designed for dynamic heterogeneous networks so far. Addressing this research gap is beneficial for analyzing and mining real world networks. We develop a novel representation learning method, change2vec, which considers a dynamic heterogeneous network as snapshots of networks with different time stamps. Instead of processing the whole network at each time stamp, change2vec models changes between two consecutive static networks by capturing newly-added and deleted nodes with their neighbour nodes as well as newly-formed or deleted edges that caused core structural changes known as triad closure or open processes. Change2vec leverages metapath based node embedding and change modeling to preserve both heterogeneous and dynamic features of a network. Experimental results show that change2vec outperforms two state-of-the-art methods in terms of clustering performance and efficiency.

The reasons for concept drift in a data stream can vary widely, from deterioration of a machine to a change in peoples' buying patterns. In order to effectively detect concept drifts, most predictive stream mining systems contain a drift detector that monitors and signals concept drifts. However, few of these systems are designed to find drifts in transactional datasets, which have unlabelled data. Transactional datasets describe events, such as orders or payments, which are traditionally analysed using association rules. In this paper, we propose a novel drift detection technique, ProChange, that has two parts. The first part is a drift detector, VR-Change, that finds both real and virtual drifts in unlabelled transactional data streams using the Hellinger distance. The second part is a drift predictor, which models the volatility of drifts using a probabilistic network to predict the location of future drifts. Using the predictor, we can dynamically adapt the confidence threshold, enabling VR-Change to be more sensitive around potential future drift points. We evaluated the performance of ProChange by comparing it against traditional detectors showing that it detects both real and virtual drifts effectively and efficiently in terms of accuracy.

Association rule mining was first introduced to examine patterns among frequent items. The original motivation for seeking these rules arose from need to examine customer purchasing behaviour in supermarket transaction data. It seeks to identify combinations of items or itemsets, whose presence in a transaction affects the likelihood of the presence of another specific item or itemsets. In recent years, there has been an increasing demand for rare association rule mining. Detecting rare patterns in data is a vital task, with numerous high-impact applications including medical, finance, and security. This survey aims to provide a general, comprehensive, and structured overview of the state-of-the-art methods for rare pattern mining. We investigate the problems in finding rare rules using traditional association rule mining. As rare association rule mining has not been well explored, there is still specific groundwork that needs to be established. We will discuss some of the major issues in rare association rule mining and also look at current algorithms. As a contribution, we give a general framework for categorizing algorithms: Apriori and Tree based. We highlight the differences between these methods. Finally, we present several real-world application using rare pattern mining in diverse domains. We conclude our survey with a discussion on open and practical challenges in the field.

Current drift detection techniques detect a change in distribution within a stream. However, there are no current techniques that analyze the change in the rate of these detected changes. We coin the term stream volatility, to describe the rate of changes in a stream. A stream has a high volatility if changes are detected frequently and has a low volatility if changes are detected infrequently. We are particularly interested in a volatility shift which is a change in the rate of change (e.g. From high volatility to low volatility). We introduce and define the concept of stream volatility, and propose a novel technique to detect volatility on data streams in the presence of concept drifts. In the experiments we show our algorithm to be both fast and efficient. We also propose a new algorithm for drift detection called SEED that is faster and more memory efficient than the existing state-of-the-art drift detection approach. A faster drift detection algorithm has a flow-on benefit to the subsequent volatility detection stage because both algorithms run concurrently on the data stream.

Recommender systems using Collaborative Filtering techniques are capable of make personalized predictions. However, these systems are highly vulnerable to profile injection attacks. Group attacks are attacks that target a group of items instead of one, and there are common attributes among these items. Such profiles will have a good probability of being similar to a large number of user profiles, making them hard to detect. We propose a novel technique for identifying group attack profiles which uses an improved metric based on Degree of Similarity with Top Neighbors (DegSim) and Rating Deviation from Mean Agreement (RDMA). We also extend our work with a detailed analysis of target item rating patterns. Experiments show that the combined methods can improve detection rates in user-based recommender systems.