Abstract Sequential recommendation models aim to learn from users’ evolving preferences. However, current state-of-the-art models suffer from an inherent popularity bias. This study developed a novel framework, BiCoRec, that adaptively accommodates users’ changing preferences for popular and niche items. Our approach leverages a co-attention mechanism to obtain a popularity-weighted user sequence representation, facilitating more accurate predictions. We then present a new training scheme that learns from future preferences using a consistency loss function. BiCoRec aimed to improve the recommendation performance of users who preferred niche items. For these users, BiCoRec achieves a 26.00% average improvement in NDCG@10 over state-of-the-art baselines. When ranking the relevant item against the entire collection, BiCoRec achieves NDCG@10 scores of 0.0102, 0.0047, 0.0021, and 0.0005 for the Movies, Fashion, Games and Music datasets.

Wildlife re-identification aims to match individuals of the same species across different observations. Current state-of-the-art (SOTA) models rely on class labels to train supervised models for individual classification. This dependence on annotated data has driven the curation of numerous large-scale wildlife datasets. This study investigates self-supervised learning Self-Supervised Learning (SSL) for wildlife re-identification. We automatically extract two distinct views of an individual using temporal image pairs from camera trap data without supervision. The image pairs train a self-supervised model from a potentially endless stream of video data. We evaluate the learnt representations against supervised features on open-world scenarios and transfer learning in various wildlife downstream tasks. The analysis of the experimental results shows that self-supervised models are more robust even with limited data. Moreover, self-supervised features outperform supervision across all downstream tasks. The code is available here https://github.com/pxpana/.

Abstract Catastrophic forgetting in neural networks during incremental learning remains a challenging problem. Previous research investigated catastrophic forgetting in fully connected networks, with some earlier work exploring activation functions and learning algorithms. Applications of neural networks have been extended to include similarity learning. Understanding how similarity learning loss functions would be affected by catastrophic forgetting is of significant interest. Our research investigates catastrophic forgetting for four well-known similarity-based loss functions during incremental class learning. The loss functions are Angular, Contrastive, Center, and Triplet loss. Our results show that the catastrophic forgetting rate differs across loss functions on multiple datasets. The Angular loss was least affected, followed by Contrastive, Triplet loss, and Center loss with good mining techniques. We implemented three existing incremental learning techniques, iCaRL, EWC, and EBLL. We further proposed a novel technique using Variational Autoencoders (VAEs) to generate representation as exemplars passed through the network’s intermediate layers. Our method outperformed three existing state-of-the-art techniques. We show that one does not require stored images (exemplars) for incremental learning with similarity learning. The generated representations from VAEs help preserve regions of the embedding space used by prior knowledge so that new knowledge does not “overwrite” it.

Abstract Portfolio management is a multi-period multi-objective optimisation problem subject to various constraints. However, portfolio management is treated as a single-period problem partly due to the computationally burdensome hyper-parameter search procedure needed to construct a multi-period Pareto frontier. This study presents the Pareto driven surrogate (ParDen-Sur) modelling framework to efficiently perform the required hyper-parameter search. ParDen-Sur extends previous surrogate frameworks by including a reservoir sampling-based look-ahead mechanism for offspring generation in evolutionary algorithms (EAs) alongside the traditional acceptance sampling scheme. We evaluate this framework against, and in conjunction with, several seminal multi-objective (MO) EAs on two datasets for both the single- and multi-period use cases. When considering hypervolume ParDen-Sur improves marginally (0.8%) over the state-of-the-art (SOTA)-NSGA-II. However, for generational distance plus and inverted generational distance plus, these improvements over the SOTA are 19.4% and 66.5%, respectively. When considering the average number of evaluations and generations to reach a 99% success rate, ParDen-Sur is shown to be 1.84× and 2.02× more effective than the SOTA. This improvement is statistically significant for the Pareto frontiers, across multiple EAs, for both datasets and use cases.

Abstract The restricted access to data in healthcare facilities due to patient privacy and confidentiality policies has led to the application of general natural language processing (NLP) techniques advancing relatively slowly in the health domain. Additionally, because clinical data is unique to various institutions and laboratories, there are not enough standards and conventions for data annotation. In places without robust death registration systems, the cause of death (COD) is determined through a verbal autopsy (VA) report. A non-clinician field agent completes a VA report using a set of standardized questions as guide to identify the symptoms of a COD. The narrative text of the VA report is used as a case study to examine the difficulties of applying NLP techniques to the healthcare domain. This paper presents a framework that leverages knowledge across multiple domains via two domain adaptation techniques: feature extraction and fine-tuning. These techniques aim to improve VA text representations for COD classification tasks in the health domain. The framework is motivated by multi-step learning, where a final learning task is realized via a sequence of intermediate learning tasks. The framework builds upon the strengths of the Bidirectional Encoder Representations from Transformers (BERT) and Embeddings from Language Models (ELMo) models pretrained on the general English and biomedical domains. These models are employed to extract features from the VA narratives. Our results demonstrate improved performance when initializing the learning of BERT embeddings with ELMo embeddings. The benefit of incorporating character-level information for learning word embeddings in the English domain, coupled with word-level information for learning word embeddings in the biomedical domain, is also evident.

Meta-learning, decision fusion, hybrid models, and representation learning are topics of investigation with significant traction in time-series forecasting research. Of these two specific areas have shown state-of-the-art results in forecasting: hybrid meta-learning models such as Exponential Smoothing- Recurrent Neural Network (ES-RNN) and Neural Basis Expansion Analysis (N-BEATS) and feature-based stacking ensembles such as Feature-based FORecast Model Averaging (FFORMA). However, a unified taxonomy for model fusion and an empirical comparison of these hybrid and feature-based stacking ensemble approaches is still missing. This study presents a unified taxonomy encompassing these topic areas. Furthermore, the study empirically evaluates several model fusion approaches and a novel combination of hybrid and feature stacking algorithms called Deep-learning FORecast Model Averaging (DeFORMA). The taxonomy contextualises the considered methods. Furthermore, the empirical analysis of the results shows that the proposed model, DeFORMA, can achieve state-of-the-art results in the M4 data set. DeFORMA, increases the mean Overall Weighted Average (OWA) in the daily, weekly and yearly subsets with competitive results in the hourly, monthly and quarterly subsets. The taxonomy and empirical results lead us to argue that significant progress is still to be made by continuing to explore the intersection of these research areas.

State-of-the-art RL algorithms have shown suboptimal performance in some market conditions with regard to the portfolio selection problem. The reason for suboptimal performance could be due to overestimation bias in actor-critic methods through the use of neural networks as the function approximator. The resulting bias leads to a suboptimal policy being learned by the agent, hindering performance. This research focuses on using the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm for portfolio selection to achieve greater results than previously achieved. In addition, an analysis of the overall effectiveness of the algorithm in various market conditions is needed to determine the TD3’s robustness. This research establishes a RL environment for portfolio selection and trains the TD3 alongside three state-of-the-art algorithms in five different market conditions. The algorithms are tested by allowing the agent to manage a portfolio in each market for a specified period. The results are used for the analysis of the algorithms. The research shows improved results achieved by the TD3 algorithm for portfolio selection compared to other state-of-the-art algorithms. Furthermore, the performance of the TD3 across the five selected markets proves the robustness of the algorithm in its use for the portfolio selection problem.

We present the constrained portfolio selection problem as a learning problem requiring hyper-parameter specification. In practice, hyper-parameters are typically selected using a validation procedure, of which there are several widely-used alternatives. However, the performance of different validation procedures is problem dependent and has not been investigated for the portfolio selection problem. This study examines the behaviour of common validation procedures, including holdout, k-fold cross-validation, Monte Carlo cross-validation, and repeated k-fold cross-validation for estimating performance and selecting hyper-parameters for constrained portfolio selection. The results demonstrate that repeated k-fold cross-validation is the best performing procedure and recommend using 5 repetitions with 3 ≤ k ≤ 10 in practice.

We investigate the benefits of using multispectral images for land cover classification. To perform this comparative analysis, we present a novel model, LandNet which is configurable with multiple deep residual networks to extract features on several combinations of bands. We consider both the classification accuracy of the various LandNet configurations. We perform this study on the EuroSAT and BigEarthNet datasets, both of which contain multispectral images from the Sentinel-2 mission. On EuroSAT, we convincingly demonstrate marked improvements in the accuracy of around 1% to 97.815% when using additional bands compared to merely using the RGB bands. On BigEarthNet we show the additional bands are able to improve the recall of the LandNet by 0.04. We achieve a precision score of 0.85, recall of 0.80 and an F-score of 0.82. The precision, recall and F-score we achieve outperform prior results achieved on BigEarthNet.

Maintenance data from power transformers are typically in the form of dissolved gas analysis time series data. This research attempts consolidating industry knowledge on the maintenance of power transformers and time series forecasting techniques into a coherent system for the purpose of predictive maintenance of power transformers. The generalisability of forecasting models is investigated by measuring performance of single models across multiple transformers, and hence, multiple data sets. A novel method of data preprocessing is utilized; an exponential smoothing technique specifically designed for the type of raw data received (aperiodically sampled, noisy data). In addition, industry specified features for fault classification from the literature were added to the data set. These other features were examined to see if they improved forecasting. The forecasting techniques evaluated included Least-Squares Support Vector Machine (LS-SVM) with hyper-parameters optimized using a Particle Swarm Optimisation; Support Vector Regressors; Naive forecasts; Mean forecasts; Auto-Regressive Integrated Moving average or ARIMA; and Exponential Smoothing. Two sets of experiments were run, which differed in how the Training, Validation, and Testing sets were chosen. These experiments were run for different input vectors; the original input vector and an input vector augmented with the industry specified features. Models trained in the second experiment outperformed all other models, when the distributions of the Testing errors were considered. When generalisability was considered, it was found that the models trained across all transformers outperformed the per-transformer models that treated the data as univariate time series.

This paper tackles face recognition in videos employing metric learning methods and similarity ranking models. The paper compares the use of the Siamese network with contrastive loss and Triplet Network with triplet loss implementing the following architectures: Google/Inception architecture, 3D Convolutional Network (C3D), and a 2-D Long short-term memory (LSTM) Recurrent Neural Network. We make use of still images and sequences from videos for training the networks and compare the performances implementing the above architectures. The dataset used was the YouTube Face Database designed for investigating the problem of face recognition in videos. The contribution of this paper is two-fold: to begin, the experiments have established 3-D Convolutional networks and 2-D LSTMs with the contrastive loss on image sequences do not outperform Google/Inception architecture with contrastive loss in top n rank face retrievals with still images. However, the 3-D Convolution networks and 2-D LSTM with triplet Loss outperform the Google/Inception with triplet loss in top n rank face retrievals on the dataset; second, a Support Vector Machine (SVM) was used in conjunction with the CNNs' learned feature representations for facial identification. The results show that feature representation learned with triplet loss is significantly better for n-shot facial identification compared to contrastive loss. The most useful feature representations for facial identification are from the 2-D LSTM with triplet loss. The experiments show that learning spatio-temporal features from video sequences is beneficial for facial recognition in videos.

Cement production is a multi-billion dollar industry, of which one of the main sub-processes, cement milling, is complex and non-linear. There is a need to model the fineness of particles exiting the milling circuit to better control the cement plant. This paper explores the relationship between the particle size of cement produced and the operation of the cement mill circuit. This paper aims to provide a model for predicting the fineness of particles exiting the milling circuit using data on the current and past states of the plant. A comprehensive literature review of the problem, as well as a discussion of potential modelling solutions, is provided. Blaine (particle fineness)is modelled using many different linear and non-linear models on 5 months of data from a Chinese cement plant. On a holdout test set a multi-layered perceptron achieved an MAE of 8.799 and a linear regression achieved a R2of 0.481. discussion of the significance of various features for predicting Blaine is also presented. The results show some limited success from non-linear data-driven models and highlight some of the unique difficulties in modelling the cement mill and present recommendations for future research.

Soccer match prediction has been a difficult domain for machine learning, currently outperformed by bookmakers' odds and human predictions due to the stochastic nature of soccer. In response, we focus on a Similarity learning approach. Our research involves using Siamese networks and RankNet pair-wise models alongside Transfer learning to predict soccer match rankings. We implemented these models in conjunction with traditional sports tally ranking as well as graph-based PageRank to attain list-wise seasonal rankings for English Premier League spanning seasons 2006 to 2018. Our models used two datasets- EPL seasonal team statistics and an augmentation of these team statistics with external (monetary and transfer) data. Our models have a lower F1 score than a standard neural network, however, perform better in identifying draws and seasonal teams rankings. Transfer Learning models performed better on match-wise rankings while for seasonal rankings, augmented data provides the best predictor (0.72 average). Our proposed Tally Rank provides more accurate seasonal rankings than a graph-based PageRank. Since there is no consistently best-performing model, other Similarity Learning/Ranking models can be considered in the future.

A significant challenge faced within the field of chemical plant design and optimisation is the uncertainty in the reaction parameter, indeterminate component failures and their impacts on the design and operation. Additionally, real-world data accumulated from these plants would contain stochastic elements that are difficult to model. To this end, stochastic and deterministic methods have been proposed to simulate the uncertainty and enable an understanding of the plant and how it may be optimised. Within the existing literature investigated, the optimisation is done under the assumption that the simulation (target function) is non-stochastic. We have found that the use of an Evolutionary Algorithm in the form of a Genetic Algorithm can find an optimal solution even when we allow the simulation to behave stochastically as it would in practical applications. Further, we note that the use of a surrogate Machine Learning model as a substitute for the stochastic simulation model leads to substantively improved solutions in significantly less time (1.82 times speedup). We argue that the use of Genetic Algorithms in the optimisation of chemical plant design, taking into account the stochastic nature of the plant and including indeterminate failures, is a worthwhile solution and that surrogate assisted evolutionary algorithms will improve this solution further.

The Galaxy Zoo project is a crowd-sourced astronomy galaxy classification endeavour whose results can have significant benefits to astronomers. The project has evolved into using crowd-sourced labelling together with machine learning to automate the classification of galaxies. If this process is to be automated using crowd-sourcing and machine learning, then understanding how these results will hold up against expert classifications on an academically accepted classification such as the Hubble tuning fork is timely. We propose a rules-based approach for deriving the Hubble type using the responses in the Galaxy Zoo as well as a Transfer Learning approach for solving this problem. The dataset we used to get the Hubble type for galaxies is the Revised Shapley-Ames catalogue of bright galaxies. Previous work in this field has mainly revolved around the Galaxy Zoo project with little to no attempt to map the Galaxy Zoo responses to a more robust method of classifying galaxies such as the Hubble tuning fork classification system. Previous research has tried to map the Galaxy Zoo responses to a set of classes like elliptical, spiral and irregular. Their work has shown promising results. Our experiments showed that by using the Galaxy Zoo response vectors, our rules-based approach was able to separate the elliptical and spiral shapes, however, it did not perform particularly well at separating the spiral shapes from one another. Our Transfer Learning model showed better potential for separating not only elliptical and spiral shapes but also for separating spiral shapes into exact Hubble types (e.g Sa, Sb and Sc).

The unique automated identification of animals of various species is a pressing challenge ecologically, environmentally and economically. A broader question relates to how one might exploit the somewhat more mature technologies and techniques used within human visual biometrics to automate this same task for other species. One specific technique is the use of region proposal networks and deep transfer learning in siamese networks for individual animal identification. We report that although it is relatively easy to achieve state of the art performance in uniquely identifying individuals for the easy target of zebras, trying to use the same pipeline to obtain useable, top-10> 85%, results for a more challenging species such as nyala is still an open research problem. We argue that uniquely identifying individuals such as nyala who actively try to disguise themselves in their environments require improved few-shot learning techniques and perhaps more data than the current open dataset we have provided to stimulate this area of research.

Although the focus of the Sensor Web has been somewhat limited to a single architectural view in the form of web services and service oriented architectures our experience has shown in a number of projects that this is not always the most effective solution, especially when deal with Big Data. The correct approach is then to hold to the overall vision of the Sensor Web as a way of gaining access to and organising sensors and sensor data and to use the appropriate architectural patterns and strategies in overcoming the challenges presented. Using these other approaches is not necessarily conflict with an open systems view nor is it non web centric.

The Sensor Web as an open complex adaptive system exhibits many characteristic that are common to self-organising systems. One of the characteristics of the Sensor Web is that of self-adaptivity in a changing environment. The changing environment may be doing so both dynamically and stochastically. When presented by a dynamic and stochastic changing environment, such as a sensor resource unexpectedly going down, a self-adaptive system should exhibit robustness. Cell-fate optimisation and signal regulatory networks provide a mechanism for self-organisation of agents in environments that are dynamic, distributed and possibly stochastic. Cell-fate optimisation and signal regulatory networks are shown to be effective mechanisms not only for addressing robustness but also for addressing adaptivity in the sensor web in general.

Scientific Workflows provides a technology that facilitates researchers by allowing them to capture in a machine processable manner the method relating to some research. This increases both provenance and repeatability of the research and allows for increased collaboration through workflow sharing. The Sensor Web is an open complex adaptive system the pervades the Internet and provides access to sensor resources. One mechanism for describing sensor resources is through the use of SensorML. It is shown that SensorML can be used as a mechanism for describing Scientific Workflows and thus facilitates completely distributed workflow descriptions on the web. However m order to fully capture the requirements as relating to open and distributed environments some extensions to SensorML will be required.

Our findings are not easy to generalise, but many of the insights offered by natural scientists in the cholera research domain through this process serve as further confirmation of the known challenges of scientific workflows. It is the potential for using the facilities of the sensor web through the perceived simplifying medium of scientific workflows that interests natural scientists. Significant technical and in particular, meta-data related challenges will need to be addressed before the sensor web becomes more accessible in this way to scientists.

The Committee of Earth Observation Satellites (CEOS) acting as the space arm of the Group on Earth Observations (GEO) will be required to inform the architecture and design of the Global Earth Observing System of System (GEOSS). Current architectural and design activities within CEOS that will direct GEOSS include but are not limited to Virtual Constellations, the Sensor Web, and Wide Area Grids (WAG). Currently many of these activities are being explored in relative isolation within the CEOS Working Group on Information Systems and Services (WGISS). If CEOS is to present a consolidated set of complimentary recommendations, it will be required that at least one of these activities be willing to expand the barriers of the field and look for ways to include all these efforts. Current views in the Sensor Web activities explore a service oriented approach as a best practice informed by a set of standard interface recommendations [1, 2, 3]. If the view of the Sensor Web is expanded beyond these specific technological and architectural constraints and embraces the Sensor Web as a concept, it is possible for the lessons and techniques in each of these other activities to be integrated into this single view.

Web resident sensor resource discovery plays a crucial role in the realisation of the Sensor Web. The vision of the Sensor Web is to create a web of sensors that can be manipulated and discovered in real time. A current research challenge in the sensor web is the discovery of relevant web sensor resources. The proposed approach towards solving the discovery problem is to implement a modified Latent Semantic Indexing by making use of an ontology for classifying Web Resident Resources found in geospatial Web portals. The paper presents the use of Latent Semantic Indexing, an information retrieval mechanism, biased by combining ontology concepts to the terms and objects, for improving the knowledge extraction from web resident documents. The use of an ontology, before indexing of terms, to create a semantic link between documents with relevant content improves automatic content extraction and document classification.