Abstract Today, the use of Artificial Intelligence (AI) is rapidly increasing in many areas of society. While model performance on various tasks continue to impress, it does so at the cost of increased model complexity, such that most state-of-the-art AI models are effectively black boxes. Where human-made decisions typically are accompanied by human-understandable explanations detailing the reasoning behind the decision, incorporating advanced AI as part of a decision-making process reduces the transparency of that process significantly. Yet, the ability to explain decisions is essential for there to be understanding and trust. As a response to this, Explainable Artificial Intelligence (XAI) has emerged as a field that aims to provide explanations of model behaviour. Methods categorised as post-hoc are designed to generate explanations for black box models after training, at no cost to model performance. In parallel with this, extensive work has been done in the field of causality to formalise the structure of human-understandable, causal explanations. This work presents a comprehensive literature review of the current state of the subfield of XAI that consist of causality-motivated post-hoc XAI methods. In order to clearly define causal XAI, a causal framework for categorising XAI is introduced, and three types of post-hoc XAI methods are identified: observational methods, internally causal methods and externally causal methods. Finally, externally causal XAI is argued a promising direction for reliable and understandable post-hoc XAI, with the ability to generate counterfactual explanations using a meaningful vocabulary, in line with the definition of counterfactual used in causal theory.

Representation learning for time series using contrastive learning has emerged as a critical technique for improving the performance of downstream tasks. To advance this effective approach, we introduce CaTT (Contrast All The Time), a new approach to unsupervised contrastive learning for time series, which takes advantage of dynamics between temporally similar moments more efficiently and effectively than existing methods. CaTT departs from conventional time-series contrastive approaches that rely on data augmentations or selected views. Instead, it uses the full temporal dimension by contrasting all time steps in parallel. This is made possible by a scalable NT-pair formulation, which extends the classic N-pair loss across both batch and temporal dimensions, making the learning process end-to-end and more efficient. CaTT learns directly from the natural structure of temporal data, using repeated or adjacent time steps as implicit supervision, without the need for pair selection heuristics. We demonstrate that this approach produces superior embeddings which allow better performance in downstream tasks. Additionally, training is faster than other contrastive learning approaches, making it suitable for large-scale and real-world time series applications. The source code is publicly available at https://github.com/sfi-norwai/CaTT.

Sleep is a crucial health metric linked to various health problems. Accurate analysis of sleep duration relies on identifying sleep and wake phases. While Polysomnography is considered the gold standard for sleep measurements, it is impractical for large population-based studies. Accelerometers offer an alternative but face challenges in accurate sleep–wake recognition (SWR). Recent advances in self-supervised learning (SSL) have shown promise in related fields. This study introduces a contribution to the field of artificial intelligence by proposing a new SSL model, long-term accelerometer to vector (LTA2V), for SWR. LTA2V learns context information during pre-training using global positional encoding and long-term sequences. Experiments on three datasets show that LTA2V outperforms other methods, achieving an average F1-score of 73.8%, outperforming the best supervised method by 3.4% and the best alternative SSL model by 1.7%. Analysis of Variance and Tukey’s Honest Significant Difference tests show that SSL methods significantly outperform rule-based and traditional machine learning methods and are on par with supervised deep learning . Despite improvements, SWR performance enhancements through SSL are limited compared to other fields, suggesting that there is limited scope for further major improvements in accelerometer-based SWR. Applying LTA2V in population-based studies can lead to more reliable data on sleep duration across populations, improving our understanding of how sleep impacts health.

We present a novel vision-based, 6-DoF grasping framework based on Deep Reinforcement Learning (DRL) that is capable of directly synthesizing continuous 6-DoF actions in cartesian space. Our proposed approach uses visual observations from an eye-in-hand RGB-D camera, and we mitigate the sim-to-real gap with a combination of domain randomization, image augmentation, and segmentation tools. Our method consists of an off-policy, maximum-entropy, Actor-Critic algorithm that learns a policy from a binary reward and a few simulated example grasps. It does not need any real-world grasping examples, is trained completely in simulation, and is deployed directly to the real world without any fine-tuning. The efficacy of our approach is demonstrated in simulation and experimentally validated in the real world on 6-DoF grasping tasks, achieving state-of-the-art results of an 86% mean zero-shot success rate on previously unseen objects, an 85% mean zero-shot success rate on a class of previously unseen adversarial objects, and a 74.3% mean zero-shot success rate on a class of previously unseen, challenging "6-DoF" objects.Raw footage of real-world validation can be found at https://youtu.be/bwPf8Imvook

Abstract Annotating accelerometer-based physical activity data remains a challenging task, limiting the creation of robust supervised machine learning models due to the scarcity of large, labeled, free-living human activity recognition (HAR) datasets. Researchers are exploring self-supervised learning (SSL) as an alternative to relying solely on labeled data approaches. However, there has been limited exploration of the impact of large-scale, unlabeled datasets for SSL pre-training on downstream HAR performance, particularly utilizing more than one accelerometer. To address this gap, a transformer encoder network is pre-trained on various amounts of unlabeled, dual-accelerometer data from the HUNT4 dataset: 10, 100, 1k, 10k, and 100k hours. The objective is to reconstruct masked segments of signal spectrograms. This pre-trained model, termed SelfPAB, serves as a feature extractor for downstream supervised HAR training across five datasets (HARTH, HAR70+, PAMAP2, Opportunity, and RealWorld). SelfPAB outperforms purely supervised baselines and other SSL methods, demonstrating notable enhancements, especially for activities with limited training data. Results show that more pre-training data improves downstream HAR performance, with the 100k-hour model exhibiting the highest performance. It surpasses purely supervised baselines by absolute F1-score improvements of 7.1% (HARTH), 14% (HAR70+), and an average of 11.26% across the PAMAP2, Opportunity, and RealWorld datasets. Compared to related SSL methods, SelfPAB displays absolute F1-score enhancements of 10.4% (HARTH), 18.8% (HAR70+), and 16% (average across PAMAP2, Opportunity, RealWorld).

Self-supervised learning (SSL) has gained prominence in the field of accelerometer-based human activity recognition (HAR) due to its ability to learn from both labeled and unlabeled data. While labeled data acquisition is costly, it is relatively easy to accumulate unlabeled sensor data. However, few works utilize large-scale, unlabeled datasets for pre-training despite its positive impact on downstream HAR performance, shown in recent work. Cross-sensor upstream training has also received limited attention. We introduce a new auxiliary task, randomized cross-sensor masked reconstruction (RCSMR), for SSL. We pre-train a transformer encoder on the large-scale HUNT4 dataset with RCSMR. The resulting model exhibits better performance on two downstream datasets with the same sensor setup as HUNT4 (HARTH and HAR70+), achieving an average F1-score of 74.03%, surpassing two other auxiliary tasks (70.51% to 72.78%) and five supervised baselines (47.51% to 58.84%). Moreover, when applied to three datasets with sensor configurations distinct from HUNT4 (USC-HAD, PAMAP2, MobiAct), RCSMR outperforms nine state-of-the-art SSL methods, with an F1-score of 72.99% compared to F1-scores ranging from 51.46% to 69.88%. We further show that certain activities exhibit improved separability when utilizing latent representations learned through RCSMR, indicating reduced sensor position and orientation bias. Our method is applied in large-scale epidemiological studies, offering valuable insights into the impact of physical activity behavior on public health.

Constructing an accurate representation model of phenomena with fewer measurements is a fundamental challenge in the Internet of Things. Leveraging sparse sensing policies to select the most informative measurements is a prominent technique for addressing resource constraints. However, designing such sensing policies requires significant domain knowledge and involves manually fine-tuned heuristics that are task-specific and often non-adaptive. In this work, we propose reducing manual-engineering efforts in designing sensing policies by using an automated approach based on deep reinforcement learning. Guided by an uncertainty-aware prediction model, the sensors learn sensing behaviors autonomously by optimizing an application goal formulated in the reward function based on the measured peaks-over-threshold. We apply the proposed approach in two use cases of monitoring air quality and indoor noise and show the adaptability and transferability of the learned policies. Compared to conventional periodic sensing methods, our results achieve, on average, an increased detection in periods of interest by 78.5% and 357.3% while reducing energy expenditure by 14.3% and 7.6% for air quality and noise monitoring, respectively. Additionally, the resulting representation models are more credible, as measured by various metrics of probabilistic modeling.

Actuation delay poses a challenge for robotic arms and cranes. This is especially the case in dynamic environments where the robot arm or the objects it is trying to manipulate are moved by exogenous forces. In this paper, we consider the task of using a robotic arm to compensate for relative motion between two vessels at sea. We construct a hybrid controller that combines an Inverse Kinematic (IK) solver with a Reinforcement Learning (RL) agent that issues small corrections to the IK input. The solution is empirically evaluated in a simulated environment under several sea states and actuation delays. We observe that more intense waves and larger actuation delays have an adverse effect on the IK controller's ability to compensate for vessel motion. The RL agent is shown to be effective at mitigating large parts of these errors, both in the average case and in the worst case. Its modest requirement for sensory information, combined with the inherent safety in only making small adjustments, also makes it a promising approach for real-world deployment.

To address poor working conditions and promote United Nations’ sustainable development goal 8.8, “protect labour rights and promote safe working environments for all workers [...]”, government agencies around the world conduct labour inspections. To carry out these inspections, inspectors traditionally use paper-based checklists as a means to survey individual organisations for working environment violations. Currently, these checklists are created by domain experts, but recent research indicates that machine learning (ML) could be used to generate dynamic checklists to increase inspection efficiency. A drawback with the dynamic checklists is that they are complex and could be difficult to understand for inspectors. They have also never been field-tested. In this paper, we therefore propose user-oriented explanation methods for Context-aware Bayesian Case-Based Reasoning (CBCBR), which is the current state-of-art ML method for generating dynamic checklists. We also introduce a prototype of CBCBR and present a field study where we test it in real-world labour inspections. The results from the study indicate that using the explainable dynamic checklists increases the efficiency of the labour inspections, and inspectors also report that they find the checklists useful. The results also suggest that current ML evaluation methods, where model prediction performance is evaluated on existing data, may not fully reflect the real-world field performance of checklists.

Energy-harvesting wireless sensors require energy management due to volatile energy sources. Existing energy managers lack adaptability to changing utility requirements or often rely on manually defined utility profiles. To address this, we study an autonomous energy manager that learns utility profiles dynamically, without the need for prior data. The new energy manager ensures that devices adapt to evolving utility needs, extending their operational capabilities in changing environments.

IoT applications with their resource-constrained sensor devices can benefit from adjusting their operations to the phenomena they sense and the environments they operate in, leading to the paradigm of self-adaptive, autonomous, or cognitive IoT. On the other side, current AI deployment platforms focus on the provision and reuse of machine learning models through containers that can be wired together to build new applications. The challenge is that composition mechanisms of the AI platforms, albeit effective due to their simplicity, are in fact too simplistic to support cognitive IoT applications, in which sensor devices also benefit from the machine learning results. Our objective is to perform a gap analysis between the requirements of cognitive IoT applications on the one side and the current functionalities of AI deployment platforms on the other side. In this work, we provide an overview of the paradigms in AI deployment platforms and the requirements of cognitive IoT applications. We study a use case for person counting in a skiing area through camera sensors, and how this use case benefits from letting the IoT sensors have access to operational knowledge in the form of visual attention models. We describe the implementation of the IoT application using an AI deployment platform, analyze its shortcomings, and necessary workarounds. From the use case, we identify and generalize four gaps that limit the usage of deployment platforms: the transparent management of multiple instances of components, a more seamless integration with IoT devices, explicit definition of data flow triggers, and the availability of templates for cognitive IoT architectures and reuse below the top-level.

Every year there are 1.9 million deaths world-wide attributed to occupational health and safety risk factors. To address poor working conditions and fulfill UN's SDG 8, "protect labour rights and promote safe working environments for all workers", governmental agencies conduct labour inspections, using checklists to survey individual organisations for working environment violations. Recent research highlights the benefits of using machine learning for creating checklists. However, the current methods only create static checklists and do not adapt them to new information that surfaces during use. In contrast, we propose a new method called Context-aware Bayesian Case-Based Reasoning (CBCBR) that creates dynamic checklists. These checklists are continuously adapted as the inspections progress, based on how they are answered. Our evaluations show that CBCBR's dynamic checklists outperform static checklists created via the current state-of-the-art methods, increasing the expected number of working environment violations found in the labour inspections.

Abstract Aquaculture as an industry is quickly expanding. As a result, new aquaculture sites are being established at more exposed locations previously deemed unfit because they are more difficult and resource demanding to safely operate than are traditional sites. To help the industry deal with these challenges, we have developed a decision support system to support decision makers in establishing better plans and make decisions that facilitate operating these sites in an optimal manner. We propose a case-based reasoning system called aquaculture case-based reasoning (AQCBR), which is able to predict the success of an aquaculture operation at a specific site, based on previously applied and recorded cases. In particular, AQCBR is trained to learn a similarity function between recorded operational situations/cases and use the most similar case to provide explanation-by-example information for its predictions. The novelty of AQCBR is that it uses extended Siamese neural networks to learn the similarity between cases. Our extensive experimental evaluation shows that extended Siamese neural networks outperform state-of-the-art methods for similarity learning in this task, demonstrating the effectiveness and the feasibility of our approach.

In exercise gaming (exergaming), reward systems are typically based on rules/templates from joint movement patterns. These rules or templates need broad ranges in definitions of correct movement patterns to accommodate varying body shapes and sizes. This can lead to inaccurate rewards and, thus, inefficient exercise, which can be detrimental to progress. If exergames are to be used in serious settings like rehabilitation, accurate rewards for correctly performed movements are crucial. This article aims to investigate the level of accuracy machine learning/deep learning models can achieve in classification of correct repetitions naturally elicited from a weight-shifting exergame. Twelve healthy elderly (10F, age 70.4 SD 11.4) are recruited. Movements are captured using a marker-based 3-D motion-capture system. Random forest (RF), support vector machine, k-nearest neighbors, and multilayer perceptron (MLP) are the employed models, trained and tested on whole body movement patterns and on subsets of joints. MLP and RF reached the highest recall and F1-score, respectively, when using combined data from joint subsets. MLP recall range are 91% to 94%, and RF F1-score range 79% to 80%. MLP and RF also reached the highest recall and F1-score in each joint subset, respectively. Here, MLP ranged from 93% to 97% recall, while RF ranged from 73% to 80% F1-score. Recall results, show that >9 out of 10 repetitions are classified correctly, indicating that MLP/RF can be used to identify correctly performed repetitions of a weight-shifting exercise when using full-body data and when using joint subset data.

Visual analytics have an important role in the exploration and analysis of large amounts of data in IoT applications. Data visualizations can provide overviews of different aspects of data and user interaction can assist exploration. Recent advances in machine learning and Artificial Intelligence have provided methods that can be used in conjunction with visual analytics to enhance user perception. However, AI methods are often used as "black boxes", making them difficult for end-users to trust. In this paper, a novel visual analytics platform is presented, targeting two goals: a) an architecture for the creation of custom interactive visual analytics dashboards using well-defined components linked to each other, and b) the inclusion of components specifically for making AI methods more explainable. The proposed architecture and components are being used in the context of the AI4IoT pilot within the AI4EU project, which targets air quality monitoring through AI and visualization.

This paper presents the design of a Clinician Dashboard to promote co-decision making between patients and clinicians. Targeted patients are those with non-specific low back pain, a leading cause of discomfort, disability and absence from work throughout the world. Targeted clinicians are those in primary care, including general practitioners, physiotherapists, and chiropractors. Here, the functional specifications for the Clinical Dashboard are delineated, and wireframes illustrating the system interface and flow of control are shown. Representative scenarios are presented to exemplify how the system could be used for co-decision making by a patient and clinician. Also included are a discussion of potential barriers to implementation and use in clinical practice and a look ahead to future work. This work has been conducted as part of the Horizon 2020 selfBACK project, which is funded by the European Commission.