Human Activity Recognition (HAR) in smart environments is a well-explored research domain, given its diverse applications which include healthcare, surveillance, building management, and many more. While the majority of HAR research focuses on recognizing the activities of a single subject, in real-world scenarios smart environments are often populated by multiple subjects that may be engaged in both independent and joint activities. This gives rise to the challenge of Multi-Subject HAR, which is an open and complex problem. This survey paper aims to offer researchers and practitioners a comprehensive analysis of Multi-Subject HAR, encompassing its potential applications, sensing solutions, methods, datasets, evaluation metrics, and ongoing challenges. In addition to presenting the latest research works in this area and identifying open issues, our major contributions consist of a comprehensive problem formalization and a thorough discussion of the evaluation metrics to assess different dimensions of multi-subject HAR systems. • We provide a comprehensive formalization for multi-subject HAR in smart environments. • We review the latest research works and datasets in this area. • We explore evaluation metrics and dataset-splitting strategies.

Explainable Artificial Intelligence (XAI) aims to uncover the inner reasoning of machine learning models. In IoT systems, XAI improves the transparency of models processing sensor data from multiple heterogeneous devices, ensuring end-users understand and trust their outputs. Among the many applications, XAI has also been applied to sensor-based Activities of Daily Living (ADLs) recognition in smart homes. Existing approaches highlight which sensor events are most important for each predicted activity, using simple rules to convert these events into natural language explanations for non-expert users. However, these methods produce rigid explanations lacking natural language flexibility and are not scalable. With the recent rise of Large Language Models (LLMs), it is worth exploring whether they can enhance explanation generation, considering their proven knowledge of human activities. This article investigates potential approaches to combine XAI and LLMs for sensor-based ADL recognition. We evaluate if LLMs can be used: (a) as explainable zero-shot ADL recognition models, avoiding costly labeled data collection, and (b) to automate the generation of explanations for existing data-driven XAI approaches when training data is available and the goal is higher recognition rates. Our critical evaluation provides insights into the benefits and challenges of using LLMs for explainable ADL recognition.

The sensor-based recognition of Activities of Daily Living (ADLs) in smart home environments enables several applications in the areas of energy management, safety, well-being, and healthcare. ADL recognition is typically based on deep learning methods requiring large datasets to be trained. Recently, several studies proved that Large Language Models (LLMs) effectively capture common-sense knowledge about human activities. However, the effectiveness of LLMs for ADL recognition in smart home environments still deserves to be investigated. In this work, we propose ADL-LLM, a novel LLM-based ADL recognition system. ADL-LLM transforms raw sensor data into textual representations, that are processed by an LLM to perform zero-shot ADL recognition. Moreover, in the scenario where a small labeled dataset is available, ADL-LLM can also be empowered with few-shot prompting. We evaluated ADL-LLM on two public datasets, showing its effectiveness in this domain.

This paper describes the development of PRACTICE, a distributed healthcare technological platform that supports various research initiatives by the University of Milan and the Angelo Bianchi Bonomi Hemophilia and Thrombosis Center, Fondazione IRCCS Ca’ Granda, Ospedale Maggiore Policlinico. PRACTICE includes three main components: a mobile app that patients can use to self-acquire ultrasound images at home, a computer-aided diagnosis web application that supports the practitioners through a set of machine learning models, and a set of web tools for image annotation, a prerequisite for training the machine learning models. Although PRACTICE was designed in the specific context of supporting the detection of joint recess blood effusions in hemophilic patients, this paper describes the main design and implementation challenges that apply to other applications of a research-oriented health platform.

Context-aware Human Activity Recognition (HAR) is a hot research area in mobile computing, and the most effective solutions in the literature are based on supervised deep learning models. However, the actual deployment of these systems is limited by the scarcity of labeled data that is required for training. Neuro-Symbolic AI (NeSy) provides an interesting research direction to mitigate this issue, by infusing common-sense knowledge about human activities and the contexts in which they can be performed into HAR deep learning classifiers. Existing NeSy methods for context-aware HAR rely on knowledge encoded in logic-based models (e.g., ontologies) whose design, implementation, and maintenance to capture new activities and contexts require significant human engineering efforts, technical knowledge, and domain expertise. Recent works show that pre-trained Large Language Models (LLMs) effectively encode common-sense knowledge about human activities. In this work, we propose ContextGPT: a novel prompt engineering approach to retrieve from LLMs common-sense knowledge about the relationship between human activities and the context in which they are performed. Unlike ontologies, ContextGPT requires limited human effort and expertise, while sharing similar privacy concerns if the reasoning is performed in the cloud. An extensive evaluation using two public datasets shows how a NeSy model obtained by infusing common-sense knowledge from ContextGPT is effective in data scarcity scenarios, leading to similar (and sometimes better) recognition rates than logic-based approaches with a fraction of the effort.

Human Activity Recognition (HAR) with mobile and wearable devices has been deeply studied in the last decades. Research groups working on this topic evaluated their proposed methods mostly on public datasets. However, most of the existing datasets only include inertial sensor data, while it is well-known that additional context data (e.g., semantic location) has the potential to significantly improve the recognition rate. Only a few datasets for context-aware HAR are publicly available, and their annotations were mostly self-reported in-the-wild by the subjects involved in data acquisition. This method harms the quality of annotations, thus discouraging the application of supervised models. In this paper, we propose DOMINO, a new public dataset for context-aware HAR. DOMINO includes 25 users (wearing a smartphone and a smartwatch) performing 14 activities. During data acquisition, the mobile devices recorded both inertial and high-level context data while our team monitored the quality of the self-reported annotations. Our experiments on DOMINO show the positive impact of considering high-level context information for Human Activity Recognition.

The use of supervised learning for Human Activity Recognition (HAR) on mobile devices leads to strong classification performances. Such an approach, however, requires large amounts of labeled data, both for the initial training of the models and for their customization on specific clients (whose data often differ greatly from the training data). This is actually impractical to obtain due to the costs, intrusiveness, and time-consuming nature of data annotation. Moreover, even with the help of a significant amount of labeled data, model deployment on heterogeneous clients faces difficulties in generalizing well on unseen data. Other domains, like Computer Vision or Natural Language Processing, have proposed the notion of pre-trained models, leveraging large corpora, to reduce the need for annotated data and better manage heterogeneity. This promising approach has not been implemented in the HAR domain so far because of the lack of public datasets of sufficient size. In this paper, we propose a novel strategy to combine publicly available datasets with the goal of learning a generalized HAR model that can be fine-tuned using a limited amount of labeled data on an unseen target domain. Our experimental evaluation, which includes experimenting with different state-of-the-art neural network architectures, shows that combining public datasets can significantly reduce the number of labeled samples required to achieve satisfactory performance on an unseen target domain.

Abstract The sensor-based recognition of Activities of Daily Living (ADLs) in smart-home environments enables several important applications, including the continuous monitoring of fragile subjects in their homes for healthcare systems. The majority of the approaches in the literature assume that only one resident is living in the home. Multi-inhabitant ADLs recognition is significantly more challenging, and only a limited effort has been devoted to address this setting by the research community. One of the major open problems is called data association, which is correctly associating each environmental sensor event (e.g., the opening of a fridge door) with the inhabitant that actually triggered it. Moreover, existing multi-inhabitant approaches rely on supervised learning, assuming a high availability of labeled data. However, collecting a comprehensive training set of ADLs (especially in multiple-residents settings) is prohibitive. In this work, we propose MICAR: a novel multi-inhabitant ADLs recognition approach that combines semi-supervised learning and knowledge-based reasoning. Data association is performed by semantic reasoning, combining high-level context information (e.g., residents’ postures and semantic locations) with triggered sensor events. The personalized stream of sensor events is processed by an incremental classifier, that is initialized with a limited amount of labeled ADLs. A novel cache-based active learning strategy is adopted to continuously improve the classifier. Our results on a dataset where up to 4 subjects perform ADLs at the same time show that MICAR reliably recognizes individual and joint activities while triggering a significantly low number of active learning queries.

In the general machine learning domain, solutions based on the integration of deep learning models with knowledge-based approaches are emerging. Indeed, such hybrid systems have the advantage of improving the recognition rate and the model’s interpretability. At the same time, they require a significantly reduced amount of labeled data to reliably train the model. However, these techniques have been poorly explored in the sensor-based Human Activity Recognition (HAR) domain. The common-sense knowledge about activity execution can potentially improve purely data-driven approaches. While a few knowledge infusion approaches have been proposed for HAR, they rely on rigid logic formalisms that do not take into account uncertainty. In this paper, we propose P-NIMBUS, a novel knowledge infusion approach for sensor-based HAR that relies on probabilistic reasoning. A probabilistic ontology is in charge of computing symbolic features that are combined with the features automatically extracted by a CNN model from raw sensor data and high-level context data. In particular, the symbolic features encode probabilistic common-sense knowledge about the activities consistent with the user’s surrounding context. These features are infused within the model before the classification layer. We experimentally evaluated P-NIMBUS on a HAR dataset of mobile devices sensor data that includes 14 different activities performed by 25 users. Our results show that P-NIMBUS outperforms state-of-the-art neuro-symbolic approaches, with the advantage of requiring a limited amount of training data to reach satisfying recognition rates (i.e., more than 80% of F1-score with only 20% of labeled data).

The recognition of human activities in sensorized smart-home environments enables a wide variety of healthcare applications, including the detection of early symptoms of cognitive decline. The most effective Human Activity Recognition (HAR) methods are based on supervised Deep Learning classifiers. Those models are usually considered as black boxes, and the rationale behind their decisions is difficult to understand for human beings. The recent advances in eXplainable Artificial Intelligence (XAI) offer promising tools to make HAR models more transparent. The state-of-the-art explainable HAR methods provide explanations for the output of classifiers that periodically predict the performed activity on short time windows (usually in the range of 15-60 seconds). However, non-technical users may be more interested in investigating explanations associated with complete activity instances (e.g., an instance of the cooking activity may last 30 minutes). Unfortunately, temporally extending the time window harms the recognition rate of HAR classifiers. In this paper, we propose DeXAR++: a novel method that generates explanations for human activity instances based on deep learning classifiers. The sensor data time windows used for classification are encoded as images. DeXAR++ aggregates the explanations generated by a computer-vision XAI approach on each time window to obtain a single explanation for approximated activity instances. Moreover, DeXAR++ includes a novel visualization approach particularly suitable for non-expert users. We evaluate DeXAR++ with both automatic and user-based evaluation methodologies on a public dataset of activities performed in smart-home environments, showing that our results outperform the ones obtained by state-of-the-art methods.

Sensor-based Human Activity Recognition (HAR) has been a hot topic in pervasive computing for several years mainly due to its applications in healthcare and well-being. Centralized supervised approaches reach very high recognition rates, but they incur privacy and scalability issues. Federated Learning (FL) has been recently proposed to mitigate these issues. Each subject only shares the weights of a personal model trained locally, instead of sharing data. A cloud server is in charge of aggregating the weights to generate a global model. However, since activity data is non-independently and identically distributed (non-IID), a single model may not be sufficiently accurate for a large number of diverse users. In this work, we propose FedCLAR, a novel federated clustering method for HAR. Based on the similarity of the local model updates, the cloud server in FedCLAR derives groups of users that exhibit similar ways of performing activities. For each group, FedCLAR uses a specialized global model to mitigate the non-IID problem. We evaluated FedCLAR on two well-known public datasets, showing that it outperforms state-of-the-art FL solutions.

Neuro-symbolic AI methods aim at integrating the capabilities of data-driven deep learning solutions with the ones of more traditional symbolic approaches. These techniques have been poorly explored in the sensor-based Human Activity Recognition (HAR) research field, even if they could lead to multiple benefits such as improving model interpretability and reducing the amount of labeled data that is necessary to reliably train the model. In this paper, we propose DUSTIN, a novel knowledge infusion approach for sensor-based HAR. DUSTIN concatenates the features automatically extracted by a CNN model from raw sensor data and high-level context data with the ones inferred by a knowledge-based reasoner. In particular, the symbolic features encode common-sense knowledge about the activities which are consistent with the context of the user, and they are infused within the model before the classification layer. We experimentally evaluated DUSTIN on a HAR dataset of mobile devices sensor data that includes 14 different activities performed by 26 users. Our results show that DUSTIN outperforms state-of-the-art neuro-symbolic approaches, with the advantage of requiring a limited amount of training data and training epochs to reach satisfying recognition rates.

Abstract One of the major open problems in sensor-based Human Activity Recognition (HAR) is the scarcity of labeled data. Among the many solutions to address this challenge, semi-supervised learning approaches represent a promising direction. However, their centralized architecture incurs in the scalability and privacy problems that arise when the process involves a large number of users. Federated learning (FL) is a promising paradigm to address these problems. However, the FL methods that have been proposed for HAR assume that the participating users can always obtain labels to train their local models (i.e., they assume a fully supervised setting). In this work, we propose FedAR: a novel hybrid method for HAR that combines semi-supervised and federated learning to take advantage of the strengths of both approaches. FedAR combines active learning and label propagation to semi-automatically annotate the local streams of unlabeled sensor data, and it relies on FL to build a global activity model in a scalable and privacy-aware fashion. FedAR also includes a transfer learning strategy to fine-tune the global model on each user. We evaluated our method on two public datasets, showing that FedAR reaches recognition rates and personalization capabilities similar to state-of-the-art FL supervised approaches. As a major advantage, FedAR only requires a very limited number of annotated data to populate a pre-trained model and a small number of active learning questions that quickly decrease while using the system, leading to an effective and scalable solution for the data scarcity problem of HAR.

Recognition of activities of daily living (ADLs) is an enabling technology for several ubiquitous computing applications. Most activity recognition systems rely on supervised learning to extract activity models from labeled datasets. A problem with that approach is the acquisition of comprehensive activity datasets, which is an expensive task. The problem is particularly challenging when focusing on complex ADLs characterized by large variability of execution. Moreover, several activity recognition systems are limited to offline recognition, while many applications claim for online activity recognition. In this paper, we propose POLARIS, a framework for unsupervised activity recognition. POLARIS can recognize complex ADLs exploiting the semantics of activities, context data, and sensors. Through ontological reasoning, our algorithm derives semantic correlations among activities and sensor events. By matching observed events with semantic correlations, a statistical reasoner formulates initial hypotheses about the occurred activities. Those hypotheses are refined through probabilistic reasoning, exploiting semantic constraints derived from the ontology. Our system supports online recognition, thanks to a novel segmentation algorithm. Extensive experiments with real-world datasets show that the accuracy of our unsupervised method is comparable to the one of supervised approaches. Moreover, the online version of our system achieves essentially the same accuracy of the offline version.

Recognizing the activities of daily living (ADLs) in multi-inhabitant settings is a challenging task. One of the major challenges is the so-called data association problem: how to assign to each user the environmental sensor events that he/she actually triggered? In this paper, we tackle this problem with a contextaware approach. Each user in the home wears a smartwatch, which is used to gather several high-level context information, like the location in the home (thanks to a micro-localization infrastructure) and the posture (e.g., sitting or standing). Context data is used to associate sensor events to the users which more likely triggered them. We show the impact of context reasoning in our framework on a dataset where up to 4 subjects perform ADLs at the same time (collaboratively or individually). We also report our experience and the lessons learned in deploying a running prototype of our method.

Providing city navigation instructions to people with motion disabilities requires the knowledge of urban features like curb ramps, steps or other obstacles along the way. Since these urban features are not available from maps and change in time, crowdsourcing this information from end-users is a scalable and promising solution. Our preliminary study on wheelchair users shows that an automatic crowdsourcing mechanism is needed, avoiding users' involvement.In this contribution we present a solution to crowdsource urban features from inertial sensors installed on a wheelchair. Activity recognition techniques based on decision trees are used to process the sensors data stream. Experimental results, conducted with data acquired from 10 real wheelchair users navigating in an outdoor environment show that our solution is effective in detecting urban features with precision around 0.9, while it is less reliable when classifying some fine-grained urban feature characteristics, like a step height. The experimental results also present our investigation aimed at identifying the best parameters for the given problem, which include number, position and type of inertial sensors, classifier type, segmentation parameters, etc.

Due to the emerging popularity of pervasive healthcare applications, tools for monitoring activities in smart homes are gaining momentum. Existing methods mainly rely on supervised learning algorithms for recognizing activities based on sensor data. A key issue with those approaches is the acquisition of comprehensive training sets of activities. Indeed, that task incurs significant costs in terms of manual labeling effort; moreover, labeling by external observers violates the individual's privacy. For these reasons, there is an increasing interest in unsupervised activity recognition methods. A popular approach relies on knowledge-based models expressed by ontologies of activities, environment and sensors. Unfortunately, those models require significant knowledge engineering efforts, and are often limited to a specific application. In this paper, we address the issues of existing methods by proposing a novel hybrid approach. Our intuition is that a generic knowledge-based model of activities can be refined to target specific individuals and environments by collaboratively acquiring feedback from inhabitants. Specifically, we propose a collaborative active learning method to refine correlations among sensor events and activity types that are initially extracted from a high-level ontology. Generic correlations are personalized to each target smart-home considering the similarity between the feedback target and the feedback provider in terms of environment and inhabitant's profiles. Moreover, thanks to this method, new sensors installed in the home are seamlessly integrated in the recognition framework. In order to reduce the burden of providing feedback, we also propose a technique to carefully select the conditions that trigger a feedback request. We conducted experiments with a real-world dataset and a generic ontology of activities. Results show that our hybrid method outperforms state-of-the-art supervised and unsupervised activity recognition techniques while triggering an acceptable number of feedback queries.

Recognition of activities of daily living (ADLs) is an enabling technology for several ubiquitous computing applications. In this field, most activity recognition systems rely on supervised learning methods to extract activity models from labeled datasets. An inherent problem of that approach consists in the acquisition of comprehensive activity datasets, which is expensive and may violate individuals' privacy. The problem is particularly challenging when focusing on complex ADLs, which are characterized by large intra- and inter-personal variability of execution. In this paper, we propose an unsupervised method to recognize complex ADLs exploiting the semantics of activities, context data, and sensing devices. Through ontological reasoning, we derive semantic correlations among activities and sensor events. By matching observed sensor events with semantic correlations, a statistical reasoner formulates initial hypotheses about the occurred activities. Those hypotheses are refined through probabilistic reasoning, exploiting semantic constraints derived from the ontology. Extensive experiments with real-world datasets show that the accuracy of our unsupervised method is comparable to the one of state of the art supervised approaches.

According to the World Health Organization, the rate of people aged 60 or more is growing faster than any other age group in almost every country, and this trend is not going to change in a near future. Since senior citizens are at high risk of non communicable diseases requiring long-term care, this trend will challenge the sustainability of the entire health system. Pervasive computing can provide innovative methods and tools for early detecting the onset of health issues. In this paper we propose a novel method relying on medical models, provided by cognitive neuroscience researchers, describing abnormal activity routines that may indicate the onset of early symptoms of mild cognitive impairment. A non-intrusive sensor-based infrastructure acquires low-level data about the interaction of the individual with home appliances and furniture, as well as data from environmental sensors. Based on those data, a novel hybrid statistical-symbolical technique is used to detect first the activities being performed and then the abnormal aspects in carrying out those activities, which are communicated to the medical center. Differently from related works, our method can detect abnormal behaviors at a fine-grained level, thus providing an important tool to support the medical diagnosis. In order to evaluate our method we have developed a prototype of the system and acquired a large dataset of abnormal behaviors carried out in an instrumented smart home. Experimental results show that our technique has a high precision while generating a small number of false positives.