A major challenge in developing robust and generalizable Human Activity Recognition (HAR) systems for smart homes is the lack of large and diverse labeled datasets. Variations in home layouts, sensor configurations, and individual behaviors further exacerbate this issue. To address this, we leverage the idea of embodied AI agents—virtual agents that perceive and act within simulated environments guided by internal world models. We introduce AgentSense, a virtual data generation pipeline in which agents live out daily routines in simulated smart homes, with behavior guided by Large Language Models (LLMs). The LLM generates diverse synthetic personas and realistic routines grounded in the environment, which are then decomposed into fine-grained actions. These actions are executed in an extended version of the VirtualHome simulator, which we augment with virtual ambient sensors that record the agents’ activities. Our approach produces rich, privacy-preserving sensor data that reflects real-world diversity. We evaluate AgentSense on five real HAR datasets. Models pretrained on the generated data consistently outperform baselines, especially in low-resource settings. Furthermore, combining the generated virtual sensor data with a small amount of real data achieves performance comparable to training on full real-world datasets. These results highlight the potential of using LLM-guided embodied agents for scalable and cost-effective sensor data generation in HAR.

Cross-modal contrastive pre-training between natural language and other modalities, e.g., vision and audio, has demonstrated astonishing performance and effectiveness across a diverse variety of tasks and domains. In this paper, we investigate whether such natural language supervision can be used for wearable sensor based Human Activity Recognition (HAR), and discover that--surprisingly--it performs substantially worse than standard end-to-end training and self-supervision. We identify the primary causes for this as: sensor heterogeneity and the lack of rich, diverse text descriptions of activities. To mitigate their impact, we also develop strategies and assess their effectiveness through an extensive experimental evaluation. These strategies lead to significant increases in activity recognition, bringing performance closer to supervised and self-supervised training, while also enabling the recognition of unseen activities and cross modal retrieval of videos. Overall, our work paves the way for better sensor-language learning, ultimately leading to the development of foundational models for HAR using wearables.

The ubiquitous availability of smartphones and smartwatches with integrated inertial measurement units (IMUs) enables straightforward capturing of human activities through collecting movement data. For specific applications of sensor-based human activity recognition (HAR), however, logistical challenges and burgeoning costs render especially the ground-truth annotation of such data a difficult endeavor, resulting in limited scale and diversity of datasets available for deriving effective HAR systems and less than ideal recognition capabilities. Transfer learning, i.e., leveraging publicly available labeled datasets to first learn useful representations that can then be fine-tuned using limited amounts of labeled data from a target domain, can alleviate some of the performance issues of contemporary HAR systems. Yet they can fail when the differences between source and target conditions are too large and/or only few samples from a target application domain are available—each of which are typical challenges in real-world human activity recognition scenarios. In this article, we present an approach for economic use of publicly available labeled HAR datasets for effective transfer learning. We introduce a novel transfer learning framework—Cross-Domain HAR—which follows the teacher-student self-training paradigm to more effectively recognize activities with very limited label information. It bridges conceptual gaps between source and target domains, including sensor locations and type of activities. Cross-Domain HAR enables substantial performance improvements over the state-of-the-art in sensor-based HAR scenarios. Through our extensive experimental evaluation on a range of benchmark datasets we specifically demonstrate the effectiveness of our approach for practically relevant few-shot activity recognition scenarios. We also present a detailed analysis into how the individual components of our framework affect downstream performance and provide practical suggestions for using the framework in real-world applications.

Wearable devices have become efficient tools for sports performance analysis. Professional systems heavily rely on the high-tech setup, which are expensive and privacy-invasive for amateur players. This paper addresses the gap between advanced professional systems and limited consumer options by proposing a low-cost, privacy-preserving approach for basketball shot detection and outcome prediction. We leverage accelerome-ter data from wrist-worn smartwatches, combined with audio recordings, to develop a system capable of identifying shot movements and predicting shot outcomes. The shot detection was achieved by a ID CNN model through accelerometer data and outcome classification was achieved by an audio classification model. We evaluated the system on 6 participants, and the macro F1 score for shot outcome classification in data streams are 81.53% and 78.07% on dominant hand and non-dominant hand, respectively. Our system opens up explorations in other domains, including medical or industrial activity recognition, where similar approaches can be applied.

Camera-based text entry using American Sign Language (ASL) fingerspelling has become more feasible due to recent advancements in recognition technology. However, there are numerous situations where camera-based text entry may not be ideal or acceptable. To address this, we present FingerSpeller, a solution that enables camera-free text entry using smart rings. FingerSpeller utilizes accelerometers embedded in five smart rings from TapStrap, a commercially available wearable keyboard, to track finger motion and recognize fingerspelling. A Hidden Markov Model (HMM) based backend with continuous Gaussian modeling facilitates accurate recognition as evaluated in a real-world deployment. In offline isolated word recognition experiments conducted on a 1,164-word dictionary, FingerSpeller achieves an average character accuracy of 91% and word accuracy of 87% across three participants. Furthermore, we demonstrate that the system can be downsized to only two rings while maintaining an accuracy level of approximately 90% compared to the original configuration. This reduction in form factor enhances user comfort and significantly improves the overall usability of the system.

We introduce a novel feature-based active learning approach that leverages leading feature selection methods from explainable AI (XAI) to elicit user corrections on both the classification label and the leading features contributing to the classification decision. Uniquely, our approach can handle such user corrections, which identify missing and superfluous features in a model-agnostic manner that can be applied to any supervised classification model. Our approach generates augmented samples in a fine-grained manner according to user corrections, and then adds them to the training data to teach relevant and irrelevant features. Our results show that our approach outperforms traditional active learning and a leading baseline in two domains, which includes improvement in classification performance from 0.572 to 0.637 given the same number of queries.

We posit that predicting sensor event sequence (SES) in a smart home can proactively support resident activities or recognize activities that have not been completed as intended and alert the resident. To realize this application, we propose a framework to support accurate SES prediction by leveraging online activity recognition. Our framework includes a novel method of applying a GPT2-based model, which is a sentence generation model, for SES prediction by taking advantage of the property that the relationship between ongoing activity and SES patterns is similar to the relationship between topic and word sequence patterns in NLP. We evaluated our method empirically using two real-world datasets where residents perform their usual daily activities. Our experimental results show the use of the GPT2-based model significantly improves the F1 value of SES prediction from 0.461 to 0.708 compared to the state-of-the-art method, and that using ongoing activity can further improve performance to 0.837. We found that the performance of the online activity recognition model required to achieve these SES predictions was about 80%, which could be achieved using simple feature engineering and modeling.

The dichotomy between the challenging nature of obtaining annotations for activities, and the more straightforward nature of data collection from wearables, has resulted in significant interest in the development of techniques that utilize large quantities of unlabeled data for learning representations. Contrastive Predictive Coding (CPC) is one such method, learning effective representations by leveraging properties of time-series data to setup a contrastive future timestep prediction task. In this work, we propose enhancements to CPC, by systematically investigating the encoder architecture, the aggregator network, and the future timestep prediction, resulting in a fully con-volutional architecture. Across sensor positions and activities, our method shows substantial improvements on four of six target datasets, demonstrating its ability to empower a wide range of application scenarios. Further, in the presence of very limited labeled data, our technique significantly outperforms both supervised and self-supervised baselines, positively impacting situations where collecting only a few seconds of labeled data may be possible. This is promising, as CPC does not require specialized data transformations or reconstructions for learning effective representations.

Smart home environments are designed to provide services that help improve the quality of life for the occupant via a variety of sensors and actuators installed throughout the space. Many automated actions taken by a smart home are governed by the output of an underlying activity recognition system. However, activity recognition systems may not be perfectly accurate, and therefore inconsistencies in smart home operations can lead users reliant on smart home predictions to wonder “Why did the smart home do that?” In this work, we build on insights from Explainable Artificial Intelligence (XAI) techniques and introduce an explainable activity recognition framework in which we leverage leading XAI methods (Local Interpretable Model-agnostic Explanations, SHapley Additive exPlanations (SHAP), Anchors) to generate natural language explanations that explain what about an activity led to the given classification. We evaluate our framework in the context of a commonly targeted smart home scenario: autonomous remote caregiver monitoring for individuals who are living alone or need assistance. Within the context of remote caregiver monitoring, we perform a two-step evaluation: (a) utilize Machine Learning experts to assess the sensibility of explanations and (b) recruit non-experts in two user remote caregiver monitoring scenarios, synchronous and asynchronous, to assess the effectiveness of explanations generated via our framework. Our results show that the XAI approach, SHAP, has a 92% success rate in generating sensible explanations. Moreover, in 83% of sampled scenarios users preferred natural language explanations over a simple activity label, underscoring the need for explainable activity recognition systems. Finally, we show that explanations generated by some XAI methods can lead users to lose confidence in the accuracy of the underlying activity recognition model, while others lead users to gain confidence. Taking all studied factors into consideration, we make a recommendation regarding which existing XAI method leads to the best performance in the domain of smart home automation and discuss a range of topics for future work to further improve explainable activity recognition.

Deviant eating behavior such as skipping meals and consuming unhealthy meals has a significant association with mental well-being in college students. However, there is more to what an individual eats. While eating patterns form a critical component of their mental well-being, insights and assessments related to the interplay of eating patterns and mental well-being remain under-explored in theory and practice. To bridge this gap, we use an existing real-time eating detection system that captures context during meals to examine how college students’ eating context associates with their mental well-being, particularly their affect, anxiety, depression, and stress. Our findings suggest that students’ irregularity or skipping meals negatively correlates with their mental well-being, whereas eating with family and friends positively correlates with improved mental well-being. We discuss the implications of our study in designing dietary intervention technologies and guiding student-centric well-being technologies.

Human activity recognition is progressing from automatically determining what a person is doing and when, to additionally analyzing the quality of these activities—typically referred to as skill assessment. In this chapter, we propose a new framework for skill assessment that generalizes across application domains and can be deployed for near-real-time applications. It is based on the notion of repeatability of activities defining skill. The analysis is based on two subsequent classification steps that analyze (1) movements or activities and (2) their qualities, that is, the actual skills of a human performing them. The first classifier is trained in either a supervised or unsupervised manner and provides confidence scores, which are then used for assessing skills. We evaluate the proposed method in two scenarios: gymnastics and surgical skill training of medical students. We demonstrate both the overall effectiveness and efficiency of the generalized assessment method, especially compared to previous work.

Objective: To estimate instantaneous oxygen uptake VO2with a small, low-cost wearable sensor during exercise and daily activities in order to enable monitoring of energy expenditure (EE) in uncontrolled settings. We aim to do so using a combination of seismocardiogram (SCG), electrocardiogram (ECG) and atmospheric pressure (AP) signals obtained from a minimally obtrusive wearable device. Methods: In this study, subjects performed a treadmill protocol in a controlled environment and an outside walking protocol in an uncontrolled environment. During testing, the COSMED K5 metabolic system collected gold standard breath-by-breath (B×B) data and a custombuilt wearable patch placed on the mid-sternum collected SCG, ECG and AP signals. We extracted features from these signals to estimate the B×B VO2data obtained from the COSMED system. Results: In estimating instantaneous VO2, we achieved our best results on the treadmill protocol using a combination of SCG (frequency) and AP features (RMSE of 3.68 ± 0.98 ml/kg/min and R2of 0.77). For the outside protocol, we achieved our best results using a combination of SCG (frequency), ECG and AP features (RMSE of 4.3 ± 1.47 ml/kg/min and R2of 0.64). In estimating VO2consumed over one minute intervals during the protocols, our median percentage error was 15.8% for the treadmill protocol and 20.5% for the outside protocol. Conclusion: SCG, ECG and AP signals from a small wearable patch can enable accurate estimation of instantaneous VO2in both controlled and uncontrolled settings. SCG signals capturing variation in cardio-mechanical processes, AP signals, and state of the art machine learning models contribute significantly to the accurate estimation of instantaneous VO2. Significance: Accurate estimation of VO2with a low cost, minimally obtrusive wearable patch can enable the monitoring of VO2and EE in everyday settings and make the many applications of these measurements more accessible to the general public.

Large-scale educational settings have been common domains for affect detection and recognition research. Most research emphasizes improvements in the accuracy of affect measurement to enhance instructors' efficiency in managing large numbers of students. However, these technologies are not designed from students' perspectives, nor designed for students' own usage. To identify the unique design considerations for affect sensors that consider student capacities and challenges, and explore the potential of affect sensors to support students' self-learning, we conducted semi-structured interviews and surveys with both online students and on-campus students enrolled in large in-person classes. Drawing on these studies we: (a) propose using affect data to support students' self-regulated learning behaviors through a "scaling for empowerment'' design perspective, (b) identify design guidelines to mitigate students' concerns regarding the use of affect data at scale, (c) provide design recommendations for the physical design of affect sensors for large educational settings.

Traditionally, the sliding window based activity recognition chain (ARC) has been dominating practical applications, in which features are carefully optimized towards scenario specifics. Recently, end-to-end, deep learning methods, that do not discriminate between representation learning and classifier optimization, have become very popular also for HAR using wearables, promising "out-of-the-box" modeling with superior recognition capabilities. In this paper, we revisit and analyze specifically the role feature representations play in HAR using wearables. In a systematic exploration we evaluate eight different feature extraction methods, including conventional heuristics and recent representation learning methods, and assess their capabilities for effective activity recognition on five benchmarks. Optimized feature learning integrated into the conventional ARC leads to comparable if not better recognition results as if using end-to-end learning methods, while at the same time offering practitioners more flexibility to optimize their systems towards specifics of wearables and their constraints and limitations.

Developing systems for Human Activity Recognition (HAR) using wearables typically relies on datasets that were manually annotated by human experts with regards to precise timings of instances of relevant activities. However, obtaining such data annotations is often very challenging in the predominantly mobile scenarios of Human Activity Recognition. As a result, labels often carry a degree of uncertainty-label jitter-with regards to: i) correct temporal alignments of activity boundaries; and ii) correctness of the actual label provided by the human annotator. In this work, we present a scheme that explicitly incorporates label jitter into the model training process. We demonstrate the effectiveness of the proposed method through a systematic experimental evaluation on standard recognition tasks for which our method leads to significant increases of mean F1 scores.

Affective computing aims to detect a person's affective state (e.g. emotion) based on observables. The link between affective states and biophysical data, collected in lab settings, has been established successfully. However, the number of realistic studies targeting affect detection in the wild is still limited. In this paper we present an exploratory field study, using physiological data of 11 healthy subjects. We aim to classify arousal, State-Trait Anxiety Inventory (STAI), stress, and valence self-reports, utilizing feature-based and convolutional neural network (CNN) methods. In addition, we extend the CNNs to multi-task CNNs, classifying all labels of interest simultaneously. Comparing the F1 score averaged over the different tasks and classifiers the CNNs reach an 1.8% higher score than the classical methods. However, the F1 scores barely exceed 45%. In the light of these results, we discuss pitfalls and challenges for physiology-based affective computing in the wild.

FingerPing is a novel sensing technique that can recognize various fine-grained hand poses by analyzing acoustic resonance features. A surface-transducer mounted on a thumb ring injects acoustic chirps (20Hz to 6,000Hz) to the body. Four receivers distributed on the wrist and thumb collect the chirps. Different hand poses of the hand create distinct paths for the acoustic chirps to travel, creating unique frequency responses at the four receivers. We demonstrate how FingerPing can differentiate up to 22 hand poses, including the thumb touching each of the 12 phalanges on the hand as well as 10 American sign language poses. A user study with 16 participants showed that our system can recognize these two sets of poses with an accuracy of 93.77% and 95.64%, respectively. We discuss the opportunities and remaining challenges for the widespread use of this input technique.

Feature extraction is a critical step in sliding-window based standard activity recognition chains. Recently, distribution based features have been introduced that showed excellent generalization capabilities across a wide range of application domains in human activity recognition scenarios based on body-worn sensors. These features capture the data distribution of individual analysis frames, yet they ignore temporal structure inherent to the signal of a frame. We explore four variants of adding temporal structure to distribution based features and demonstrate their potential for statistically significant improvements of activity recognition in general. The addition of temporal structure comes with a moderate increase in computational complexity rendering the proposed methods applicable to mobile and embedded scenarios.

Sliding window based activity recognition chains represent the state-of-the-art for many mobile and embedded scenarios as they are common in wearable computing. The length of the analysis frames is a crucial system parameter that directly influences the effectiveness of the overall approach. In this paper we present a method that optimizes the window length - individually for each target activity. Instead of employing a single, multi-class recognition system that is based on a generic window length, we combine individually optimized activity detectors into an Ensemble based recognition approach. We demonstrate the effectiveness of the approach through an experimental evaluation on eight benchmark datasets. The proposed method leads to significant improvements across a range of activity recognition application domains.

Washing hands is one of the easiest yet most effective ways to prevent spreading illnesses and diseases. However, not adhering to thorough handwashing routines is a substantial problem worldwide. For example, in hospital operations lack of hygiene leads to healthcare associated infections. We present WristWash, a wrist-worn sensing platform that integrates an inertial measurement unit and a Hidden Markov Model-based analysis method that enables automated assessments of handwashing routines according to recommendations provided by the World Health Organization (WHO). We evaluated Wrist-Wash in a case study with 12 participants. WristWash is able to successfully recognize the 13 steps of the WHO handwashing procedure with an average accuracy of 92% with user-dependent models, and with 85% for user-independent modeling. We further explored the system's robustness by conducting another case study with six participants, this time in an unconstrained environment, to test variations in the hand-washing routine and to show the potential for real-world deployments.

Deep Learning methods have become very attractive in the wider, wearables-based human activity recognition (HAR) research community. The majority of models are based on either convolutional or explicitly temporal models, or combinations of both. In this paper we introduce attention models into HAR research as a data driven approach for exploring relevant temporal context. Attention models learn a set of weights over input data, which we leverage to weight the temporal context being considered to model each sensor reading. We construct attention models for HAR by adding attention layers to a state-of-the-art deep learning HAR model (DeepConvLSTM) and evaluate our approach on benchmark datasets achieving significant increase in performance. Finally, we visualize the learned weights to better understand what constitutes relevant temporal context.

We present SeeSaw, a synchronous gesture interface for commodity smartwatches to support watch-hand only input with no additional hardware. Our algorithm, which uses correlation to determine whether the user is rotating their wrist in synchrony with a tactile and visual prompt, minimizes false-trigger events while maintaining fast input during situational impairments. Results from a 12 person evaluation of the system, used to respond to notifications on the watch during walking and simulated driving, show interaction speeds of 4.0 s - 5.5 s, which is comparable to the swipe-based interface control condition. SeeSaw is also evaluated as an input interface for watches used in conjunction with a head-worn display. A six subject study showed a 95% success rate in dismissing notifications and a 3.57 s mean dismissal time.

Extensions to auto-context segmentation are proposed and applied to segmentation of multiple organs in porcine offal as a component of an envisaged system for post-mortem inspection at abbatoir. In common with multi-part segmentation of many biological objects, challenges include variations in configuration, orientation, shape, and appearance, as well as inter-part occlusion and missing parts. Auto-context uses context information about inferred class labels and can be effective in such settings. Whereas auto-context uses a fixed prior atlas, we describe an adaptive atlas method better suited to represent the multimodal distribution of segmentation maps. We also design integral context features to enhance context representation. These methods are evaluated on a dataset captured at abbatoir and compared to a method based on conditional random fields. Results demonstrate the appropriateness of auto-context and the beneficial effects of the proposed extensions for this application.

User identification on interactive surfaces is a desirable feature that is not inherently supported by existing technologies. We have conducted an extensive survey of existing identification techniques, which led us to formulate a unified model for user identification. We start by introducing this model that (1) classifies existing user identification approaches in five categories according to the identification technology, (2) identifies eight characteristic identification system parameters, and (3) proposes a way for visualizing the system's characteristics as points on a radar chart to allow for quick comparison and contrast between systems. This model is then used to present our survey of existing user identification approaches and visualize their characteristics, highlighting their strengths and limitations. The model also makes it possible to visually represent requirements of systems that require user identification, identify existing approaches that can meet an application's requirements, and help report on and evaluate new approaches to user identification systematically.

Mass adoption and innovation in the field of the Internet of Things has transformed the environments we live in, from stale siloes of technologies into rich interactive playgrounds. Nevertheless, the vast majority of surface area in these spaces are being overlooked and under-utilized in today's research. Surface imaging provides the means to extend and include typically out-of-reach, disconnected objects into these playgrounds. However, existing surface imaging technologies are impractical to embed in everyday environments, restricting researchers from exploring the design and interaction opportunities they can afforded these spaces. In this paper, we propose IRIS, a modular surface imaging prototype capable of providing scalable, low-cost, high-resolution surface imaging. We describe a real-world case study where IRIS is used to identify and track fresh fruit produce being prepared -- a task that is typical infeasible with existing technologies. Through IRIS, we hope to enable the community to exploit these under-explored surface areas and enhance the rich, interactive, connected environments we inhabit.

The application of Organic User Interface (OUI) technologies will revolutionize interior design, through the development of interactive and actuated surfaces, furnishings and decorative artefacts. However, to adequately explore these new design landscapes we must support multidisciplinary collaboration between Architects, Interior Designers and Technologists. Herein, we present the results of two workshops, with a total of 45 participants from the disciplines of Architecture and Interior Design, supported by a group of HCI researchers. Our objective was to study how design disciplines can productively engage with smart materials as a design resource using an evolving set of techniques to prototype new interactive interior spaces. Our paper reports on our experiences across the two workshops and contributes an understanding of techniques for supporting multidisciplinary collaboration when designing interactive interior spaces.

Organic User Interfaces (OUIs) are flexible, actuated interfaces characterized by being aesthetically pleasing, intuitively manipulated and ubiquitously embedded in our daily life. In this paper, we critically survey the state-of-the-art for OUIs in interactive architecture research at two levels: 1) Architecture and Landscape; and 2) Interior Design. We postulate that OUIs have specific qualities that offer great potential for building interactive interiors and entire architectures that have the potential to -finally- transform the vision of smart homes and ubiquitous computing environments (calm computing) into reality. We formulate a manifesto for OUI Architecture in both exterior and interior design, arguing that OUIs should be at the core of a new interdisciplinary field driving research and practice in architecture. Based on this research agenda we propose concerted efforts to be made to begin addressing the challenges and opportunities of OUIs. This agenda offers us the strongest means through which to deliver a future of interactive architecture.

Difficulties can arise from the segmentation of three-dimensional objects formed by multiple non-rigid parts represented in two-dimensional images. Problems involving parts whose spatial arrangement is subject to weak restrictions, and whose appearance and form change across images, can be particularly challenging. Segmentation methods that take into account spatial context information have addressed these types of problem, which often involve image data of a multi-modal nature. An attractive feature of the auto-context (AC) technique is that a prior "atlas", typically obtained by averaging multiple label maps created by experts, can be used as an initial source of contextual data. However, a prior obtained in this way is likely to hide the inherent multi-modality of the data. We propose a modification of AC in which a probabilistic atlas of part locations is iteratively improved and made available as an additional source of information. We illustrate this technique with the problem of segmenting individual organs in images of pig offal, reporting statistically significant improvements in relation to both conventional AC and a state-of-the-art technique based on conditional random fields.

Management of Parkinson's Disease (PD) could be improved significantly if reliable, objective information about fluctuations in disease severity can be obtained in ecologically valid surroundings such as the private home. Although automatic assessment in PD has been studied extensively, so far no approach has been devised that is useful for clinical practice. Analysis approaches common for the field lack the capability of exploiting data from realistic environments, which represents a major barrier towards practical assessment systems. The very unreliable and infrequent labelling of ambiguous, low resolution movement data collected in such environments represents a very challenging analysis setting, where advances would have significant societal impact in our ageing population. In this work we propose an assessment system that abides practical usability constraints and applies deep learning to differentiate disease state in data collected in naturalistic settings. Based on a large data-set collected from 34 people with PD we illustrate that deep learning outperforms other approaches in generalisation performance, despite the unreliable labelling characteristic for this problem setting, and how such systems could improve current clinical practice.

The ability to generalise towards either new users or unforeseen behaviours is a key requirement for activity recognition systems in ubiquitous computing. Differences in recognition performance for the two application cases can be significant, and user-dependent performance is typically assumed to be an upper bound on performance. We demonstrate that this assumption does not hold for the widely used cross-validation evaluation scheme that is typically employed both during system bootstrapping and for reporting results. We describe how the characteristics of segmented time-series data render random cross-validation a poor fit, as adjacent segments are not statistically independent. We develop an alternative approach -- meta-segmented cross validation -- that explicitly circumvents this issue and evaluate it on two data-sets. Results indicate a significant drop in performance across a variety of feature extraction and classification methods if this bias is removed, and that prolonged, repetitive activities are particularly affected.

The next generation of human activity recognition applications in ubiquitous computing scenarios focuses on assessing the quality of activities, which goes beyond mere identification of activities of interest. Objective quality assessments are often difficult to achieve, hard to quantify, and typically require domain specific background information that bias the overall judgement and limit generalisation. In this paper we propose a framework for skill assessment in activity recognition that enables automatic quality analysis of human activities. Our approach is based on a hierarchical rule induction technique that effectively abstracts from noise-prone activity data and assesses activity data at different temporal contexts. Our approach requires minimal domain specific knowledge about the activities of interest, which makes it largely generalisable. By means of an extensive case study we demonstrate the effectiveness of the proposed framework in the context of dexterity training of 15 medical students engaging in 50 attempts of surgical activities.

Research on actuated interfaces has shown that people respond in certain socialized ways to interfaces that exhibit autonomous behaviours. We wished to explore the elements of design that drive people to regard an autonomous, interactive system as a social agent. To explore perceptions of autonomous behaviour in interfaces we created Diri - an autonomous helium balloon, used to document activity in spaces. We implemented two different technological sophistications of Diri, to compare the outcomes of our design decisions. We present our design process, technical details and evaluation workshops, concluding with implications for designing for autonomous behaviour in interfaces.

The sport of dressage has become very popular not only amongst professional athletes but increasingly also for private horse owners. In well-defined tests, rider and horse execute movements, which demonstrate the strength, endurance, and dexterity of the animal as well as the quality of the interaction between rider and horse. Whilst at a professional level intensive expert coaching to refine the skill set of horse and rider is standard, such an approach to progression is not usually viable for the large amateur population. In this paper we present a framework for automated generation of quality feedback in dressage tests. Using on-body sensing and automated measurement of key performance attributes we are able to monitor the quality of horse movements in an objective way. We validated the developed framework in a large-scale deployment study and report on the practical usefulness of automatically generated quality feedback in amateur dressage.

Mind maps, i.e. the spatial organization of ideas and concepts around a central topic and the visualization of their relations, represent a very powerful and thus popular means to support creative thinking and problem solving processes. Typically created on traditional whiteboards, they represent an important technique for collaborative brainstorming sessions. We describe a camera-based system to analyze hand-drawn mind maps written on a whiteboard. The goal of the presented system is to produce digital representations of such mind maps, which would enable digital asset management, i.e. storage and retrieval of manually created documents. Our system is based on image acquisition by means of a camera, followed by the segmentation of the particular whiteboard image focusing on the extraction of written context, i.e. the ideas captured by the mind map. The spatial arrangement of these ideas is recovered using layout analysis based on unsupervised clustering, which results in graph representations of mind maps. Finally, handwriting recognition derives textual transcripts of the ideas captured by the mind map. We demonstrate the capabilities of our mind map reading system by means of an experimental evaluation, where we analyze images of mind maps that have been drawn on whiteboards, without any further constraints other than the underlying topic. In addition to the promising recognition results, we also discuss training strategies, which effectively allow for system bootstrapping using out-of-domain sample data. The latter is important when addressing creative thinking processes where domain-related training data are difficult to obtain as they focus on novelty by definition.

The last fifty years have seen an exponential increase in the amount of technology embedded in and passing though home environments, creating spaces that are now technologically as well as socially, psychologically and environmentally complex. It is only in the last fifteen years that the study of Human-Computer Interaction in the home has entered the mainstream, with essential challenges and themes (Edwards & Grinter 2001, Blythe & Monk 2002) defined, and groundbreaking environments for research created and analyzed (Kidd et al., 1999). Diverse aspects within the domestic use of technologies, from childcare to health and wellbeing, to food, media, and energy, have now become the focus of detailed bodies of research in their own right. This trend has brought with it unique methodological concerns which need to be considered by those researching and designing interactions with technologies in the home (Coughlan et al., 2013). This special issue has emerged from an ongoing interest within the Human-Computer Interaction community in the practice of performing research and design activities in and about the home. While home based technology research has been performed and widely reported previously we focus on the methods themselves, exploring the development of novel methods, and the implication of applying existing methods in home environments.

We present data-driven techniques to augment Bag of Words (BoW) models, which allow for more robust modeling and recognition of complex long-term activities, especially when the structure and topology of the activities are not known a priori. Our approach specifically addresses the limitations of standard BoW approaches, which fail to represent the underlying temporal and causal information that is inherent in activity streams. In addition, we also propose the use of randomly sampled regular expressions to discover and encode patterns in activities. We demonstrate the effectiveness of our approach in experimental evaluations where we successfully recognize activities and detect anomalies in four complex datasets.

The Break-Time Barometer is a social awareness system, which was developed as part of an exploratory study of the use of situated sensing and displays to promote cohesion in a newly-dispersed workplace. The Break-Time Barometer specifically aims to use an ambient persuasion approach in order to encourage people to join existing breaks, which take place within this community. Drawing upon a privacy-sensitive ubiquitous sensing infrastructure, the system of-fers information about potentially break-related activity in social spaces within this workplace, including alerts when specific events are detected. The system was developed using a user-centered iterative design approach. A qualitative mixed methods evaluation of a full deployment identified a diverse set of reactions to both the system and the design goal, and further elaborated the challenges of designing for social connectedness in this complex workplace context.

Health and well-being of dogs, either domesticated pets or service animals, are major concerns that are taken seriously for ethical, emotional, and financial reasons. Welfare assessments in dogs rely on objective observations of both frequency and variability of individual behaviour traits, which is often difficult to obtain in a dog's everyday life. In this paper we have identified a set of activities, which are linked to behaviour traits that are relevant for a dog's wellbeing. We developed a collar-worn accelerometry platform that records dog behaviours in naturalistic environments. A statistical classification framework is used for recognising dog activities. In an experimental evaluation we analysed the naturalistic behaviour of 18 dogs and were able to recognise a total of 17 different activities with approximately 70% classification accuracy. The presented system is the first of its kind that allows for robust and detailed analysis of dog activities in naturalistic environments.

In recent years the sport of climbing has seen consistent increase in popularity. Climbing requires a complex skill set for successful and safe exercising. While elite climbers receive intensive expert coaching to refine this skill set, this progression approach is not viable for the amateur population. We have developed ClimbAX - a climbing performance analysis system that aims for replicating expert assessments and thus represents a first step towards an automatic coaching system for climbing enthusiasts. Through an accelerometer based wearable sensing platform, climber's movements are captured. An automatic analysis procedure detects climbing sessions and moves, which form the basis for subsequent performance assessment. The assessment parameters are derived from sports science literature and include: power, control, stability, speed. ClimbAX was evaluated in a large case study with 53 climbers under competition settings. We report a strong correlation between predicted scores and official competition results, which demonstrate the effectiveness of our automatic skill assessment system.

We describe FoodBoard, an instrumented chopping board that uses optical fibers and embedded camera imaging to identify unpackaged ingredients during food preparation on its surface. By embedding the sensing directly, and robustly, in the surface of a chopping board we also demonstrate how surface contact optical sensing can be used to realize the portability and privacy required of technology used in a setting such as a domestic kitchen. FoodBoard was subjected to a close to real-world evaluation in which 12 users prepared actual meals. FoodBoard compared favourably with existing unpackaged food recognition systems, classifying a larger number of distinct food ingredients (12 incl. meat, fruit, vegetables) with an average accuracy of 82.8%.

Ubiquitous computing technologies have traditionally striven to augment objects and the environment with sensing capabilities to enable them to respond appropriately to the needs of the individuals in the environment. This paper considers how such technologies might be harnessed to support language learning, and specifically Task-Based Learning (TBL). Task-Based Learning (TBL) involves doing meaningful tasks in a foreign language, emphasising the language's use in practice. TBL is seen as a highly engaging and motivating approach to learning a language, but is difficult to do in the classroom. Here, learners typically engage in activities that only simulate 'real-world' tasks, and as such only rehearse language use, rather than applying the language in practice. In this paper, we explore how an instrumented, context-aware environment whose design is grounded in pedagogical principles can support TBL. We present the French Kitchen, an instrumented kitchen for English speakers who are learning French, and describe a 46-participant evaluation of the kitchen. Based on the evaluation, we provide a set of design recommendations for those building instrumented systems for TBL.

The Internet of Things connects digital information sources with physical objects - which transforms an artifact from being a passive object into a 'thing' that may link to data, store data and even offer data to users. Digital Object Memories (DOMe) comprise hardware and software components, which together provide an open and universal platform for capturing, associating, and interacting with the digital information of connected objects - including storage, documentation and provision of information concerning actions an object is or might be involved in. The goal of this continuation of an established workshop series (predecessor events include DIPSO 2007-09 in conjunction with Ubicomp 2007-09, DOMe in conjunction with Intelligent Environment 2009, DOMe-IoT 2010 in conjunction with Ubicomp 2010, and NOMe-IoT in conjunction with Ubicomp 2011) is to twofold: 1.) initiate a conversation concerning the potential for objects to develop agency; and 2.) explore how data that is associated with an object may leverage real-world actions. Here, DOMe 2012 provides a hybrid interdisciplinary workshop format that will combine traditional presentations and discussion with practice-based experimentation.

Digital Object Memories (DOMes) comprise hardware and software components, which together provide an open and universal platform for capturing and interacting with the digital information of connected objects - including storage, documentation and provision of information concerning actions an object is or might be involved in. We envisage that connected objects equipped with DOMes will be enabled to make suggestions and propositions to human users - which implies that an object may have a level of agency. The latter concept is a striking possibility that may change the way that we perceive, interact, and relate to objects. The goal of this established workshop series is to twofold: 1.) initiate a conversation concerning the potential for objects to develop agency; and 2.) explore how data that is associated with an object may leverage real-world actions. DOMe-IoT 2012 provides a hybrid interdisciplinary workshop format that will combine traditional presentations and discussion with practice-based experimentation.

Severe behavior problems of children with developmental disabilities often require intervention by specialists. These specialists rely on direct observation of the behavior, usually in a controlled clinical environment. In this paper, we present a technique for using on-body accelerometers to assist in automated classification of problem behavior during such direct observation. Using simulated data of episodes of severe behavior acted out by trained specialists, we demonstrate how machine learning techniques can be used to segment relevant behavioral episodes from a continuous sensor stream and to classify them into distinct categories of severe behavior (aggression, disruption, and self-injury). We further validate our approach by demonstrating it produces no false positives when applied to a publicly accessible dataset of activities of daily living. Finally, we show promising classification results when our sensing and analysis system is applied to data from a real assessment session conducted with a child exhibiting problem behaviors.

Activity recognition in the home has been long recognized as the foundation for many desirable applications in fields such as home automation, sustainability, and healthcare. However, building a practical home activity monitoring system remains a challenge. Striking a balance between cost, privacy, ease of installation and scalability continues to be an elusive goal. In this paper, we explore infrastructure-mediated sensing combined with a vector space model learning approach as the basis of an activity recognition system for the home. We examine the performance of our single-sensor water-based system in recognizing eleven high-level activities in the kitchen and bathroom, such as cooking and shaving. Results from two studies show that our system can estimate activities with overall accuracy of 82.69% for one individual and 70.11% for a group of 23 participants. As far as we know, our work is the first to employ infrastructure-mediated sensing for inferring high-level human activities in a home setting.

We present GymSkill, a personal trainer for ubiquitous monitoring and assessment of physical activity using standard fitness equipment. The system records and analyzes exercises using the sensors of a personal smartphone attached to the gym equipment. Novel fine-grained activity recognition techniques based on pyramidal Principal Component Breakdown Analysis (PCBA) provide a quantitative analysis of the quality of human movements. In addition to overall quality judgments, GymSkill identifies interesting portions of the recorded sensor data and provides suggestions for improving the individual performance, thereby extending existing work. The system was evaluated in a case study where 6 participants performed a variety of exercises on balance boards. GymSkill successfully assessed the quality of the exercises, in agreement with the professional judgment provided by a physician. User feedback suggests that GymSkill has the potential to serve as an effective tool for motivating and supporting lay people to overcome sedentary, unhealthy lifestyles. GymSkill is available in the Android Market as `VMI Fit'.

We present the development of a socially acceptable cueing device for drooling in Parkinson's disease (PD). Sialorrhea, or drooling, is a significant problem associated with PD and has a strong negative emotional impact on those who experience it. Previous studies have shown the potential for managing drooling by using a cueing device. However, the devices used in these studies were deemed unacceptable by their users due to factors such as hearing impairment and social embarrassment. We conducted exploratory scoping work and high fidelity iterative prototyping with people with PD to get their input on the design of a cueing aid and this has given us an insight into challenges that confront users with PD and limit device usability and acceptability. The key finding from working with people with PD was the need for the device to be socially acceptable.

No abstract available.

This paper considers the problem of multi-modal saliency and attention. Saliency is a cue that is often used for directing attention of a computer vision system, e.g., in smart environments or for robots. Unlike the majority of recent publications on visual/audio saliency, we aim at a well grounded integration of several modalities. The proposed framework is based on fuzzy aggregations and offers a flexible, plausible, and efficient way for combining multi-modal saliency information. Besides incorporating different modalities, we extend classical 2D saliency maps to multi-camera and multi-modal 3D saliency spaces. For experimental validation we realized the proposed system within a smart environment. The evaluation took place for a demanding setup under real-life conditions, including focus of attention selection for multiple subjects and concurrently active modalities.

Since their first inception more than half a century ago, automatic reading systems have evolved substantially, thereby showing impressive performance on machine-printed text. The recognition of handwriting can, however, still be considered an open research problem due to its substantial variation in appearance. With the introduction of Markovian models to the field, a promising modeling and recognition paradigm was established for automatic offline handwriting recognition. However, so far, no standard procedures for building Markov-model-based recognizers could be established though trends toward unified approaches can be identified. It is therefore the goal of this survey to provide a comprehensive overview of the application of Markov models in the research field of offline handwriting recognition, covering both the widely used hidden Markov models and the less complex Markov-chain or n -gram models. First, we will introduce the typical architecture of a Markov-model-based offline handwriting recognition system and make the reader familiar with the essential theoretical concepts behind Markovian models. Then, we will give a thorough review of the solutions proposed in the literature for the open problems how to apply Markov-model-based approaches to automatic offline handwriting recognition.

The early detection of drug-induced organ toxicities is one of the major goals in safety pharmacology. Automating this process by classification of metabolic changes based on the analysis of1H nuclear magnetic resonance spectra improves this process. In this paper we propose an ensemble classification system based on support vector machines trained on diverse ldquoviewsrdquo on the data. These views are created by variation of preprocessing techniques and the final classification is achieved by voting on an optimized selection of all experts. Results of an experimental evaluation on a challenging data-set from industrial safety pharmacology show the effectiveness of the proposed approach w.r.t. the detection of drug-induced toxicity.

Ensembles of multiple (active) cameras yield an important ingredient in modern tracking and surveillance applications. They overcome the limited fields-of-view of single cameras, however, require robust procedures for handing over tracking tasks from one camera to another. In this paper a calibration-free procedure is proposed that allows for fast and reliable camera hand-over in Ambient Intelligence (AmI) applications. The approach is based on online acquisition of scenario-specific target models and especially solves the problem of significant changes in object view during hand-over. Real-world results acquired in an AmI environment prove the effectiveness of our technique.

We present a system that enables pointing-based unconstrained interaction with a smart conference room using an arbitrary multicamera setup. For each individual camera stream, areas exhibiting strong motion are identified. In these areas, face and hand hypotheses are detected. The detections of multiple cameras are then combined to 3D hypotheses from which deictic gestures are identified and a pointing direction is derived. This is then used to identify objects in the scene. Since we use a combination of simple yet effective techniques, the system runs in real-time and is very responsive. We present evaluation results on realistic data that show the capabilities of the presented approach.

The use of context dependent modeling units in handwriting recognition has been considered by many authors as promising substantial performance improvements in systems based on Hidden-Markov models. Interestingly, in the literature only a few approaches limited to online recognition are documented to make use of this technology. Therefore, we investigated whether context dependent modeling also offers advantages for offline recognition systems. The moderate performance improvements we achieved on a challenging unconstrained handwriting recognition task suggest that context dependent modeling can not easily be exploited for offline recognition. In this paper we will present the principles behind context dependent modeling and discuss the reasons for its limited applicability in recognizing offline handwriting data.

Most successful systems for the recognition of unconstrained handwriting currently rely on expert-crafted feature sets that compute local geometric properties from text images. However, by applying appearance based analysis techniques appropriate features could be derived from training data automatically. Therefore, in this paper, several different methods for computing appearance-based feature representations are investigated and compared to the performance of a state-of-the-art writer-independent recognition system based on geometric features. In extensive experiments, promising results were obtained on a challenging recognition task.

In this paper we describe system architectures for robust MLLR based environmental adaptation of continuous speech recognition systems. Inspired by an existing broadcast news transcription system we refined the identification of acoustic scenarios by using a combined GMM/HMM method. Thus environmental adaptation regarding arbitrary acoustic scenarios beyond speaker changes becomes possible. For deploying acoustic adaptation in interactive applications, such as human machine interaction, a time-synchronous adaptation approach is proposed. For different corpora the evaluation of our approaches shows significant improvements in recognition accuracy while satisfying the constraint of time-synchronous processing.