This paper investigates a new annotation technique that reduces significantly the amount of time to annotate training data for gesture recognition. Conventionally, the annotations comprise the start and end times, and the corresponding labels of gestures in sensor recordings. In this work, we propose a one-time point annotation in which labelers do not have to select the start and end time carefully, but just mark a one-time point within the time a gesture is happening. The technique gives more freedom and reduces significantly the burden for labelers. To make the one-time point annotations applicable, we propose a novel BoundarySearch algorithm to find automatically the correct temporal boundaries of gestures by discovering data patterns around their given one-time point annotations. The corrected annotations are then used to train gesture models. We evaluate the method on three applications from wearable gesture recognition with various gesture classes (10-17 classes) recorded with different sensor modalities. The results show that training on the corrected annotations can achieve performances close to a fully supervised training on clean annotations (lower by just up to 5 percent F1-score on average). Furthermore, the BoundarySearch algorithm is also evaluated on the ChaLearn 2014 multi-modal gesture recognition challenge recorded with Kinect sensors from computer vision and achieves similar results.

The machine recognition of user trajectories and activities is fundamental to devise context-aware applications for support and monitoring in daily life. So far, tracking and activity recognition were mostly considered as orthogonal problems, which limits the richness of possible context inference. In this work, we introduce the novel unified computational and representational framework S-SMART that simultaneously models the environment state (semantic mapping), localizes the user within this map (tracking), and recognizes interactions with the environment (activity recognition). Thus, S-SMART identifies which activities the user executes where (e.g., turning a handle next to a window ), and reflects the outcome of these actions by updating the world model (e.g., the window is now open ). This in turn conditions the future possibility of executing actions at specific places (e.g., closing the window is likely to be the next action at this location). S-SMART works in a self-contained manner and iteratively builds the semantic map from wearable sensors only. This enables the seamless deployment to new environments. We characterize S-SMART in an experimental dataset with people performing hand actions as part of their usual routines at home and in office buildings. The framework combines dead reckoning from a foot-worn motion sensor with template-matching-based action recognition, identifying objects in the environment (windows, doors, water taps, phones, etc.) and tracking their state (open/closed, etc.). In real-life recordings with up to 23 action classes, S-SMART consistently outperforms independent systems for positioning and activity recognition, and constructs accurate semantic maps. This environment representation enables novel applications that build upon information about the arrangement and state of the user’s surroundings. For example, it may be possible to remind elderly people of a window that they left open before leaving the house, or of a plant they did not water yet, using solely wearable sensors.

Sensor technology that is unobtrusively integrated into the clothing and equipment of an athlete can support the training of sport activities and monitor the athlete's progress. In this paper, we propose two wearable systems that support ice hockey players in the training of skating and shooting. These assistants measure the motions of players and compare them with reference executions of the same activities by professional players. A third system that we introduce monitors the player;s activities during a hockey game and creates a match report for objective performance measurement. For each of the three proposed applications, we present a prototype setup that we evaluate with amateur and professional players. The main findings are i) that with a skate-worn motion sensor and user-dependent training, eight skating motions can be spotted with an accuracy above 90%, ii) that stick-integrated sensors enable the measurement of relevant shot features, which differentiate professional from amateur athletes, and iii) that it is possible to spot important ice hockey activities in the signals of body-worn motion sensors worn during a game.

Freezing of gait (FoG) is a common gait impairment among patients with advanced Parkinson's disease. FoG is associated with falls and negatively impacts the patient's quality of life. Wearable systems that detect FoG in real time have been developed to help patients resume walking by means of rhythmic cueing. Current methods focus on detection, which require FoG events to happen first, while their prediction opens the road to preemptive cueing, which might help subjects to avoid freeze altogether. We analyzed electrocardiography (ECG) and skin-conductance (SC) data from 11 subjects who experience FoG in daily life, and found statistically significant changes in ECG and SC data just before the FoG episodes, compared to normal walking. Based on these findings, we developed an anomaly-based algorithm for predicting gait freeze from relevant SC features. We were able to predict 71.3% from 184 FoG with an average of 4.2 s before a freeze episode happened. Our findings enable the possibility of wearable systems, which predict with few seconds before an upcoming FoG from SC, and start external cues to help the user avoid the gait freeze.

Activity recognition systems based on body-worn motion sensors suffer from a decrease in performance during the deployment and run-time phases, because of probable changes in the sensors (e.g. displacement or rotatation), which is the case in many real-life scenarios (e.g. mobile phone in a pocket). Existing approaches to achieve robustness tend to sacrifice information (e.g. by rotation-invariant features) or reduce the weight of the anomalous sensors at the classifier fusion stage (adaptive fusion), ignoring data which might still be perfectly meaningful, although different from the training data. We propose to use adaptation to rebuild the classifier models of the sensors which have changed position by a two-step approach: in the first step, we run an anomaly detection algorithm to automatically detect which sensors are delivering unexpected data; subsequently, we trigger a system self-training process, so that the remaining classifiers retrain the “anomalous” sensors. We show the benefit of this approach in a real activity recognition dataset comprising data from 8 sensors to recognize locomotion. The approach achieves similar accuracy compared to the upper baseline, obtained by retraining the anomalous classifiers on the new data.

Human activity recognition is a core component of context-aware, ubiquitous computing systems. Traditionally, this task is accomplished by analyzing signals of wearable motion sensors. While such signals can effectively distinguish various low-level activities (e.g. walking or standing), two issues exist: First, high-level activities (e.g. watching movies or attending lectures) are difficult to distinguish from motion data alone. Second, instrumentation of complex body sensor network at population scale is impractical. In this work, we take an alternative approach of leveraging rich, dynamic, and crowd-generated self-report data as the basis for in-situ activity recognition. By treating the user as the "sensor", we make use of implicit signals emitted from natural use of mobile smart-phones. Applying an L1-regularized Linear SVM on features derived from textual content, semantic location, and time, we are able to infer 10 meaningful classes of daily life activities with a mean accuracy of up to 83.9%. Our work illustrates a promising first step towards comprehensive, high-level activity recognition using free, crowd-generated, social media data.

Template matching methods using Dynamic Time Warping (DTW) have been used recently for online gesture recognition from body-worn motion sensors. However, DTW has been shown sensitive under the strong presence of noise in time series. In sensor readings, labeling temporal boundaries of daily gestures precisely is rarely achievable as they are often intertwined. Moreover, the variation in daily gesture execution always exists. Therefore, here we propose two template matching methods utilizing the Longest Common Subsequence (LCSS) to improve robustness against such noise for online gesture recognition. Segmented LCSS utilizes a sliding window to define the unknown boundaries of gestures in the continuous coming sensor readings and detects efficiently a possibly shorter gesture within it. WarpingLCSS is our novel variant of LCSS to determine occurrences of gestures without segmenting data and performs one order of magnitude faster than the Segmented LCSS. The WarpingLCSS requires low-resource settings to process new arriving samples, thus it is suitable for real-time gesture recognition implemented directly on the small wearable devices. We compare our methods with the existing template matching methods based on Dynamic Time Warping (DTW) on two real-world gesture datasets from arm-worn accelerometer data. The results demonstrate that the LCSS approaches outperform the existing template matching approaches (about 12% in accuracy) in the dataset that suffers from boundary noise and execution variation.

The field of research on activity recognition is relatively young compared to others, like computer vision. In more mature fields, algorithms are usually tested on standardized, reference datasets. This way, algorithms coming from different groups can be tested in a fair manner, which accelerates the process of developing new knowledge. Collecting a reference dataset under realistic settings for activity recognition poses many challenges due to the large amount of sensors and sensor modalities which are needed to provide a sufficiently complete playground. We here report on some lessons learned while collecting such a reference dataset with a heterogeneous setup. We argue for the importance of a few principles to obtain a clean dataset, starting from the sampling and acquisition, down to the synchronization and labeling of the data.

Human activity recognition is a thriving research field. There are lots of studies in different sub-areas of activity recognition proposing different methods. However, unlike other applications, there is lack of established benchmarking problems for activity recognition. Typically, each research group tests and reports the performance of their algorithms on their own datasets using experimental setups specially conceived for that specific purpose. In this work, we introduce a versatile human activity dataset conceived to fill that void. We illustrate its use by presenting comparative results of different classification techniques, and discuss about several metrics that can be used to assess their performance. Being an initial benchmarking, we expect that the possibility to replicate and outperform the presented results will contribute to further advances in state-of-the-art methods.

Non-stationary data distributions are a challenge in activity recognition from body worn motion sensors. Classifier models have to be adapted online to maintain a high recognition performance. Typical approaches for online learning are either unsupervised and potentially unstable, or require ground truth information which may be expensive to obtain. As an alternative we propose a teacher signal that can be provided by the user in a minimally obtrusive way. It indicates if the predicted activity for a feature vector is correct or wrong. To exploit this information we propose a novel incremental online learning strategy to adapt a k-nearest-neighbor classifier from instances that are indicated to be correctly or wrongly classified. We characterize our approach on an artificial dataset with abrupt distribution change that simulates a new user of an activity recognition system. The adapted classifier reaches the same accuracy as a classifier trained specifically for the new data distribution. The learning based on the provided correct - error signal also results in a faster learning speed compared to online learning from ground truth. We validate our approach on a real world gesture recognition dataset. The adapted classifiers achieve an accuracy of 78.6% compared to the subject independent baseline of 68.3%.

Opportunistic sensing allows to efficiently collect information about the physical world and the persons behaving in it. This may mainstream human context and activity recognition in wearable and pervasive computing by removing requirements for a specific deployed infrastructure. In this paper we introduce the newly started European research project OPPORTUNITY within which we develop mobile opportunistic activity and context recognition systems. We outline the project's objective, the approach we follow along opportunistic sensing, data processing and interpretation, and autonomous adaptation and evolution to environmental and user changes, and we outline preliminary results.