Ballistocardiograms (BCG) is a passive, non-contact heart rate detection technology that requires no action on the part of the individual. However, during the BCG signal acquisition process, the surface pressure generated by cardiac contraction is easily disturbed by external factors, and as people's health deteriorates, the j-peak (the main peak of the BCG signal) is no longer prominent. Our aim is to establish a non-contact, self-supervised heart rate detection method based on physiological information, to improve the accuracy and robustness of BCG heart rate detection under wider and more adverse conditions. The algorithm is guided by the heart rate estimation based on BCG itself, thereby reconstructing a signal with physiological significance. We also propose a heartbeat mapping algorithm based on Bidirectional Long Short-Term Memory Network (BiLSTM) for extracting global deep features, achieving real-time heartbeat prediction, and eliminating local deviations brought about by reconstruction. To verify the effectiveness of the proposed method, this paper evaluated 40 young subjects and 4 elderly subjects. Compared with the existing state-of-the-art methods, beat-to-beat heart rate estimation and heartbeat detection both performed excellently, surpassing most methods using precise labels. The experimental results show that the proposed method achieves effective heartbeat detection, demonstrating robustness and effectiveness in the face of unavoidable noise and variations.

As the demand for real-time signal processing increases in various fields, such as healthcare, artificial intelligence, machine learning, and scientific research, there is a need for more efficient methods to analyze large amounts of data. To address this challenge and explore the opportunities to accelerate different signal processing algorithms, this paper proposes the integration of graphics processing units (GPUs) with database management systems (DBMS) using the PostgreSQL server programming interface (SPI). The performance of the proposed method is evaluated by comparing central processing unit (CPU) and GPU approaches for feature extraction using a data processing pipeline for heart rate estimation from hydraulic bed sensor data. Furthermore, the paper analyzes timing metrics, usability, adaptability, and discusses precision differences between CPU and GPU code by performing different thread and block configurations.

or respiratory illnesses due to heart-related issues are often misdiagnosed, under-diagnosed or ignored at early stages. Continuous health monitoring using ambient sensors has the potential to ameliorate this problem for older adults at aging-in-place facilities. In this paper, we leverage continuous respiratory health data collected by using ambient hydraulic bed sensors installed in the apartments of older adults in aging-in-place Americare facilities to find data-adaptive indicators related to shortness of breath. We used unlabeled data collected unobtrusively over the span of three years from a COPD-diagnosed individual and used data mining to label the data. These labeled data are then used to train a predictive model to make future predictions in older adults related to shortness of breath abnormality. To pick the continuous changes in respiratory health we make predictions for shorter time windows (60-s). Hence, to summarize each day's predictions we propose an abnormal breathing index (ABI) in this paper. To showcase the trajectory of the shortness of breath abnormality over time (in terms of days), we also propose trend analysis on the ABI quarterly and incrementally. We have evaluated six individual cases retrospectively to highlight the potential and use cases of our approach.

With the increasing interconnection of smart sensors in long-term care facilities, the amount of data available for multi-modal Big Data analytics is advantageous. Using raw smart sensor data, researchers can derive and extract physiological features useful for health monitoring. Nonetheless, with the immense amount of smart sensor data, the ability to utilize these data for multi-modal analytics as the data grows presents a great challenge for researchers and long-term care facilities. This paper proposes a database design system for multi-modal derived time-series featured data (respiration and restlessness) by using techniques such as hierarchical time-indexed databases and dense numerical array storage. We present evaluations and findings for our proposed database system design for multi-modal time-series feature data to assess the various performance characteristics in data access time, storage, and usability; demonstrating an extremely scalable design and simple integration with existing analytic tools via SQL interfaces. Furthermore, we introduce a data-processing pipeline enabling Big Data analytics for multi-modal time-series feature data.

The ability to explain the predictions of machine learning models has become increasingly important, especially in healthcare applications. Streaming clustering is an effective tool to recognize normal baseline patterns and to detect early signs of changes in data streams. However, many streaming clustering algorithms are not designed to explain to the users how predictions are made. In this paper, we extend a streaming clustering algorithm, the sequential possibilistic Gaussian mixture model (SPGMM) for early detection of health change to provide algorithm explainability for the results. Four approaches are discussed to explain either the cluster differences or the reason for the algorithm warnings: (i) linguistic summarization for warnings; (ii) annotation distribution of clusters; (iii) SHapley Additive exPlanations (SHAP); (iv) functional health score. The four approaches are validated on one older adult monitored with a collection of motion, bed, and depth sensors over three years. The results obtained on the older adult show that the four approaches aid understanding of how the clusters and warnings are generated, providing strong support for clinicians to take corresponding actions.

In this paper, we examine the accuracy of a respiration rate algorithm using a radio frequency (RF) sensor while testing different sleeping postures. The goal is to use the algorithm in a contactless monitoring system for psychiatric patients in an unstructured hospital setting, in which the sleeping posture is unknown. Here, we first explore different postures in a lab setting. The application in the psychiatric hospital requires a non-wearable, non-accessible sensor that can track patients’ motion in the bed, estimate the patient’s respiration rate during sleep, and estimate the patient’s restless time during the night. The RF signal was collected for thirteen different sleeping postures, using a Vayyar Radar system with a carrier frequency of 6.014 GHz to capture all reflections by the FMCW (frequency modulated continuous wave) signal. Respiration belt data are used as ground truth. Preliminary results indicate that RF sensors can capture the respiration torso movement in different sleeping postures at a distance of 2.3m. The respiration rate estimation is 90% accurate when the torso area is directed toward the RF sensor, 87% accurate when the head is reversed to the opposite side of bed, and 86% accurate when sitting in bed reading.

Sleep stage classification can be used to monitor sleep quality and diagnose sleep disorders. Sleep disorders can be correlated to health conditions such as Alzheimer’s and Parkinson’s disease. This project uses a hydraulic bed sensor positioned under the mattress, as well as a deep learning approach, for sleep stage classification. Our motivation is to provide an automatic, non-invasive and more accessible method of classifying sleep stages by using deep learning to analyze data gathered from the hydraulic bed sensor. The test subjects for this project were elderly patients with sleep disorders. Polysomnography (PSG) data, the current gold standard, was also collected in a Sleep Lab to serve as the ground truth for the bed sensor data. In this study, sleep stages are categorized into 3 categories: Wake, Rapid Eye Movement (REM), and Non-Rapid Eye Movement (NREM). This paper uses a Convolutional Neural Network (CNN)-Long-Short Term Memory (LSTM) hybrid model with 2 CNNs of different filter sizes for feature extraction. These features are then fed into the LSTM for classification. Our results show an average accuracy of about 76% using the leave-one-subject-out (LOSO) validation. These results are promising and show that the hydraulic bed sensor combined with a deep learning approach is capable of providing an automatic and non-invasive method of classifying sleep stages.

The rapid aging of the population worldwide requires increased attention from healthcare providers and the entire society. For the elderly to live independently, many health issues related to old age, such as frailty and risk of falling, need increased attention and monitoring. When monitoring daily routines for older adults, it is desirable to detect the early signs of health changes before serious health events, such as hospitalizations, happen so that timely and adequate preventive care may be provided. By deploying multi-sensor systems in homes of the elderly, we can track trajectories of daily behaviors in a feature space defined using the sensor data. In this article, we investigate a methodology for tracking the evolution of the behavior trajectories over long periods (years) using high-dimensional streaming clustering and provide very early indicators of changes in health. If we assume that habitual behaviors correspond to clusters in feature space and diseases produce a change in behavior, albeit not highly specific, tracking trajectory deviations can provide hints of early illness. Retrospectively, we visualize the streaming clustering results and track how the behavior clusters evolve in feature space with the help of two dimension-reduction algorithms: Principal Component Analysis and t-distributed Stochastic Neighbor Embedding. Moreover, our tracking algorithm in the original high-dimensional feature space generates early health warning alerts if a negative trend is detected in the behavior trajectory. We validated our algorithm on synthetic data and tested it on a pilot dataset of four TigerPlace residents monitored with a collection of motion, bed, and depth sensors over 10 years. We used the TigerPlace electronic health records to understand the residents’ behavior patterns and to evaluate the health warnings generated by our algorithm. The results obtained on the TigerPlace dataset show that most of the warnings produced by our algorithm can be linked to health events documented in the electronic health records, providing strong support for a prospective deployment of the approach.

Non-invasive heart rate estimation is of great importance in daily monitoring of cardiovascular diseases. In this paper, a bidirectional long short term memory (bi-LSTM) regression network is developed for non-invasive heart rate estimation from the ballistocardiograms (BCG) signals. The proposed deep regression model provides an effective solution to the existing challenges in BCG heart rate estimation, such as the mismatch between the BCG signals and ground-truth reference, multi-sensor fusion and effective time series feature learning. Allowing label uncertainty in the estimation can reduce the manual cost of data annotation while further improving the heart rate estimation performance. Compared with the state-of-the-art BCG heart rate estimation methods, the strong fitting and generalization ability of the proposed deep regression model maintains better robustness to noise (e.g., sensor noise) and perturbations (e.g., body movements) in the BCG signals and provides a more reliable solution for long term heart rate monitoring.

In this paper, we describe two methods for learning the room structure via radio wave reflections for longitudinal health monitoring of older adults in a naturalistic home setting. The goal is to use these data as part of a monitoring system that can be easily installed in a home with minimal configuration, for the purpose of detecting very early signs of illness and functional decline. Two studies are conducted using RF (radio frequency) sensing. The first method learns the structure from the RF clutter patterns, and uses the beat frequency of the maximum peak in each chirp to calculate the wall position. The second method learns the room structure from active movement patterns, and uses the open space between the clusters of active movement patterns to estimate the possible wall locations. Comparing the two results from these methods provides a more robust wall location. In addition, a background filter is designed based on the wall position, and the activity level of people in different rooms is estimated using a fuzzy rule system applied to the RF motion data. We evaluate our approach in a naturalistic setting. Preliminary results indicate that RF sensors can be used to capture both room structure and overall activity patterns.

Stroke is a leading cause of long-term adult disability. Many stroke patients participate in rehabilitation programs prescribed by an occupational therapist to aid in recovery; however, occupational therapists rely on in-clinic assessments and often-unreliable self-assessments at home to track a patient's progress, limiting their ability to monitor how patients perform outside of a clinical setting. Our Daily Activity Recognition and Assessment System collects depth and skeletal data passively from within the patient's home to assess long-term recovery and provide metrics to an occupational therapist to allow for more individualized rehabilitation plans. Using data from a wall-mounted depth sensor, we adapt a hierarchical co-occurrence network to identify actions from pre-segmented skeletal data. We then perform assessments on the classified actions to track key recovery metrics: normalized jerk, speed of motions, and extent of reach. We also introduce novel filters to identify high quality data for analysis. Our sensor was installed in a stroke patient's kitchen for seven days, generating the first action recognition data set from a stroke patient in a naturalistic environment. We use this data in conjunction with the NTU-RGB-D data set to validate our recognition and assessment algorithms. We achieved 90.1% accuracy by replicating the results of the NTU-RGB-D data set and a maximum of 59.6% accuracy on our kitchen data set.

Data stream processing has gained much attention lately, in the era of big data. Streaming clustering is an effective tool to recognize normal baseline and to detect outliers in sequentially presented data. Perhaps more importantly would be the ability to predict that incoming data indicates movement towards a likely anomaly. In this paper, a Gaussian Mixture Model (GMM) is employed to represent different patterns in the data stream. The Sequential Possibilistic One-Means (SP1M) is used for initialization, and is incorporated into the GMM framework to recognize new mixture components in the data stream. The new proposed algorithm is called Sequential Possibilistic Gaussian Mixture Model (SPGMM). Furthermore, two methods of trajectory analysis, the “maximum typicality decline” and the “trend value measurement,” are used together with SPGMM to detect early signs of pattern changes before unusual pattern data arrive in the stream. The proposed SPGMM is tested on synthetic and real-world datasets, and is shown to have excellent performance on predicting early signs of pattern changes in these sequential streams.

The quality of sleep has a significant impact on health and life. This study adopts the structure of hierarchical classification to develop an automatic sleep stage classification system using ballistocardiogram (BCG) signals. A leave-one-subject-out cross validation (LOSO-CS) procedure is used for testing classification performance. Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Deep Neural Networks DNNs are complementary in their modeling capabilities; while CNNs have the advantage of reducing frequency variations, LSTMs are good at temporal modeling. A transfer learning (TL) technique is used to pre-train our CNN model on posture data and then fine-tune it on the sleep stage data. We used a ballistocardiography (BCG) bed sensor to collect both posture and sleep stage data to provide a non-invasive, in-home monitoring system that tracks changes in health of the subjects over time. Polysomnography (PSG) data from a sleep lab was used as the ground truth for sleep stages, with the emphasis on three sleep stages, specifically, awake, rapid eye movement (REM) and non-REM sleep (NREM). Our results show an accuracy of 95.3%, 84% and 93.1% for awake, REM and NREM respectively on a group of patients from the sleep lab.

Stroke is a leading cause of long-term adult disability. Stroke patients can recover through rehabilitation programs prescribed by occupational therapists (OT); however, an individualized rehabilitation program can reduce recovery times compared to traditional ones. In this paper, we propose a daily activity observation system (DAOS) that uses a Kinect v2 sensor to collect and retrieve motion data. The DAOS has a robust interface to extract depth and skeleton data, and supports data collection in an unstructured kitchen environment. Depth data are used to perform action recognition and track problematic movements, while skeleton data are used to calculate mean velocities of hand joints, max extensions, symmetry of hand movements, and other assessment metrics for therapists. Histogram of oriented 4D normals is used for action recognition. The action recognition accuracy is 97% on a multi-class kitchen action dataset. Through action recognition and accurate assessment, we present a novel system that can assist therapists and their ability to provide quality care to stroke patients.

The deployment of Gigabit Apps owing to their high-bandwidth and low-latency nature pushes the limits of today's end-to-end networking, and reveals new bottlenecks at multiple layers of networking, virtualization, application and user experience. In this paper, we use an exemplar smart health related Gigabit App use case viz., PhysicalTherapy-as-a-Service to show how a multi-layer instrumentation approach of measurement points was critical to successfully deploy our lab-tested App out to residential homes with Google Fiber connections. The salient instrumentation strategies involved an organized co-design method between the App Developer and Network Engineer roles, and a multi-domain network performance monitoring featuring perfSONAR extensions, both of which were realized through our Narada Metrics framework. Our instrumentation strategies engendered a “socio-technical tool” for co-ordination between multi-layer stakeholders in identifying and overcoming the intertwined bottlenecks, and in tuning the App performance. Our results highlight the new instrumentation and measurement challenges to foster multi-layer stakeholder collaboration, and provide rare insights to the budding Gigabit App developer community for performance engineering their Apps to serve residential users.

This paper proposes a spatial language generation system to communicate with a person about the location of an object in an indoor environment. It aims at finding a short, accurate and human-like description for building a natural and friendly interface between robots and humans using spatial language interaction. The system performs an inverse procedure to spatial language grounding which links natural commands to robot actions. The system works in two steps. It will first search for the best matching grounding model which describes the spatial relations between the target object and the references; then it will generate the natural language by mimicking a human's talking style. A corpus of 149 spatial language commands for an indoor environment fetch task is used to train the language generation model. An early-stage experiment is conducted and the results illustrate a potential for further development.

A method for detecting falls in the homes of older adults using the Microsoft Kinect and a two-stage fall detection system is presented. The first stage of the detection system characterizes a person's vertical state in individual depth image frames, and then segments on ground events from the vertical state time series obtained by tracking the person over time. The second stage uses an ensemble of decision trees to compute a confidence that a fall preceded on a ground event. Evaluation was conducted in the actual homes of older adults, using a combined nine years of continuous data collected in 13 apartments. The dataset includes 454 falls, 445 falls performed by trained stunt actors and nine naturally occurring resident falls. The extensive data collection allows for characterization of system performance under real-world conditions to a degree that has not been shown in other studies. Cross validation results are included for standing, sitting, and lying down positions, near (within 4 m) versus far fall locations, and occluded versus not occluded fallers. The method is compared against five state-of-the-art fall detection algorithms and significantly better results are achieved.

Complex activities such as instrumental activities of daily living (IADLs) can be identified by creating a hierarchical model of fuzzy rules. In this work, we present a framework to model a specific IADL — "making the bed". For this activity recognition, the need for a three level Fuzzy Inference System (FIS) model is shown. Simple features such as bounding box parameters were extracted from the foreground images and combined with 3D features extracted from the Kinect depth data. This was then fed as input to the three layered FIS for further analysis. Data collected from several participants were tested and evaluated. Such a framework can be used to model several other IADLS as well as basic activities of daily living (ADLs). Analysis of ADLs can be used to compare daily patterns in older adults to measure changes in behavior. This can then be used to predict health changes to assist older adults in leading independent lifestyles for longer time periods.

This paper proposes a system that allows the use of natural spatial language to control a robot performing a fetch task in an indoor environment. The system processes spatial referencing language and extracts a tree structure of language chunks. The spatial language system is then grounded to a robot navigation instruction in the form of a sequence of actions based on spatial references to furniture and room structure; the best navigation instruction is selected by scoring. In addition, the Reference-Direction-Target (RDT) model is proposed to represent indoor robot actions. To control the robot for the fetch task, a behavior model is designed based on the RDT model. An assistive robot has been designed and programmed based on this system. The proposed spatial language grounding model and robot behavior model are tested experimentally in three sets of experiments. Results show that the system enables a robot to follow spatial language commands in a physical indoor environment even if the referenced furniture items are re-positioned.

We present an approach for activity state recognition implemented on data collected from various sensors-standard web cameras under normal illumination, web cameras using infrared lighting, and the inexpensive Microsoft Kinect camera system. Sensors such as the Kinect ensure that activity segmentation is possible during the daytime as well as night. This is especially useful for activity monitoring of older adults since falls are more prevalent at night than during the day. This paper is an application of fuzzy set techniques to a new domain. The approach described herein is capable of accurately detecting several different activity states related to fall detection and fall risk assessment including sitting, being upright, and being on the floor to ensure that elderly residents get the help they need quickly in case of emergencies and ultimately to help prevent such emergencies.

We present algorithms to segment the activities of sitting and standing, and identify the regions of sit-to-stand (STS) transitions in a given image sequence. As a means of fall risk assessment, we propose methods to measure STS time using the 3-D modeling of a human body in voxel space as well as ellipse fitting algorithms and image features to capture orientation of the body. The proposed algorithms were tested on ten older adults with ages ranging from 83 to 97. Two techniques in combination yielded the best results, namely the voxel height in conjunction with the ellipse fit. Accurate STS time was computed on various STSs and verified using a marker-based motion capture system. This application can be used as part of a continuous video monitoring system in the homes of older adults and can provide valuable information to help detect fall risk and enable early interventions.

We present a unique occlusion and foreground overlap detection technique from depth sensor data using a fuzzy rule-based system. Features such as bounding box parameters and skeletonization were extracted from the foreground images and then input to the Fuzzy Inference System. Overlap and occlusion confidence measures were taken for each frame in the image sequence and compared against the extracted ground truth. This technique can help filter out occluded regions in the image sequence which, in an Eldercare environment, can then be used to compute accurate estimates of fall risk parameters such as stride time, stride length, and walking speed on a daily basis in in order to monitor the well-being of older adults in an ambient assisted living facility.

Methods and experimental results are presented for interpreting 3D spatial language descriptions used for human to robot communication in a fetch task. The work is based on human subject experiments in which spatial language descriptions were logged from younger and older adult participants. A spatial language model is proposed, and methods are presented for translating natural spatial language descriptions into robot commands that allow the robot to find the requested object. Robot command representation and robot behavior are also discussed. Experimental results compare path metrics of the robot system and human subjects in a common simulation environment. The overall success rate of the robot trials is 85%.

In this paper, we present an approach for modeling and comparing small sets of 2-D objects based on their spatial relationships. This situation can arise in the conflation of a hand- or machine-drafted map to a satellite image, or in the correspondence problem of matching two images taken under different viewing conditions. We focus on the specific problem of matching a sketched map containing several 2-D objects to hand-segmented satellite imagery. We define a similarity measure between the spatial configurations of two object sets, which uses attributed relational graphs to represent scene information. Objects are represented as graph nodes and edges are defined by the histograms of forces between object pairs. We develop a memetic algorithm based on a (μ+λ) evolution strategy to solve this scene-matching problem with three domain-specific local search operators that are compared experimentally.

In this paper, we propose a webcam-based system for in-home gait assessment of older adults. A methodology has been developed to extract gait parameters including walking speed, step time, and step length from a 3-D voxel reconstruction, which is built from two calibrated webcam views. The gait parameters are validated with a GAITRite mat and a Vicon motion capture system in the laboratory with 13 participants and 44 tests, and again with GAITRite for 8 older adults in senior housing. Excellent agreement with intraclass correlation coefficients of 0.99 and repeatability coefficients between 0.7% and 6.6% was found for walking speed, step time, and step length given the limitation of frame rate and voxel resolution. The system was further tested with ten seniors in a scripted scenario representing everyday activities in an unstructured environment. The system results demonstrate the capability of being used as a daily gait assessment tool for fall risk assessment and other medical applications. Furthermore, we found that residents displayed different gait patterns during their clinical GAITRite tests compared to the realistic scenario, namely a mean increase of 21% in walking speed, a mean decrease of 12% in step time, and a mean increase of 6% in step length. These findings provide support for continuous gait assessment in the home for capturing habitual gait.

This paper outlines a new study that investigates spatial language for use in human-robot communication. The scenario studied is a home setting in which the elderly resident has misplaced an object, such as eyeglasses, and the robot will help the resident find the object. We present results from phase I of the study in which we investigate spatial language generated to a human addressee or a robot addressee in a virtual environment.

In this paper, we present a methodology for analyzing passive infrared motion sensor data logged in the homes of seniors. The objective is to capture activity patterns that represent different health conditions. Recognizing changes in the activity patterns can then be used to provide early detection of health changes. A visualization of motion sensor data is introduced in the form of a density map that uses different colors to show varying levels of activity. For evaluating the activity density level accurately, time away from home is determined first using a system of fuzzy rules. In addition, a dissimilarity between two density maps is computed using texture features for automatically determining changes in activity patterns, which may indicate a health problem. The activity density maps are being used in an aging in place senior housing community to aid clinicians in early illness detection. Three case studies of elderly residents are included to illustrate how the density map and dissimilarity measure can be used to track general activity level and daily patterns over time, showing changes in physical, cognitive, and mental health.

In this paper, we present an improved evolutionary method for the task of locating a group of buildings based solely on their relative spatial relationships. This problem arises in the general text-to-sketch problem of conflating a hand or machine drafted sketch of building locations to a satellite image. We use the histograms of forces to capture the relative position information between buildings and develop a method to compare building sets. This represents an extension to our previous work, allowing for larger placement perturbations and changes in orientation.

This paper presents an approach to the task of locating a group of buildings based solely on their relative spatial relationships. This situation can occur in the problem of conflation of a hand or machine drafted map to a satellite image or in matching of two images taken under different viewing conditions (the correspondence problem). Of importance to us is the general text-to-sketch problem where a sketch of building locations must be matched to actual satellite imagery. Information about the nature of these relative positions is captured by the histograms of forces. In this paper, we consider a modified genetic algorithm that allows us to search for a specific group of buildings within a large geospatial database using the histograms of forces in the matching process. A novel mutation operator is introduced to adapt the standard GA to this environment.

The ability to rise from a chair is an important parameter to assess the balance deficits of a person. In particular, this can be an indication of risk for falling in elderly persons. Our goal is automated assessment of fall risk using video data. Towards this goal, we present a simple yet effective method of detecting transition, i.e. sit-to-stand and stand-to-sit, from image frames using fuzzy clustering methods on image moments. The technique described in this paper is shown to be robust even in the presence of noise and has been tested on several data sequences using different subjects yielding promising results.

Video-based tracking of contours on the human body has been shown to be useful for many applications, including gait and gesture recognition, posture estimation, and activity analysis. We present a contour tracking method that incorporates a novel edge feature and fuzzy contour template. We apply our method in tracking the motions of older adults exercising in a gym environment. The output of our system is a dynamic fuzzy representation of the spine angle of the subject. We show that the method described in this paper is capable of tracking contours even in cases where human silhouette extraction is poor.

As part of an interdisciplinary collaboration on elder-care monitoring, a sensor suite for the home has been augmented with video cameras. Multiple cameras are used to view the same environment and the world is quantized into nonoverlapping volume elements (voxels). Through the use of silhouettes, a privacy protected image representation of the human acquired from multiple cameras, a 3-D representation of the human is built in real time, called voxel person. Features are extracted from voxel person and fuzzy logic is used to reason about the membership degree of a predetermined number of states at each frame. Fuzzy logic enables human activity, which is inherently fuzzy and case-based, to be reliably modeled. Membership values provide the foundation for rejecting unknown activities, something that nearly all current approaches are insufficient in doing. We discuss temporal fuzzy confidence curves for the common elderly abnormal activity of falling. The automated system is also compared to a ground truth acquired by a human. The proposed soft computing activity analysis framework is extremely flexible. Rules can be modified, added, or removed, allowing per-resident customization based on knowledge about their cognitive and functionality ability. To the best of our knowledge, this is a new application of fuzzy logic in a novel approach to modeling and monitoring human activity, in particular, the well-being of an elderly resident, from video.

Video cameras are a relatively low-cost, rich source of information that can be used for ldquowell-beingrdquo assessment and abnormal event detection for the goal of allowing elders to live longer and healthier independent lives. We previously reported a soft-computing fall detection system, based on two levels from a hierarchy of fuzzy inference using linguistic summarizations of activity acquired temporally from a three dimensional voxel representation of the human found by back projecting silhouettes acquired from multiple cameras. This framework is extremely flexible and rules can be modified, added, or removed, allowing for per-resident customization based on knowledge about their cognitive and physical ability. In this paper, we show the flexibility of our activity analysis framework by extending it to additional common elderly activities and contextual awareness is added for reasoning based on location or static objects in the apartment.

Uncertainty measures model different types of uncertainty that are inherent in complex information systems. Measures that model either fuzzy or probabilistic uncertainty types have been explored in the literature. This paper shows that a combination of fuzzy and probabilistic uncertainty types, combined with the generalized maximum uncertainty principle, can be applied to time-series sequence classification and analysis. We present a novel algorithm that selects a wavelet from a wavelet library such that it best represents a time-series sequence, in a maximum uncertainty sense. Transformation coefficients are combined together in feature vectors that capture sequence temporal trends. A neural network is trained and tested using extracted gait sequence temporal features. Results have shown that models that combine together fuzzy and probabilistic uncertainty types better classify time-series gait sequences.

This paper explores several novel approaches to solve the Morris water maze task. In this spatial memory task, the robot must learn how to associate perceptual information with a particular location to aid in navigating to the goal. A self-organizing feature map (SOFM) is used to discretize the perceptual space. The robot must then learn to associate these perceptual states with an action used to navigate through the environment. Two navigational approaches are proposed. The first approach involves computing a probabilistic graph between SOFM nodes and then searching the graph to locate a path to the goal. The second approach uses temporal difference learning to learn the association between an SOFM node and an action that will direct it to the goal. The paper compares the effectiveness of these two approaches and discusses their respective utility.

The goal of this work is to determine the object correspondence between a sketched map and the scene depicted by the sketch, e.g., as represented by an occupancy grid map (OGM) built by a robot. We describe a novel method based on spatial relations for accomplishing this task. Our method is based on using the histogram of forces as scene descriptors. We generate a correspondence map between two scene descriptors and evaluate its confidence. From this map, we generate a one-to-one object correspondence map for the two scenes such that the object correspondence confidence value is maximized. Challenges include the fact that the two scenes may differ in terms of the shape and size of the objects, their orientation, and the objects might be shifted due to translation. The approach is evaluated using several hand drawn sketches that were collected as a part of a user study. We believe that the ability to perform scene matching will make our sketch interface more robust and easier to use, thereby providing us with a more intuitive way of communicating with the robots

Extracting a human silhouette from an image is the enabling step for many high-level vision processing tasks, such as human tracking and activity analysis. Although there are a number of silhouette extraction algorithms proposed in the literature, most approaches work efficiently only in constrained environments where the background is relatively simple and static. In a previous paper, we addressed some of the challenges in silhouette extraction and human tracking in a real-world unconstrained environment where the background is complex and dynamic. We extracted features from image regions, accumulated the feature information over time, fused high-level knowledge with low-level features, and built a time-varying background model. A problem with our system is that by adapting the background model, objects moved by a human are difficult to handle. In order to reinsert them into the background, we run the risk of cutting off part of the human silhouette, such as in a quick arm movement. In this paper, we develop a fuzzy logic inference system to detach the silhouette of a moving object from the human body. Our experimental results demonstrate that the fuzzy inference system is very efficient and robust.

This paper describes the application of fuzzy logic to the Short Physical Performance Battery (SPPB) test, a series of timed physical activities that have been created to evaluate, discriminate, and predict physical functional performance for both research and clinical purposes, primarily for physically impaired older adults. The original scoring system of SPPB test uses crisp time boundaries to assign the subject to discrete classes of performance. The crisp (and somewhat arbitrary) nature of the crisp thresholds can easily produce anomalies. Fuzzy Logic theory allows the natural description, in linguistic terms, of input/output relationships rather than relying on precise numerical threshold values. This advantage, dealing with the complicated systems in simple way, is the main reason why fuzzy logic theory is widely applied. In this paper, we offer a new approach for scoring the SPPB test. We demonstrate that in the proposed system, the Fuzzy Short Physical Performance Battery (FSPPB), we can improve the sensitivity and data distribution of the scoring system for the SPPB test. We present the procedures of constructing a fuzzy inference system using fuzzy logic to score the SPPB test and compare the original scoring system with our fuzzy scoring system. As part of a large project in technology for Eldercare, our goal is to accurately measure trends in physical performance of seniors over time.

One of the key components for natural interaction between humans and robots is the ability to understand the spatial relationships that exist in the natural world. Previous research has shown that modeling the 2D spatial relationships of FRONT, BEHIND, LEFT, RIGHT, and BETWEEN can be accomplished with results consistent with that of a human being. Upcoming research will involve a human subject study to investigate the use of spatial relationships in 3D space. This will be the first step in extending previous research of the 2D spatial relations into a 3D representation through the use of 3D object point clouds generated by the SIFT algorithm and stereo vision. This will allow for the enrichment of our human-robot dialog to include phrases such as "Bring me the coffee cup on top of the desk and to the right of the computer.

Extracting a human silhouette from an image is the enabling step for many high-level vision processing tasks, such as human tracking and activity analysis. Although there are a number of silhouette extraction and human tracking algorithms proposed in the literature, most approaches work efficiently only in constrained environments where the background is relatively simple and static. In this work, we propose to address the challenges in silhouette extraction and human tracking in a real-world unconstrained environment where the background is complex and dynamic. We extract features from image regions, accumulate the feature information over time, fuse the high-level knowledge with low-level features, and build a time-varying background model. We develop a fuzzy decision process to detach foreground moving objects from the human body. Our experimental results demonstrate that the algorithm is very efficient and robust.

In this paper, we describe a prototype interface that facilitates the control of a mobile robot team by a single operator, using a sketch interface on a tablet PC. The user sketches a qualitative map of the scene and includes the robots in approximate starting positions. Both path and target position commands are supported as well as editing capabilities. Sensor feedback from the robots is included in the display such that the sketch interface acts as a two-way communication device between the user and the robots. The paper also includes results of a usability study, in which users were asked to perform a series of tasks

In this paper, we discuss an important aspect of cognitive mobile robotics stemming from a new project in which an adaptive working memory is investigated for robot control and learning. Specifically, our approach is built on the premise that qualitative spatial reasoning is an appropriate framework to pose, learn, and solve navigational tasks. As such, the robot must be able to acquire and maintain landmarks in a form that facilitates learning and subsequent travel. Much research on landmark recognition has focused on either point landmarks or on landmark objects that come from segmentation and feature extraction. Here, we combine these approaches in the following sense. Potential landmark points are acquired in the point mode, but aggregations of them are utilized to represent "interesting" objects that can then be maintained throughout the path. In this paper, we investigate whether consistent aggregations can be maintained and thus serve as candidate chunks for the working memory system. The approach was tested on a video sequence of 1200 frames. Examples from this outdoor video are shown to corroborate the approach.

The goal of this work is to illustrate and evaluate a novel method for communicating with a mobile robot, namely, by drawing a sketch. The user draws a sketched route map to direct a mobile robot along a specified path. In this paper we focus on the navigation of the sketched path in the real environment. Challenges include sketch inaccuracies such as distortion or abstraction and low sensory resolution of the robot. Our method is based on utilizing spatial relations to extract a sequence of qualitative landmark states from the sketched map, which in turn the robot follows in a real environment to replicate the sketched route. Several examples are included.

In this correspondence, we introduce our work on sketch understanding, focusing here on the analysis of a sketched route map. A route map is drawn to help someone navigate along a path for the purpose of reaching a goal. A hand-sketched route map does not generally contain complete map information and is not necessarily drawn to scale, but yet it contains the correct qualitative information for route navigation. Here we propose a methodology for extracting a qualitative model of a sketched route map, based on human navigation strategies, using spatial relationships. Linguistic descriptions are generated from the sketch, both in the form of detailed descriptions at discrete path steps and also as a high-level route description. To describe the path linguistically, one must first be able to understand the path in a qualitative sense. We assert that the translation of a sketch into linguistic descriptions illustrates that the essential qualitative path knowledge has been extracted. The methodology is demonstrated using example sketches drawn on a handheld PDA.

In conversation, people often use spatial relationships to describe their environment, e.g., "There is a desk in front of me and a doorway behind it," and to issue directives, e.g., "go around the desk and through the doorway." In our research, we have been investigating the use of spatial relationships to establish a natural communication mechanism between people and robots, in particular, for novice users. In this paper, the work on robot spatial relationships is combined with a multimodal robot interface. We show how linguistic spatial descriptions and other spatial information can be extracted from an evidence grid map and how this information can be used in a natural, human-robot dialog. Examples using spatial language are included for both robot-to-human feedback and also human-to-robot commands. We also discuss some linguistic consequences in the semantic representations of spatial and locative information based on this work.

This paper discusses the use of combined uncertainty methods in the computation of wavelets that best represent horse gait signals. Combined uncertainty computes a composite of two types of uncertainties, fuzzy and probabilistic. First, we introduce fuzzy uncertainty properties and classes. Next, the gait analysis problem is discussed in the context of correctly classifying wavelet-transformed sound gait from lame horse gait signals. Continuous wavelets are selected using generalized information theory-related concepts that are enhanced through the application of uncertainty management models. Our experimental results show that models developed by maximizing combined uncertainty produce better results, in terms of neural network correct classification percentage, compared to those computed using only fuzzy uncertainty.

The goal of this work is to create a robot interface that allows a novice user to guide, control, and/or program a mobile robot to perform some task. To illustrate our interface we have chosen the example of robot navigation. We present a way to use a PDA computer to sketch a map and a robot route on this map. Qualitative instructions are then extracted from such a map to form a list of sequential steps that the robot has to follow to complete its task based on landmark states. We also demonstrate the feasibility of our approach on a Nomad200 robot simulator.

By utilizing morphological shared-weight neural networks (MSNN) that have been trained for face recognition, common access restriction points can be enhanced to identify particular individuals of interest. A trained MSNN is a computational intelligence structure that learns representation of a specific face that encodes in its connection weights the feature extraction and classification abilities needed to identify an instance of that face. It has been shown effective in analyzing images that contain the target in a group of faces, even with the target face at varying orientations and lighting, as well as occluded target faces. The experiments presented here show the possible application of the MSNN to perform watch-list scanning of faces as individuals pass through access screening areas.

In our research on intuitive means for humans and intelligent, mobile robots to collaborate, we use a multimodal interface that supports speech and gestural inputs. As a preliminary step to evaluate our approach and to identify practical areas for future work, we conducted a wizard-of-Oz pilot study with five participants who each collaborated with a robot on a search task in a separate room. The goal was to find a sign in the robot's environment with the word "FOO" printed on it. Using a subset of our multimodal interface, participants were told to direct the collaboration. As their subordinate, the robot would understand their utterances and gestures, and recognize objects and structures in the search space. Participants conversed with the robot through a wireless microphone and headphone and, for gestural input, used a touch screen displaying alternative views of the robot's environment to indicate locations and objects.

In human to human communication, a hand-drawn route map is often sketched to show a desired navigation path. In this paper, we describe a PDA sketching interface that can be used to direct a mobile robot along a specified path. Because sketched route maps are not drawn precisely or necessarily to scale, we do not attempt to analyze precise path information, but rather qualitative route information is extracted. The paper focuses on the front-end sketch understanding and includes a description of the interactive features, such as deleting, moving, and labeling landmarks. Results of a user evaluation are also presented, in which participants report that the sketching interface was as easy as using pencil and paper by a 2:1 margin.

In conversation, people often use spatial relationships to describe their environment, e.g., "There is a desk in front of me and a doorway behind it", and to issue directives, e.g., "Go around the desk and through the doorway." In our research, we have been investigating the use of spatial relationships to establish a natural communication mechanism between people and robots, in particular, for novice users. In this paper, the work on robot spatial relationships is combined with a multimodal robot interface developed at the Naval Research Lab. We show how linguistic spatial descriptions and other spatial information can be extracted from an evidence grid map and how this information can be used in a natural, human-robot dialog.

We show how linguistic expressions can be generated to describe the spatial relations between a mobile robot and its environment, using readings from a ring of sonar sensors. Our work is motivated by the study of human-robot communication for non-expert users. The eventual goal is to use these linguistic expressions for navigation of the mobile robot in an unknown environment, where the expressions represent the qualitative state of the robot with respect to its environment, in terms that are easily understood by human users. In the paper, we describe the histogram of forces and its application to sonar sensors on a mobile robot. Several environment examples are also included with the generated linguistic descriptions.

Being able to interact and communicate with robots in the same way we interact with people has long been a goal of AI and robotics researchers. In this paper, we propose a novel approach to communicating a navigation task to a robot, which allows the user to sketch an approximate map on a PDA and then sketch the desired robot trajectory relative to the map. State information is extracted from the drawing in the form of relative, robot-centered spatial descriptions, which are used for task representation and as a navigation language between the human user and the robot. Examples are included of two hand-drawn maps and the linguistic spatial descriptions generated from the maps.

We present two methods of rapidly (less than 1 ms) identifying contact formations from force sensor patterns, including friction and measurement uncertainty. Both principally use force signals instead of positions and detailed geometric models. First, fuzzy sets are used to model patterns and sensor uncertainty; membership functions are generated automatically from training data. Second, a neural network is used to generate confidence levels for each contact formation. Experimental results are presented for both classifiers, showing excellent results. New insights into the data sets are discussed, and a modified training method is presented that further improves the performance. The classification techniques are discussed in the context of robot programming by demonstration.

Robots have been used successfully in structured settings, where the environment is controlled; this research is inspired by the vision of robots moving beyond structured, controlled settings. The work focuses on the problem of teaching robots force-based assembly skills from human demonstration. To avoid position dependencies, force-based discrete states (contact formations) are used to describe qualitatively how contact is being made with the environment. Sensorimotor skills are modeled using a hybrid control model, which provides a mechanism for combining continuous low-level force control with higher-level discrete event control. A change in qualitative, discrete state constitutes an event and triggers a new control command to the robot, which moves the assembly toward a new contact formation. In this way, the skill execution is not dependent on absolute position but rather responds to changes in the force-based qualitative state. Experimental results are presented which validate the approach and show how skill acquisition can be accomplished even with an imperfect demonstration.

Current manufacturing methods for robotic-controlled assembly rely on accurate positioning to ensure task completion, often through the use of special fixtures and precise calibration of the workspace. The reliance on precision positioning to achieve proper alignment creates problems in both programming and control of contact-based tasks. As a means of addressing these problems, we have been investigating the use of qualitative contact states (QCS) for modeling and learning low-level, force-based skills. Sensorimotor skills are modeled using force-based discrete states, which describe qualitatively how contact is being made with the environment. The qualitative states can be identified from force signals by viewing them as projected clusters in the force sensor space. In this paper, we investigate the automatic clustering of force data by applying a competitive agglomeration algorithm to extract clusters which can be used for QCS classifier training. Experimental results are included using an automotive transmission assembly.

Contact formations have proven useful for programming robots by demonstration for operations involving contact. These techniques require real time recognition of contact formations. Single ended contact formation (SECF) classifiers using only the force/torque measured at the wrist of the robot have been shown to be quite effective for this purpose. To function properly, however, previous SECF classifiers have required a sizable training set and a constant pose between the force/torque sensor and the manipulated object. Thus, if an object is re-grasped and the pose changes, one expects to have to repeat the creation of the training set. We discuss the impact of sensor-object pose changes have on two successful classifiers. Experimental data shows that they perform poorly when sensor-object pose changes. We discuss a method to regain the performance of both classifiers while minimizing the retraining necessary.

Robots have been used successfully in structured settings, where the environment is controlled; this research is inspired by the vision of robots moving beyond the structured, controlled settings. The work focuses on the problem of learning low-level force-based assembly skills from human demonstration. To avoid position dependencies, force-based discrete states are used to describe qualitatively how contact is being made with the environment. Sensorimotor skills are modeled using a hybrid control model, which provides a mechanism for combining continuous low-level force control with higher level discrete event control. A change in qualitative, discrete state constitutes an event and triggers a new control command to the robot. In this way, the skill execution is not dependent on absolute position but rather responds to changes in the force-based qualitative state. Experimental results are presented which validate the approach and show how skill acquisition can be accomplished even with an imperfect demonstration.

The motivation behind this work is to transfer force-based assembly skills to robots by using human demonstration. For this purpose, we model the skills as a sequence of contact formations (which describe how a workpiece touches its environment) and desired transitions between contact formations. In this paper, we present a method of identifying single-ended contact formations from force sensor patterns. Instead of using geometric models of the workpieces, fuzzy logic is used to learn and model the patterns in the force signals. Membership functions are generated automatically from training data and then used by the fuzzy classifier. This classification scheme is used to learn desired sequences of contact formations which comprise a force-based skill. Experimental results are presented which use the technique to extract skill information from human demonstration data.

This paper presents a pattern recognition approach to identifying contact formations from force sensor signals. The approach is sensor-based and does not use geometric models of the workpieces. The design of a fuzzy classifier is described, when membership functions are generated automatically from training data. The technique is demonstrated using supervised learning. Test results are included for experiments using both rigid and non-rigid workpieces. The technique is discussed in the context of robot programming by human demonstration.

The Universities Space Automation and Robotics Consortium (USARC) was established to promote research into robotics and telerobotics for remote applications. An important part of the work of the Consortium has been the design and implementation of the Telerobotic Construction Set (TCS), which enables the building of modular telerobotic networks. Online and off-line performance analyses are integrated into TCS using quantitative analytical models. This provides the capabilities of measuring and predicting performance and system work loads quantitatively under different conditions, such as different levels of remote autonomy or different operator interfaces. This paper presents the methodology used in the performance/workload analysis and describes experiments, measuring teleoperation performance under time delay.