Telehealth rehabilitation systems aimed at providing physical and occupational therapy in the home face considerable challenges in terms of clinician and therapist buy-in, system and training costs, and patient and caregiver acceptance. Understanding the optimal workflow process to support practitioners in delivering quality care in partnership with assistive technologies is significant. We describe the iterative co-development of our hybrid physical/digital workflow process for assisting therapists with the setup and calibration of a computer vision based system for remote rehabilitation. Through an interdisciplinary collaboration, we present promising preliminary concepts for streamlining the translation of research outcomes into everyday healthcare experiences.

Physical rehabilitation typically requires therapists to make judgements about patient movement and functional improvement using subjective observation. This process makes it challenging to quantitatively track, compute and predict long-term patient improvement. We therefore propose a novel methodical approach to the standardized and interpretable quantification of patient movement during rehabilitation. We describe the expert-led development of a movement assessment rubric and an accompanying quantitative rating system. We present our movement capture and annotation computational tools designed to implement the rubric and assist therapists in the quantitative documentation and assessment of rehabilitation. We describe results from a movement capture study of the tool with nine stroke survivors and a movement rating study with four therapists. Findings from these studies highlight potential optimal methodical process paths for individuals engaged in capturing, understanding and predicting human movement performance.

Delivering long term, unsupervised stroke rehabilitation in the home is a complex challenge that requires robust, low cost, scalable, and engaging solutions. We present HOMER, an interactive system that uses novel therapy artifacts, a computer vision approach, and a tablet interface to provide users with a flexible solution suitable for home based rehabilitation. HOMER builds on our prior work developing systems for lightly supervised rehabilitation use in the clinic, by identifying key features for functional movement analysis, adopting a simplified classification assessment approach, and supporting transferability of therapy outcomes to daily living experiences through the design of novel rehabilitation artifacts. A small pilot study with unimpaired subjects indicates the potential of the system in effectively assessing movement and establishing a creative environment for training.

In this paper, we propose a general framework for tuning component-level kinematic features using therapists' overall impressions of movement quality, in the context of a home-based adaptive mixed reality rehabilitation (HAMRR) system. We propose a linear combination of nonlinear kinematic features to model wrist movement, and propose an approach to learn feature thresholds and weights using high-level labels of overall movement quality provided by a therapist. The kinematic features are chosen such that they correlate with the quality of wrist movements to clinical assessment scores. Further, the proposed features are designed to be reliably extracted from an inexpensive and portable motion capture system using a single reflective marker on the wrist. Using a dataset collected from ten stroke survivors, we demonstrate that the framework can be reliably used for movement quality assessment in HAMRR systems. The system is currently being deployed for large-scale evaluations, and will represent an increasingly important application area of motion capture and activity analysis.

This paper describes the Digital Culture BA degree: an engineering-arts undergraduate curriculum that combines competency-based education (CBE) and knowledge-oriented education (KOE) structures and related Pull-Push approaches. The degree has been offered for three years at Arizona State University, has 200 enrolled students and is continuing to grow. The degree embeds nine knowledge-oriented concentrations, each offered by a relevant participating department, within an interdisciplinary CBE context. The CBE part of the degree provides customized access to 40 interdisciplinary digital culture courses from 12 different academic units by connecting these courses through a set of core competencies. Access to courses is not determined by fixed prerequisites but rather by having one of several possible combinations of lower level competencies. This flexible curriculum is attractive to students, promotes integrative collaborative learning that inspires innovation, and prepares the type of engineering-arts experts and complex problem solvers that are currently needed in creative industries. This type of degree also presents several important challenges for educators and administrators. To address these challenges we developed project based assessment approaches, custom web-based software for advising a very diverse student body, as well as online tools for facilitating peer critique and feedback in large creative classrooms.

This paper presents an interactive home-based adaptive mixed reality system (HAMRR) for upper extremity stroke rehabilitation. This home-based system is an extension of a previously designed and currently implemented clinical system. The goal of HAMRR is to restore motor function to chronic stroke survivors by providing an engaging long-term reaching task therapy at home. The HAMMR system tracks movement of the wrist and torso, and provides real-time, post-trial, and post-set multimodal feedback to encourage the stroke survivor to self-assess his or her movement and engage in active learning of new movement strategies. This experiential media system uses a computational adaptation scheme to create a continuously challenging and unique multi-year therapy experience through the use of multiple, integrated audio and visual feedback streams. Novel design features include creating an over-arching story for the participant, the ability of the system to adapt the feedback over multiple time scales, and the ability for this system to integrate into any home.

This paper presents a novel system architecture and evaluation metrics for an Adaptive Mixed Reality Rehabilitation (AMRR) system for stroke patient. This system provides a purposeful, engaging, hybrid (visual, auditory and physical) scene that encourages patients to improve their performance of a reaching and grasping task and promotes learning of generalizable movement strategies. This system is adaptive in that it provides assistive adaptation tools to help the rehabilitation team customize the training strategy. Our key insight is to combine the patients, rehabilitation team, multimodal hybrid environments and adaptation tools together as an adaptive experiential mixed reality system.

In this article, we present a media adaptation framework for an immersive biofeedback system for stroke patient rehabilitation. In our biofeedback system, media adaptation refers to changes in audio/visual feedback as well as changes in physical environment. Effective media adaptation frameworks help patients recover generative plans for arm movement with potential for significantly shortened therapeutic time. The media adaptation problem has significant challenges—(a) high dimensionality of adaptation parameter space; (b) variability in the patient performance across and within sessions; (c) the actual rehabilitation plan is typically a non-first-order Markov process, making the learning task hard. Our key insight is to understand media adaptation as a real-time feedback control problem. We use a mixture-of-experts based Dynamic Decision Network (DDN) for online media adaptation. We train DDN mixtures per patient, per session. The mixture models address two basic questions—(a) given a specific adaptation suggested by the domain experts, predict the patient performance, and (b) given the expected performance, determine the optimal adaptation decision. The questions are answered through an optimality criterion based search on DDN models trained in previous sessions. We have also developed new validation metrics and have very good results for both questions on actual stroke rehabilitation data.

In this paper, we present a media adaptation framework for an immersive biofeedback system for stroke patient rehabilitation. In our biofeedback system, media adaptation refers to changes in audio/visual feedback as well as changes in physical environment. Effective media adaptation frameworks help patients recover generative plans for arm movement with potential for significantly shortened therapeutic time. The media adaptation problem has significant challenges - (a) high dimensionality of adaptation parameter space (b) variability in the patient performance across and within sessions(c) the actual rehabilitation plan is typically a non first-order Markov process, making the learning task hard.

This paper presents a novel real-time, multi-modal biofeedback system for stroke patient therapy. The problem is important as traditional mechanisms of rehabilitation are monotonous, and do not incorporate detailed quantitative assessment of recovery in addition to traditional clinical schemes. We have been working on developing an experiential media system that integrates task dependent physical therapy and cognitive stimuli within an interactive, multimodal environment. The environment provides a purposeful, engaging, visual and auditory scene in which patients can practice functional therapeutic reaching tasks, while receiving different types of simultaneous feedback indicating measures of both performance and results. There are three contributions of this paper - (a) identification of features and goals for the functional task (b) The development of sophisticated feedback (auditory and visual) mechanisms that match the semantics of action of the task. We additionally develop novel action-feedback coupling mechanisms. (c) New metrics to validate the ability of the system to promote learnability, stylization and engagement. We have validated the system for nine subjects with excellent results.

This paper presents a novel real-time, multi-modal biofeedback system for stoke patient therapy. The problem is important as traditional mechanisms of rehabilitation are monotonous, and do not incorporate detailed quantitative assessment of recovery in addition to traditional clinical schemes. We have been working on developing an experiential media system that integrates task dependent physical therapy and cognitive stimuli within an interactive, multimodal environment. The environment provides a purposeful, engaging, visual and auditory scene in which patients can practice functional therapeutic reaching tasks, while receiving different types of simultaneous feedback indicating measures of both performance and results. There are two contributions of this paper - (a) identification of features and goals for the functional task, (b) the development of sophisticated feedback (auditory and visual) mechanisms that match the semantics of action of the task.

In this paper we propose a framework for the computational extraction of time characteristics of a single choreographic work. Computational frameworks can aid in revealing non-salient compositional structures in modern dance. The computational extraction of such features allows for the creation of interactive works where the movement and the digital feedback (graphics, sound etc) are integrally connected at deep level of structures. It also facilitates a better understanding of the choreographic process. There are two key contributions in this paper: (a) a systematic analysis of the observable and non-salient aspects of solo dance form, (b) computational analysis of temporal phrasing structures guided by critical understanding of observable form. Our analysis results are excellent indicating the presence of rich, latent temporal organization in specific semi-improvisatory modern dance works that may provide rich structural material for interactivity.

Movement-based interactive dance has recently attracted great interest in the performing arts. While utilizing motion capture technology, the goal of this project was to design the necessary real-time motion analysis engine, staging, and communication systems for the completion of a movement-based interactive multimedia dance performance. The movement analysis engine measured the correlation of dance movement between three people wearing similar sets of retro-reflective markers in a motion capture volume. This analysis provided the framework for the creation of an interactive dance piece, Lucidity, which will be described in detail. Staging such a work also presented additional challenges. These challenges and our proposed solutions will be discussed. We conclude with a description of the final work and a summary of our future research objectives.

In this paper, we present a robust pause detection algorithm for an interactive dance system using 3D motion capture data. The algorithm uses joint angle and shape context as two types of feature vectors obtained from both labeled and unlabeled data streams to detect pauses of the mover in the space. By looking at variances of the feature vectors over a time window, the probability of pause is computed. Satisfactory interactive dance performances have been successfully created and presented using the reported system.

In this paper, we report a real-time gesture driven interactive system with multimodal feedback for performing arts, especially dance. The system consists of two major parts., a gesture recognition engine and a multimodal feedback engine. The gesture recognition engine provides real-time recognition of the performer's gesture based on the 3D marker coordinates from a marker-based motion capture system. According to the recognition results, the multimodal feedback engine produces associated visual and audio feedback to the performer. This interactive system is simple to implement and robust to errors in 3D marker data. Satisfactory interactive dance performances have been successfully created and presented using the reported system

Complex human motion sequences (such as dances) are typically analyzed by segmenting them into shorter motion sequences, called gestures. However, this segmentation process is subjective, and varies considerably from one human observer to another. In this paper, we propose an algorithm called hierarchical activity segmentation. This algorithm employs a dynamic hierarchical layered structure to represent the human anatomy, and uses low-level motion parameters to characterize motion in the various layers of this hierarchy, which correspond to different segments of the human body. This characterization is used with a naive Bayesian classifier to derive creator profiles from empirical data. Then those profiles are used to predict how creators will segment gestures in other motion sequences. When the predictions were tested with a library of 3D motion capture sequences, which were segmented by 2 choreographers they were found to be reasonably accurate.