Understanding behavior requires datasets that capture humans while carrying out complex tasks. The kitchen is an excellent environment for assessing human motor and cognitive function, as many complex actions are naturally exhibited in kitchens from chopping to cleaning. Here, we introduce the EPFL-Smart-Kitchen-30 dataset, collected in a noninvasive motion capture platform inside a kitchen environment. Nine static RGB-D cameras, inertial measurement units (IMUs) and one head-mounted HoloLens~2 headset were used to capture 3D hand, body, and eye movements. The EPFL-Smart-Kitchen-30 dataset is a multi-view action dataset with synchronized exocentric, egocentric, depth, IMUs, eye gaze, body and hand kinematics spanning 29.7 hours of 16 subjects cooking four different recipes. Action sequences were densely annotated with 33.78 action segments per minute. Leveraging this multi-modal dataset, we propose four benchmarks to advance behavior understanding and modeling through 1) a vision-language benchmark, 2) a semantic text-to-motion generation benchmark, 3) a multi-modal action recognition benchmark, 4) a pose-based action segmentation benchmark. We expect the EPFL-Smart-Kitchen-30 dataset to pave the way for better methods as well as insights to understand the nature of ecologically-valid human behavior. Code and data are available at https://amathislab.github.io/EPFL-Smart-Kitchen

Gamified rehabilitation tackles the problem of keeping patients engaged in, and motivated to do physical rehabilitation to improve its efficacy. However, with respect to standard rehabilitation, patients are freer to move about and may compensate their motion difficulties with parasite movements, which would greatly reduce the efficacy of the rehabilitation. To identify and characterize compensatory motions, we collected and analyzed video data of people playing the "tangible Pacman" game (an upper-limb rehabilitation game in which a patient moves a semi-passive robot, the "Pacman", on a map to collect 6 apples, while being chased by one or two autonomous robots, the "ghosts"). Participants include 10 healthy elderly adults and 10 chronic stroke patients, who played multiple runs of the game, with different sized maps and various game configurations. By analyzing the video recordings we successfully identified higher shoulder and torso lateral tilt compensation in stroke patients and developed a proof-of-concept compensatory motion detection system which relies on a wearable Inertial Measurement Unit and ROS to provide in-game, real-time visual feedback on compensation.

Rehabilitation aims to ameliorate deficits in motor control via intensive practice with the affected limb. Current strategies, such as one-on-one therapy done in rehabilitation centers, have limitations such as treatment frequency and intensity, cost and requirement of mobility. Thus, a promising strategy is home-based therapy that includes task specific exercises. However, traditional rehabilitation tasks may frustrate the patient due to their repetitive nature and may result in lack of motivation and poor rehabilitation. In this article, we propose the design and verification of an effective upper extremity rehabilitation game with a tangible robotic platform named Cellulo as a novel solution to these issues. We first describe the process of determining the design rationales to tune speed, accuracy and challenge. Then we detail our iterative participatory design process and test sessions conducted with the help of stroke, brachial plexus and cerebral palsy patients (18 in total) and 7 therapists in 4 different therapy centers. We present the initial quantitative results, which support several aspects of our design rationales and conclude with our future study plans.