Understanding behavior requires datasets that capture humans while carrying out complex tasks. The kitchen is an excellent environment for assessing human motor and cognitive function, as many complex actions are naturally exhibited in kitchens from chopping to cleaning. Here, we introduce the EPFL-Smart-Kitchen-30 dataset, collected in a noninvasive motion capture platform inside a kitchen environment. Nine static RGB-D cameras, inertial measurement units (IMUs) and one head-mounted HoloLens~2 headset were used to capture 3D hand, body, and eye movements. The EPFL-Smart-Kitchen-30 dataset is a multi-view action dataset with synchronized exocentric, egocentric, depth, IMUs, eye gaze, body and hand kinematics spanning 29.7 hours of 16 subjects cooking four different recipes. Action sequences were densely annotated with 33.78 action segments per minute. Leveraging this multi-modal dataset, we propose four benchmarks to advance behavior understanding and modeling through 1) a vision-language benchmark, 2) a semantic text-to-motion generation benchmark, 3) a multi-modal action recognition benchmark, 4) a pose-based action segmentation benchmark. We expect the EPFL-Smart-Kitchen-30 dataset to pave the way for better methods as well as insights to understand the nature of ecologically-valid human behavior. Code and data are available at https://amathislab.github.io/EPFL-Smart-Kitchen

Freezing of Gait (FoG) is a common condition in patients with Parkinson's disease (PD). It often leads to falls, and it severely affects the patient's quality of life. Although the neural mechanism of FoG is not well-known, wearable sensor-based assistive systems have been shown to effectively monitor FoG and help patients resume walking through rhythmic auditory cues when FoG is detected in real-time. With the development of technologies based on wearable sensors, accurate detection of FoG events is important for resume walking, clinical diagnosis, and treatment. Here, we propose a deep residual network to detect FoG. Offline analysis performed on a publicly available dataset with 10 patients shows the superiority of the proposed approach compared to traditional method (Moore's algorithm) and several deep learning techniques. Under 1s window size, the proposed method can achieve 85.7% sensitivity and 94.0% specificity. The geometric mean of the proposed method is 37.4% ahead of Moore's algorithm. Our approach can help improve the patients with PD quality of life and evaluate symptoms of FoG.