In this paper, we present a comprehensive dataset comprising 37.9 hours of sensor data collected from humanoid robots, including 18.3 hours of walking and 2,519 recorded falls. This extensive dataset is a valuable resource for various robotics and machine learning applications. Leveraging this data, we propose RePro-TCN, a Temporal Convolutional Network (TCN) enhanced with two novel extensions: Relaxed Loss Formulation and Progressive Forecasting. Predicting falls is a critical capability in humanoid robotics for implementing countermeasures such as lunging or stopping the walk. Thanks to the new dataset, we train RePro-TCN and demonstrate its superiority over previous approaches under real-world conditions that were previously unattainable.

Humans use a variety of approaches to reference objects in the external world, including verbal descriptions, hand and head gestures, eye gaze or any combination of them.The amount of useful information from each modality, however, may vary depending on the specific person and on several other factors.For this reason, it is important to learn the correct combination of inputs for inferring the best-fitting reference.In this paper, we investigate speaker-dependent and independent fusion strategies in a multimodal reference resolution task.We show that without any change in the modality models, only through an optimized fusion technique, it is possible to reduce the error rate of the system on a reference resolution task by more than 50%.