Heart rate is an important indicator of newborn health status. Conventional wired heart rate monitoring is affected by motion, can limit parental bonding and is prone to damage the fragile newborn skin. Video-based heart rate monitoring in adults has shown potential for the assessment of cardiac functions in optimal acquisition conditions. However, automated methods adapted for neonates, trained with limited sample sizes and capable of handling occlusions and variable illumination, still need to be explored. This work proposes a new deep-learning pipeline for video-based neonatal heart rate measurement, integrating color and infrared signals from mul-tiple neonatal body regions. We train deep learning models for pose estimation and heart rate detection in a pilot cohort of five neonates recorded in the clinic for up to 60 minutes. Our methods show generalization in a leave-one-out cross-validation scheme; the newborn pose estimation model presents high performance (average precision=0.85±0.07), while the heart rate detection model achieves a mean absolute error of 3.83±1.22 beats per minute compared to the electrocardiogram heart rate. Automated video-based heart rate measurement could provide a non-contact, low-cost complement to current HR monitoring technologies in the clinic or outpatient settings.