The vast majority of cardiovascular diseases are avoidable or treatable by preventive measures and early de-tection. To efficiently detect early signs and risk factors, car-diovascular parameters can be monitored continuously with small sensor patches, which improve the comfort of patients. However, processing the sensor data is a challenging task with the demanding needs of robustness, reliability, performance and efficiency. The field of deep learning has tremendous potential to provide a way to analyze cardiovascular sensor data to detect anomalies which alleviates the workload of doctors for more effective data interpretation. In this work, we show the feasibility of applying deep learning for the classification of synchronized electrocardiogram and phonocardiogram recordings under very tight resource constraints. Our model employs an early fusion of data and uses convolutional layers to solve the problem of binary classification of anomalies. Our experiments show that our model matches the accuracy of the current state-of-the-art model on the “training-a” dataset of the Physionet Challenge 2016 database while being more than two orders of magnitude more efficient in memory footprint and compute cost. Further, we demonstrate the applicability of our model on edge devices, such as sensor patches, by estimating processor performance, power consumption, and silicon area.