Contactless exercise monitoring is a new trend that makes people feel more comfortable and unconstrained. However, the decrease in accuracy of the pulse rate measurement caused by large motion artifacts is an urgent problem to be solved. In this paper, we proposed a novel approach to monitor step count and improve the accuracy of remote pulse rate measurement based on the image detection method. We designed a chrominance-based adaptive filter and normalization (CADN) method and a domain selection scheme (DSS) to enhance the accuracy of contactless pulse rate measurement during exercise. Various exercises such as biking, stepping, and treadmill running were conducted to evaluate motion robustness of the proposed CADN + DSS and the accuracy of step counts. The results reveal that the detection rates of the proposed step count method are 99.52% and 99.77% for stepping and treadmill exercise, respectively. The pulse rate accuracy is compared with two state-of-the-art algorithms-chrominance and chrominance-based adaptive filter. The results show the proposed CADN+DSS method provides a lower discrepancy between the detected pulse rate and a ground-truth device (Polar H7) for all activities. We expand the scope of contactless measurement for physical activity detection and develop an unfettered step count and pulse rate measurement method in exercise. Therefore, step count and pulse rate can be measured synchronously without relying on any contact sensors.