Obstructive sleep apnea (OSA) is a prevalent sleep disorder, affecting approximately one billion individuals globally. In this study, we aim to address the limitations of Polysomnography (PSG), the gold standard for OSA diagnosis, by developing SlAction, a non-intrusive system that utilizes infrared videos for OSA detection in daily sleep settings. Considering the privacy-sensitive nature of sleep videos, SlAction is designed to analyze data directly on the camera-capturing device, eliminating the need to transmit video data to a server. With the collaboration of clinical experts, we extensively analyze the largest dataset worldwide that we collected, establishing correlations between OSA events and human motions during sleep. Our novel approach achieved an OSA prediction performance with an F1 score of 0.88. Notably, even when running on a low-spec CPU, our SlAction operates approximately 75 times faster than previous work evaluated on high-performance GPU servers.