Sleep is essential for health and well-being, yet the gold-standard method for sleep stage assessment, polysomnography (PSG), requires overnight monitoring with numerous wired sensors and manual scoring, limiting its scalability and comfort in real-world settings. We present ViSS, the first end-to-end deep learning framework for fully non-contact sleep staging directly from raw infrared video. To capture the dual temporal structure of sleep, ViSS integrates per-epoch motion encoding and long-range sequence modeling in a compact hierarchical architecture, without relying on intermediate physiological signals. Evaluated on the largest infrared video dataset to date, including both healthy individuals and patients, ViSS achieves 80% accuracy and 0.78 macro F1-score using fewer than 7 million parameters. These results establish infrared video as a viable modality for automated sleep staging and highlight the potential of ViSS for scalable, privacy-preserving home-based assessment.