Regular inspection of potential risks in railroad surroundings is essential for operational safety. Uncrewed aerial vehicles (UAVs) offer an effective solution with aerial mobility and long-distance coverage. However, existing methods struggle with rare but extremely high risks characterized by limited samples and complex feature distributions. To address this, we propose SRLF (Sparse Representation Learning Framework), which decomposes sparse risks (SR) perception into three components: capture, excavation, and learning. First, Buffer Decouple Learning (BDL) decouples objectness from classification to capture and enhance foreground perception. Second, Feature Space Dynamic Sampling (FSDS) leverages adaptive quantity sampling from multivariate Gaussian distributions to excavate discriminative SR representations. Third, Triple Similarity Loss (TSL) constructs a triple comparison mechanism to contrastively shape uncertainty surfaces between SRs and common safety hazards (CSHs). Finally, extensive experiments conducted on the UAV-based railroad surroundings dataset demonstrate that SRLF can achieve a high detection rate of CSHs (95.6% mAP) while maintaining low miss-detection rate for SRs (81.9% Recall and 0.5% FPR95).