Short video streaming platforms widely employ video prefetching to ensure users' quality of experience (QoE), but frequent user swipes lead to massive data wastage, creating a significant financial burden for vendors. Existing academic and industrial solutions fail to strike a balance, often sacrificing either data savings or the authentic user-perceived QoE. We introduce a framework that intelligently reduces streaming data consumption without compromising user experience. The core idea is to make prefetching decisions adaptive to both user swiping behavior and dynamic network conditions. Real-world evaluations show our framework significantly outperforms the state-of-the-art solutions on data wastage as well as user-perceived QoE.