Cooling feedback has been shown to improve user experience in virtual reality. However, many existing methods face limitations, such as requiring skin contact, relying on airflow, having bulky components, or delivering unstable sensations. These issues limit their applicability to wearable systems, especially in scenarios demanding high responsiveness and perceptual consistency. We propose ChillFactor, a highly responsive, non-contact cooling interface for head-mounted displays (HMDs). It uses ultrasonic atomization of a predetermined alcohol solution to generate evaporative cooling near the skin, reducing airflow-induced artifacts and providing relatively consistent cold sensations. To validate the design, we first evaluated the system's cooling performance. At maximum output, it lowered skin temperature by approximately 2.6 °C within five seconds, a rate sufficient to induce an almost immediate perception of cold. We subsequently integrated ChillFactor into an HMD and conducted a user study with 12 participants in immersive VR scenarios. Participants reported perceiving the cold within 0.66 seconds and noted significantly higher levels of presence and realism compared to conditions without cooling feedback, demonstrating the system's effectiveness in enhancing immersion through rapid and adjustable thermal cues. This work provides a practical foundation for non-contact cooling feedback and expands the design space for multisensory interaction in virtual environments.