In Virtual Reality (VR) applications, interacting with distant objects relies heavily on mid-air object manipulation. Yet, the inherent distance between the user and the object often restricts movement precision. This paper introduces the Signed Distance Field (SDF) method for mid-air object manipulation and combines it with the ray casting interaction technique to investigate its effect on user performance and user experience. To increase movement accuracy, we leverage the speed-accuracy trade-off to dynamically adjust object manipulation speed based on the SDF algorithm’s output. Our study with 18 participants examines the effects of SDF across three different tasks with different complexity. Our results showed that ray casting with SDF reduces the number of errors in complex tasks without slowing down the participants and improves the user experience. We hope that our proposed assistive system, designed for tasks and applications, can be used as an interaction technique to enable more accurate manipulation of distant objects in fields like surgical planning, architecture, and games.