To help the computing device always understand the spacial relationship between the user's gesture and the ambient objects, a methodology is proposed to find the user's virtual eye center in the wearable camera coordinate system and then calculate accurately where a user is pointing at to perform the natural interaction. First, the wearable RGB-D sensor is affixed around the user forehead. A tool-free calibration is done by having the user move their fingers along their lines of sight from his eye center to the random selected targets. The fingertips are detected in the depth camera and then the interaction of these lines of sight is calculated. Then we present how to find where the user is pointing at in different scenarios with a depth map, a detected object and a controlled virtual element. To validate our methods, we perform a point-to-screen experiment. Results demonstrate that when a user is interacting with a display up to 1.5 meters away, our natural gesture interface has an average error of 2.1cm. In conclusion, the presented technique is a viable option for a reliable user interaction.