We have developed a virtual olfactory environment using an olfactory display and computational fluid dynamics (CFD) simulation to provide odor concentration change according to user’s movement in real-time in a virtual space. Although CFD can calculate the odor distribution in the complicated geometry, its computational cost was expensive and did not work in real-time in the previous study. In this study, real-time CFD based on GPU calculation was introduced to generate olfactory VR environment. Using real-time CFD we investigated influence of the user’s body with its location and orientation changing irregularly. In the sensory test to find the odor direction, the correct answer rate was over 70% when the body influence was considered, while it was just a chance hit when the influence was not considered. The experimental result indicates the usefulness of considering the effect of the user’s body since we cannot avoid that influence.