We propose a linear algorithm that is useful for realizing geometric registration between the view of a real scene and that of a virtual object in an image-based rendering framework. In a unified framework, the novel view synthesis of a virtual object based on three views' matching constraints and the recovery of the camera pose that is necessary for the base image selection can be performed. The feasibility of the algorithm is demonstrated by using ground-truth synthesized data and real scene data.