We propose a system that allows multiple groups of users at remote locations to naturally share a 3D image of real objects. All users can interactively observe a 3D stereoscopic image without distortion from their own viewpoints. The system basically consists of a combination of systems: imaging and display. The imaging system captures the real object's images from certain number of viewpoints which are used to generate stereoscopic images from arbitrary viewpoints based on an image-based rendering technique implemented on GPU. The display system interactively generates a 3D virtual image of the real object based on the IllusionHole technique. Users at one place just put the real object on their imaging system to capture a set of its images from sparse viewpoints around it; other groups of multiple users at remote places connected by networks observe its virtual image from arbitrary viewpoints on their display systems.