This paper presents a novel approach to object recognition involving a sparse 2D model and matching using video. The model is generated on the basis of geometry and image measurables only. We first identify the underlying topological structure of an image dataset containing different views of the objects and represent it as a neighborhood graph. The graph is then refined by identifying redundant images and removing them using morphing. This gives a smaller dataset leading to reduced space requirements and faster matching. Finally we exploit motion continuity in video and extend our algorithm to perform matching based on video input and demonstrate that the results obtained using a video sequence are much robust than using a single image. Our approach is novel in that we do not require any knowledge of camera calibration or viewpoint while generating the model. We also do not assume any constraint on motion of object in test video other than following a smooth trajectory.