While image inpainting has recently become widely available in image manipulation tools, existing approaches to video inpainting typically do not even achieve interactive frame rates yet as they are highly computationally expensive. Further, they either apply severe restrictions on the movement of the camera or do not provide a high-quality coherent video stream. In this paper we will present our approach to high-quality real-time capable image and video inpainting. Our PixMix approach even allows for the manipulation of live video streams, providing the basis for real Diminished Reality (DR) applications. We will show how our approach generates coherent video streams dealing with quite heterogeneous background environments and non-trivial camera movements, even applying constraints in real-time.

Diminished Reality (DR) allows to remove objects from a video stream while preseving a frame to frame coherence. Some approaches apply a pseudo-DR, allowing for the removal of objects only, while their background can be observed by a second camera. Most real DR approaches are highly computational expensive, not even allowing for interactive rates and/or apply significant restrictions regarding the uniformity of the background, or allow linear camera movements or even a static camera only. In this paper we will present a real-time capable Diminished Reality approach for high-quality image manipulation. Our approach achieves a significantly better performance and image quality for almost planar but non-trivial image backgrounds. Our Diminished Reality pipeline provides coherent video streams even for nonlinear camera movements due to the integration of homography based object tracking.

Feature based tracking approaches become more and more common for Augmented Reality (AR). However, most upcoming AR solutions are designed for mobile devices, in particular for smartphones and tablet computers, lacking sufficient performance for the execution of state-of-the art feature based approaches at interactive frame rates. In this paper we will present our approach significantly increasing the speed of feature based tracking, thus allowing for real-time applications even on mobile devices. Our approach applies a randomized pose initialization, is applicable to any feature detector and does not require any feature appearance attributes, such as descriptors or ferns.

While Augmented Reality has always been restricted to adding artificial content to the real environment, Diminished Reality allows for removing real world content. Existing approaches however, either require complex setups or are not applicable in real-time. In this paper we present our approach for removing real-world objects from a live video stream of the user's real environment. Our approach is based on a simple setup and neither requires any pre-processing nor any information on the structure and location of the objects to be removed or on their background. Our approach is based on the identification of the objects to be removed combined with an image completion and synthesis algorithm. The performance of our approach is one to two magnitudes better than that of previous work in the area of image completion, providing real-time object cancellation on standard laptop or tablet computers.

While tracking technologies based on fiducial markers have dominated the development of Augmented Reality (AR) applications for almost a decade, various real-time capable approaches to markerless tracking have recently been presented. However, most existing approaches do not yet achieve sufficient frame rates for AR on mobile phones or at least require an extensive training phase in advance.