In this letter, a guidance strategy is proposed to optimize synthetic aperture sampling for occlusion removal with drones based on point-cloud representation of occluders. Prerecorded light detecting and ranging (LiDAR) scans are utilized to compute the visibility of fixed inspection regions on the ground that are intended for recurrent monitoring from the air. By utilizing Helmholtz reciprocity, the drone-collected LiDAR scans are used to computationally obtain a reciprocal visibility (RV) of potential drone positions in the air from points of interest on the ground. This visibility forms a basis for a novel navigation strategy. This strategy was shown to drive drones to optimal aerial monitoring positions, thus reducing occlusion and, consequently, the sampling time. Compared with previous unguided sampling, we achieve a 5%–20% higher visibility with 9–17 times less samples in our experiments.

Airborne optical sectioning, an effective aerial synthetic aperture imaging technique for revealing artifacts occluded by forests, requires precise measurements of drone poses. In this letter, we present a new approach for reducing pose estimation errors beyond the possibilities of conventional Perspective-n-Point solutions by considering the underlying optimization as a focusing problem. We present an efficient image integration technique, which also reduces the parameter search space to achieve realistic processing times, and improves the quality of resulting synthetic integral images.

In this letter, we describe and validate the first fully automatic parameter optimization for thermal synthetic aperture visualization. It replaces previous manual exploration of the parameter space, which is time-consuming and error-prone. We prove that the visibility of targets in thermal integral images is proportional to the variance of the targets' image. Since this is invariant to occlusion, it represents a suitable objective function for optimization. Our findings have the potential to enable fully autonomous search and recuse operations with camera drones.

This paper summarizes our progress toward a first fully transparent, flexible, and scalable thin-film image sensor and reviews lensless imaging approaches. Combining sensing and imaging in multiple layers has the potential to enable entire thin-film camera systems that make the capturing of images, light fields, and depth information possible.

Light-field cameras offer new imaging possibilities compared to conventional digital cameras. However, the additional angular domain of light fields prohibits direct application of frequently used image processing algorithms, such as warping, retargeting, or stitching. We present a general and efficient framework for nonuniform light-field warping, which forms the basis for extending many of these image processing techniques to light fields. It propagates arbitrary spatial deformations defined in one light-field perspective consistently to all other perspectives by means of 4D patch matching instead of relying on explicit depth reconstruction. This allows processing light-field recordings of complex scenes with non-Lambertian properties such as transparency and refraction. We show application examples of our framework in panorama light-field imaging, light-field retargeting, and artistic manipulation of light fields.

Light-field cameras offer new imaging possibilities compared to conventional digital cameras. However, the additional angular domain of light fields prohibits direct application of frequently used image processing algorithms, such as warping, retargeting, or stitching. We present a general and efficient framework for nonuniform light-field warping, which forms the basis for extending many of these image processing techniques to light fields. It propagates arbitrary spatial deformations defined in one light-field perspective consistently to all other perspectives by means of 4D patch matching instead of relying on explicit depth reconstruction. This allows processing light-field recordings of complex scenes with non-Lambertian properties such as transparency and refraction. We show application examples of our framework in panorama light-field imaging, light-field retargeting, and artistic manipulation of light fields.

We present a simple guided super-resolution technique for increasing directional resolution without reliance on depth estimation or image correspondences. Rather, it searches for best-matching multidimensional (4D or 3D) patches within the entire captured data set to compose new directional images that are consistent in both the spatial and the directional domains. We describe algorithms for guided upsampling, iterative guided upsampling, and sampling code estimation. Our experimental results reveal that the outcomes of existing light-field camera arrays and lightstage systems can be improved without additional hardware requirements or recording effort simply by realignment of cameras or light sources to change their sampling patterns.

Abstract We present a novel approach to recording and computing panorama light fields. In contrast to previous methods that estimate panorama light fields from focal stacks or naive multi‐perspective image stitching, our approach is the first that processes ray entries directly and does not require depth reconstruction or matching of image features. Arbitrarily complex scenes can therefore be captured while preserving correct occlusion boundaries, anisotropic reflections, refractions, and other light effects that go beyond diffuse reflections of Lambertian surfaces.

Abstract Capturing exposure sequences to compute high dynamic range (HDR) images causes motion blur in cases of camera movement. This also applies to light‐field cameras: frames rendered from multiple blurred HDR light‐field perspectives are also blurred. While the recording times of exposure sequences cannot be reduced for a single‐sensor camera, we demonstrate how this can be achieved for a camera array. Thus, we decrease capturing time and reduce motion blur for HDR light‐field video recording. Applying a spatio‐temporal exposure pattern while capturing frames with a camera array reduces the overall recording time and enables the estimation of camera movement within one light‐field video frame. By estimating depth maps and local point spread functions (PSFs) from multiple perspectives with the same exposure, regional motion deblurring can be supported. Missing exposures at various perspectives are then interpolated.

Abstract We present a caching framework with a novel probability‐based prefetching and eviction strategy applied to atomic cache units that enables interactive rendering of gigaray light fields. Further, we describe two new use cases that are supported by our framework: panoramic light fields, including a robust imaging technique and an appropriate parameterization scheme for real‐time rendering and caching; and light‐field‐cached volume rendering, which supports interactive exploration of large volumetric datasets using light‐field rendering. We consider applications such as light‐field photography and the visualization of large image stacks from modern scanning microscopes.

Abstract We present a first approach to light‐field retargeting using z‐stack seam carving, which allows light‐field compression and extension while retaining angular consistency. Our algorithm first converts an input light field into a set of perspective‐sheared focal stacks. It then applies 3D deconvolution to convert the focal stacks into z‐stacks, and seam‐carves the z‐stack of the center perspective. The computed seams of the center perspective are sheared and applied to the z‐stacks of all off‐center perspectives. Finally, the carved z‐stacks are converted back into the perspective images of the output light field. To our knowledge, this is the first approach to light‐field retargeting. Unlike existing stereo‐pair retargeting or 3D retargeting techniques, it does not require depth information.

In this paper, we show that optical inverse tone-mapping (OITM) in light microscopy can improve the visibility of specimens, both when observed directly through the oculars and when imaged with a camera. In contrast to previous microscopy techniques, we premodulate the illumination based on the local modulation properties of the specimen itself. We explain how the modulation of uniform white light by a specimen can be estimated in real time, even though the specimen is continuously but not uniformly illuminated. This information is processed and back-projected constantly, allowing the illumination to be adjusted on the fly if the specimen is moved or the focus or magnification of the microscope is changed. The contrast of the specimen's optical image can be enhanced, and high-intensity highlights can be suppressed. A formal pilot study with users indicates that this optimizes the visibility of spatial structures when observed through the oculars. We also demonstrate that the signal-to-noise (S/N) ratio in digital images of the specimen is higher if captured under an optimized rather than a uniform illumination. In contrast to advanced scanning techniques that maximize the S/N ratio using multiple measurements, our approach is fast because it requires only two images. This can improve image analysis in digital microscopy applications with real-time capturing requirements.

The two papers in this special section are extended versions of papers originally presented at the International Symposium on Mixed and Augmented Reality (ISMAR) 2007. These two papers won awards at the symposium.

Coding a projector's aperture plane with adaptive patterns together with inverse filtering allow the depth-of-field of projected imagery to be increased. We present two prototypes and corresponding algorithms for static and programmable apertures. We also explain how these patterns can be computed at interactive rates, by taking into account the image content and limitations of the human visual system. Applications such as projector defocus compensation, high-quality projector depixelation, and increased temporal contrast of projected video sequences can be supported. Coded apertures are a step towards next-generation auto-iris projector lenses.

The three papers in this special section are extended versions of papers presented at the International Symposium on Mixed and Augmented Reality (ISMAR).

We present a novel image classification technique for detecting multiple objects (called subobjects) in a single image. In addition to image classifiers, we apply spatial relationships among the subobjects to verify and to predict the locations of detected and undetected subobjects, respectively. By continuously refining the spatial relationships throughout the detection process, even locations of completely occluded exhibits can be determined. This approach is applied in the context of PhoneGuide, an adaptive museum guidance system for camera-equipped mobile phones.

In this paper, we present a novel technique for adapting local image classifiers that are applied for object recognition on mobile phones through ad-hoc network communication between the devices. By continuously accumulating and exchanging collected user feedback among mobile phones that are located within signal range, we show that our approach improves the overall classification rate and adapts to dynamic changes quickly. This technique is applied in the context of PhoneGuide -- an adaptive museum guidance system.

Abstract This article focuses on real‐time image correction techniques that enable projector‐camera systems to display images onto screens that are not optimized for projections, such as geometrically complex, coloured and textured surfaces. It reviews hardware‐accelerated methods like pixel‐precise geometric warping, radiometric compensation, multi‐focal projection and the correction of general light modulation effects. Online and offline calibration as well as invisible coding methods are explained. Novel attempts in super‐resolution, high‐dynamic range and high‐speed projection are discussed. These techniques open a variety of new applications for projection displays. Some of them will also be presented in this report.

We present a simple and cost-efficient way of extending contrast, perceived tonal resolution, and color space of reflective media, such as paper prints, hardcopy photographs, or electronic paper displays. A calibrated projector-camera system is applied for automatic registration, radiometric scanning and superimposition. A second modulation of the projected light on the surface of such media results in a high dynamic range visualization. This holds application potential for a variety of domains, such as radiology, astronomy, optical microscopy, conservation and restoration of historic art, modern art and entertainment installations.

We synchronize film cameras and LED lighting with off-the-shelf video projectors. Radiometric compensation allows displaying keying patterns and other spatial codes on arbitrary real world surfaces. A fast temporal multiplexing of coded projection and flash illumination enables professional keying, environment matting, displaying moderator information, scene reconstruction, and camera tracking for non-studio film sets without being limited to the constraints of a virtual studio. This makes digital video composition more flexible, since static studio equipment, such as blue screens, teleprompters, or tracking devices, is not required. Authentic film locations can be supported with our portable system without causing a lot of installation effort.

Recent radiometric compensation techniques make it possible to project images onto colored and textured surfaces. This is realized with projector-camera systems by scanning the projection surface on a per-pixel basis. Using the captured information, a compensation image is calculated that neutralizes geometric distortions and color blending caused by the underlying surface. As a result, the brightness and the contrast of the input image is reduced compared to a conventional projection onto a white canvas. If the input image is not manipulated in its intensities, the compensation image can contain values that are outside the dynamic range of the projector. These will lead to clipping errors and to visible artifacts on the surface. In this article, we present an innovative algorithm that dynamically adjusts the content of the input images before radiometric compensation is carried out. This reduces the perceived visual artifacts while simultaneously preserving a maximum of luminance and contrast. The algorithm is implemented entirely on the GPU and is the first of its kind to run in real-time.

In this paper we present an innovative adaptive imperceptible pattern projection technique that takes into account parameters of human visual perception. A coded image is temporally integrated into the projected image, which is invisible to the human eye but can be reconstructed by a synchronized camera. The embedded code is dynamically adjusted on the fly to guarantee its imperceptibility and to adapt it to the current camera pose. Linked with real-time flash keying, for instance, this enables in-shot optical tracking using a dynamic multi-resolution marker technique. A sample prototype has been realized that demonstrates the application of our method in the context of augmentations in television studios.

We present a system that employs a custom-built pan-tilt-zoom camera for laser pointer tracking in arbitrary real environments. Once placed in a room, it carries out a fully automatic self-registration, registrations of projectors, and sampling of surface parameters, such as geometry and reflectivity. After these steps, it can be used for tracking a laser spot on the surface as well as an LED marker in 3D space, using inter-playing fish-eye context and controllable detail cameras. The captured surface information can be used for masking out areas that are problematic for laser pointer tracking, and for guiding geometric and radiometric image correction techniques that enable a projector-based augmentation on arbitrary surfaces. We describe a distributed software framework that couples laser pointer tracking for interaction, projector-based AR as well as video see-through AR for visualizations, with the domain specific functionality of existing desktop tools for architectural planning, simulation and building surveying.

Radiometric compensation techniques allow seamless projections onto complex everyday surfaces. Implemented with projector-camera systems they support the presentation of visual content in situations where projection-optimized screens are not available or not desired - as in museums, historic sites, air-plane cabins, or stage performances. We propose a novel approach that employs the full light transport between projectors and a camera to account for many illumination aspects, such as interreflections, refractions, shadows, and defocus. Precomputing the inverse light transport in combination with an efficient implementation on the GPU makes the real-time compensation of captured local and global light modulations possible.

Immersive and semi-immersive projection displays, such as CAVEs, walls, workbenches, cylinders, and domes are being used to support virtual reality applications in many professional domains. The visualization of data with these displays, however, requires dedicated rooms for setting up non-mobile screens, and allows the interaction with purely synthetic information only. Current research activities investigate conceptual and technical possibilities for realizing such visualizations in real world environments. They strive for enabling immersive and semi-immersive virtual reality, as well as augmented reality experiences without the need for special display surfaces or permanent screen configurations. Enabling visualizations without projection-optimized screens but on complex everyday surfaces, however, leads to several difficulties: The projected images are geometrically warped, color-blended, and regionally defocused. Furthermore, the application of multiple projectors requires geometric registrations, as well as luminance and chrominance matching and intensity fading to display a single consistent image. Thus, projected images have to be corrected in real-time and on a pixel-precise basis. In addition, other surface properties, such as depth information and global illumination effects can be extracted and human perception factors can be taken into account for improving the final image quality as well as enabling new possibilities. Monoscopic and stereoscopic graphics make semi-immersive, immersive and augmented visualizations in everyday environments possible—without the need for special projection screens.

We present a real-time reverse radiosity method for compensating indirect scattering effects that occur with immersive and semi-immersive projection displays. It computes a numerical solution directly on the GPU and is implemented with pixel shading and multi-pass rendering which together realizes a Jacobi solver for sparse matrix linear equation systems. Our method is validated and evaluated based on a stereoscopic two-sided wall display. The images appear more brilliant and uniform when compensating the scattering contribution.

In this paper, we describe a novel multifocal projection concept that applies conventional video projectors and camera feedback. Multiple projectors with differently adjusted focal planes, but overlapping image areas are used. They can be either differently positioned in the environment or can be integrated into a single projection unit. The defocus created on an arbitrary surface is estimated automatically for each projector pixel. If this is known, a final image with minimal defocus can be composed in real-time from individual pixel contributions of all projectors. Our technique is independent of the surfaces' geometry, color and texture, the environment light, as well as of the projectors' position, orientation, luminance, and chrominance.

We show how view-dependent image-based and geometric warping, radiometric compensation, and multi-focal projection enable a view-dependent stereoscopic visualization on ordinary (geometrically complex, colored and textured) surfaces within everyday environments. Special display configurations for immersive or semi-immersive AR/VR applications that require permanent and artificial projection canvases might become unnecessary. We demonstrate several ad-hoc visualization examples in a real architectural and museum application context.

We present PhoneGuide -- an enhanced museum guidance system that uses camera-equipped mobile phones and on-device object recognition.Our main technical achievement is a simple and light-weight object recognition approach that is realized with single-layer perceptron neuronal networks. In contrast to related systems which perform computationally intensive image processing tasks on remote servers, our intention is to carry out all computations directly on the phone. This ensures little or even no network traffic and consequently decreases cost for online times. Our laboratory experiments and field surveys have shown that photographed museum exhibits can be recognized with a probability of over 90%.We have evaluated different feature sets to optimize the recognition rate and performance. Our experiments revealed that normalized color features are most effective for our method. Choosing such a feature set allows recognizing an object below one second on up-to-date phones. The amount of data that is required for differentiating 50 objects from multiple perspectives is less than 6KBytes.

We present a first running video see-through augmented reality system on a consumer cell-phone. It supports the detection and differentiation of different markers, and correct integration of rendered 3D graphics into the live video stream via a weak perspective projection camera model and an OpenGL rendering pipeline.

We present techniques which create a consistent illumination between real and virtual objects inside an application specific optical see-through display: the virtual showcase. We use projectors and cameras to capture reflectance information from diffuse real objects and to illuminate them under new synthetic lighting conditions. Matching direct and indirect lighting effects, such as shading, shadows, reflections and color bleeding can be approximated at interactive rates in such a controlled mixed environment.

No abstract available.

This paper presents projector-based illumination techniques for creating correct occlusion effects for optical see-through setups. We project view-dependent occlusion shadows onto the real surfaces that are located behind virtual objects. This results in a perfect occlusion of real objects by virtual ones. We have implemented and tested our approach in the context of the Virtual Showcase display. We describe a hardware extension for projecting light into the showcase and present our rendering techniques for displaying occlusion shadows for single and multi-user environments as well as for single and multi-light-projector configurations. We also report on the limitations of our system for multi-user situations and describe our experiences with a first experimental prototype.

The paper introduces the idea of using real mirrors in combination with rear-projection systems for the purpose of interacting with and navigating through the displayed information. Subsequently, a derived application is described. For this, we use a hand-held planar mirror and address two fundamental problems of applying head tracking with rear-projection planes: the limited viewing volume of these environments and their incapability of simultaneously supporting multiple observers. Furthermore, we describe the possibility of combining a reflective pad with a transparent one, thus introducing a complementary tool for interaction and navigation.

In this paper, we introduce the concept of Extended VR (extending viewing space and interaction space of back-projection VR systems), by describing the use of a hand-held semi-transparent mirror to support augmented reality tasks with back-projection systems. This setup overcomes the problem of occlusion of virtual objects by real ones linked with such display systems. The presented approach allows an intuitive and effective application of immersive or semi-immersive virtual reality tasks and interaction techniques to an augmented surrounding space. Thereby, we use the tracked mirror as an interactive image-plane that merges the reflected graphics, which are displayed on the projection plane, with the transmitted image of the real environment. In our implementation, we also address traditional augmented reality problems, such as real-object registration and virtual-object occlusion. The presentation is complemented by a hypothesis of conceivable further setups that apply transflective surfaces to support an Extended VR environment.

This paper investigates the applicability of context-free grammars to recognize and interpret representative 3D freehand sketches in terms of creating objects, performing transformations, and controlling the design space with the information contained by the sketches. Dynamic gesture recognition is the foundation of a two-layered concept. The subsequent layer performs multiple gesture recognition by parsing a sequence of single gestures depending on a predefined grammar, and simultaneously extracts the required information on the fly.

The Virtual Table presents stereoscopic graphics to a user in a workbench‐like setting. For this device, a user interface and new interaction techniques have been developed based on transparent props ‐a tracked hand‐held pen and a pad. These props, particularly the pad, are augmented with 3D graphics from the Virtual Table’s display that can serve as a palette for tools and controls as well as a window‐like see‐through interface, a plane‐shaped and through‐the‐plane tool, supporting a variety of new interaction techniques. This paper reports on an extension of this user‐interface design space which uses gestural input to create and control solid geometries for CAD and conceptual design. The application of gestural interfaces is a common method for interacting with virtual environments on a habitual and natural basis. The motion‐based gesture recognition presented here uses Fuzzy Logic to support a predictable, flexible, and efficient learning process. This new interaction paradigm greatly increases the Virtual Table’s suitability for design tasks. Traditional CAD dialogue can be combined with intuitive rapid sketching of geometry on the pad. Additionally, the resulting events and objects can be associated with scene details below the translucent tablet.