Omnidirectional capture of dynamic scenes facilitates the creation of immersive virtual reality assets and holistic scene understanding. Outward-facing multi-fisheye camera rigs offer an efficient solution for full-scene coverage, using fewer lenses than conventional pinhole arrays while enabling all-directional observation of complex, time-varying environments. By continuously recording scene evolution from every angle, these systems naturally enable a richer characterization of dynamic interactions. Despite these advantages, dynamic scene modeling in this setting remains underexplored. Existing methods, typically designed for fixed pinhole configurations or monocular setups, rely heavily on photometric cues and often neglect the strong geometric and semantic priors inherent in multi-fisheye omnidirectional data. To address this gap, we present OmniPrior, a Gaussian Splatting-based framework for outward-facing, multi-fisheye omnidirectional capture. Our approach incorporates metric-geometry-aware initialization with multi-prior guidance, introducing a dynamicness-aware Gaussian representation that encodes both object motion and subtle temporal variations. The resulting representations are physically consistent and temporally stable. Extensive experiments validate the effectiveness of our method in novel view synthesis across new viewpoints and timestamps. We demonstrate its utility in two representative applications derived from our learned representations: 6DoF rendering with flexible FoV and motion-freeze rendering.

Consumer-level omnidirectional video offers an economically viable means to create virtual reality (VR) assets, enabling users to explore and interact within a fully immersive visual environment. However, editing such videos, particularly those with 360${}^{\circ }$∘ views and dynamic objects, poses significant challenges. Existing approaches to representing and manipulating omnidirectional content-whether designed for typical 2D perspective imagery or panoramas-often fail to adequately capture the complex spatiotemporal relationships crucial for producing high-quality, editable outputs in dynamic, panoramic settings. To overcome these challenges, we introduce OmniPlane, a novel method that leverages spherical spatiotemporal feature grids to empower the representation and editability of real-world dynamic omnidirectional environments casually captured by commodity omnidirectional cameras. OmniPlane computes spatiotemporal features by fusing vectors or matrices from each learnable spatial and spatiotemporal feature plane within a spherical coordinate system, complemented by a specifically designed weighted sampling strategy respecting the inherent spherical distribution of omnidirectional content. These learned feature planes can be flexibly decomposed into palette-based color bases. This innovative method not only enhances the representation capability of omnidirectional content and dynamics but also enables the recoloring of omnidirectional videos. Extensive experiments and a dedicated user study validate the superior performance of our proposed method in facilitating recolorable representations of dynamic omnidirectional environments.

High-quality data is necessary for modern machine learning. However, the acquisition of such data is difficult due to noisy and ambiguous annotations of humans. The aggregation of such annotations to determine the label of an image leads to a lower data quality. We propose a data-centric image classification benchmark with nine real-world datasets and multiple annotations per image to allow researchers to investigate and quantify the impact of such data quality issues. With the benchmark we can study the impact of annotation costs and (semi-)supervised methods on the data quality for image classification by applying a novel methodology to a range of different algorithms and diverse datasets. Our benchmark uses a two-phase approach via a data label improvement method in the first phase and a fixed evaluation model in the second phase. Thereby, we give a measure for the relation between the input labeling effort and the performance of (semi-)supervised algorithms to enable a deeper insight into how labels should be created for effective model training. Across thousands of experiments, we show that one annotation is not enough and that the inclusion of multiple annotations allows for a better approximation of the real underlying class distribution. We identify that hard labels can not capture the ambiguity of the data and this might lead to the common issue of overconfident models. Based on the presented datasets, benchmarked methods, and analysis, we create multiple research opportunities for the future directed at the improvement of label noise estimation approaches, data annotation schemes, realistic (semi-)supervised learning, or more reliable image collection.

The design of a plenoptic camera requires the combination of two dissimilar optical systems, namely a main lens and an array of microlenses. And while the construction process of a conventional camera is mainly concerned with focusing the image onto a single plane, in the case of plenoptic cameras there can be additional requirements such as a predefined depth of field or a desired range of disparities in neighboring microlens images. Due to this complexity, the manual creation of multiple plenoptic camera setups is often a time-consuming task. In this work we assume a simulation framework as well as the main lens data given and present a method to calculate the remaining aperture, sensor and microlens array parameters under different sets of constraints. Our ray tracing-based approach is shown to result in models outperforming their pendants generated with the commonly used paraxial approximations in terms of image quality, while still meeting the desired constraints. Both the implementation and evaluation setup including 30 plenoptic camera designs are made publicly available.

Stixels have been successfully applied to a wide range of vision tasks in autonomous driving, recently including instance segmentation. However, due to their sparse occurrence in the image, until now Stixels seldomly served as input for Deep Learning algorithms, restricting their utility for such approaches. In this work we present StixelPointNet, a novel method to perform fast instance segmentation directly on Stixels. By regarding the Stixel representation as unstructured data similar to point clouds, architectures like PointNet are able to learn features from Stixels. We use a bounding box detector to propose candidate instances, for which the relevant Stixels are extracted from the input image. On these Stixels, a PointNet models learns binary segmentations, which we then unify throughout the whole image in a final selection step. StixelPointNet achieves state-of-the-art performance on Stixel-level, is considerably faster than pixel-based segmentation methods, and shows that with our approach the Stixel domain can be introduced to many new 3D Deep Learning tasks.

Camera calibration methods usually consist of capturing images of known calibration patterns and using the detected correspondences to optimize the parameters of the assumed camera model. A meaningful evaluation of these methods relies on the availability of realistic synthetic data. In previous works concerned with conventional cameras the synthetic data was mainly created by rendering perfect images with a pinhole camera and subsequently adding distortions and aberrations to the renderings and correspondences according to the assumed camera model. This method can bias the evaluation since not every camera perfectly complies with an assumed model. Furthermore, in the field of plenoptic camera calibration there is no synthetic ground truth data available at all. We address these problems by proposing a method based on backward ray tracing to create realistic ground truth data that can be used for an unbiased evaluation of calibration methods for both types of cameras.

Shearlet Transform (ST) is one of the most effective methods for Densely-Sampled Light Field (DSLF) reconstruction from a Sparsely-Sampled Light Field (SSLF). However, ST requires a precise disparity estimation of the SSLF. To this end, in this paper a state-of-the-art optical flow method, i.e. PWC-Net, is employed to estimate bidirectional disparity maps between neighboring views in the SSLF. Moreover, to take full advantage of optical flow and ST for DSLF reconstruction, a novel learning-based method, referred to as Flow-Assisted Shearlet Transform (FAST), is proposed in this paper. Specifically, FAST consists of two deep convolutional neural networks, i.e. disparity refinement network and view synthesis network, which fully leverage the disparity information to synthesize novel views via warping and blending and to improve the novel view synthesis performance of ST. Experimental results demonstrate the superiority of the proposed FAST method over the other state-of-the-art DSLF reconstruction methods on nine challenging real-world SSLF sub-datasets with large disparity ranges (up to 26 pixels).

Shearlet Transform (ST) is one of the most effective algorithms for the Densely-Sampled Light Field (DSLF) reconstruction from a Sparsely-Sampled Light Field (SSLF) with a large disparity range. However, ST requires a precise estimation of the disparity range of the SSLF in order to design a shearlet system with decent scales and to pre-shear the sparsely-sampled Epipolar-Plane Images (EPIs) of the SSLF. To overcome this limitation, a novel coarse-to-fine DSLF reconstruction method, referred to as Mask-Accelerated Shearlet Transform (MAST), is proposed in this paper. Specifically, a state-of-the-art learning-based optical flow method, FlowNet2, is employed to estimate the disparities of a SSLF. The estimated disparities are then utilized to roughly estimate the densely-sampled EPIs for the sparsely-sampled EPIs of the SSLF. Finally, an elaborately-designed soft mask for a coarsely-inpainted EPI is exploited to perform an iterative refinement on this EPI. Experimental results on nine challenging horizontal-parallax real-world SSLF datasets with large disparity ranges (up to 35 pixels) demonstrate the effectiveness and efficiency of the proposed method over the other state-of-the-art approaches.

MLA-based focused plenoptic cameras, also called type 2.0 cameras, have advantages over type 1.0 plenoptic cameras, because of their better inherent spatial image resolution and their compromise between depth of focus and angular resolution. However, they are more difficult to process since they require a depth estimation first to compute the all-in-focus image from the raw MLA image data. Current toolboxes for plenoptic cameras only support the type 1.0 cameras (like Lytro) and cannot handle type 2.0 cameras (like Raytrix). In addition, there is a lack of ground truth data and high quality benchmarking data for focussed plenoptic cameras. This contribution will discuss the requirements for processing type 2.0 images and will supply the reader with an open-source toolbox for comparing depth estimation methods. Different depth-estimation methods for MLA-based imaging will be available and an easy extension for other processing algorithms like compression will be included. In addition, we will supply benchmarking data of focused plenoptic cameras by synthetic ground truth datasets and high-quality real images captured under controlled conditions by Raytrix cameras.

A new light-field movie capture device using a Kinect V2 sensor and two 4K GoPro cameras is presented in this paper. Due to the uncontrollable tilt of the Kinect V2 camera, it is hard to obtain a constant rigid transformation between the stereo- and ToF-camera systems. To this end, a novel self-calibration method is proposed, which takes advantage of the geometric constraints from the scene and the cameras. Specifically, a camera orientation approximation approach is utilized to estimate the rigid transformation of the stereo-ToF system based on reliable point pairs filtered by the geometric constraints. Besides, a depth correction step is exploited to improve the depth accuracy of the Kinect V2 sensor. Moreover, a depth fusion strategy for the stereo- and ToF-depth data is proposed to provide more accurate depth images in 4K resolution. Experimental results demonstrate the effectiveness of the proposed depth correction step, stereo-ToF calibration method and depth fusion strategy.

In a lot of multi-Kinect V2-based systems, the registration of these Kinect V2 sensors is an important step which directly affects the system precision. The coarse-to-fine method using calibration objects is an effective way to solve the Kinect V2 registration problem. However, for the registration of Kinect V2 cameras with large displacements, this kind of method may fail. To this end, a novel Kinect V2 registration method, which is also based on the coarse-to-fine framework, is proposed by using camera and scene constraints. Specifically, in the coarse estimation stage, scene constraints are explored using off-the-shelf feature point detectors and camera constraints are explored using homography and fundamental matrices. In the estimation refinement stage, an Iterative Closest Point (ICP)-based point cloud registration method is utilized. Experimental results show that the proposed Kinect V2 registration method using camera and scene constraints performs much better in precision than using calibration objects in the large-displacement environment.

In this paper we deal with the camera pose estimation problem from a set of 2D/3D line correspondences, which is also known as PnL (Perspective-n-Line) problem. We carry out our study by comparing PnL with the well-studied PnP (Perspective-n-Point) problem, and our contributions are three-fold: (1) We provide a complete 3D configuration analysis for P3L, which includes the well-known P3P problem as well as several existing analyses as special cases. (2) By exploring the similarity between PnL and PnP, we propose a new subset-based PnL approach as well as a series of linear-formulation-based PnL approaches inspired by their PnP counterparts. (3) The proposed linear-formulation-based methods can be easily extended to deal with the line and point features simultaneously.

We propose a fast calibration method for projector-camera pairs which does not require any special calibration objects or initial estimates of the calibration parameters. Our method is based on a structured light approach to establish correspondences between the camera and the projector view. Using the vanishing points in the camera and the projector view the internal as well as the external calibration parameters are estimated. In addition, we propose an interactive projection mapping scheme which allows the user to directly place two-dimensional media elements in the tangent planes of the target surface without any manual perspective corrections.

This contribution deals with image restoration in optical systems with coherent illumination, which is an important topic in astronomy, coherent microscopy and radar imaging. Such optical systems suffer from wavefront distortions, which are caused by imperfect imaging components and conditions. Known image restoration algorithms work well for incoherent imaging, they fail in case of coherent images. In this paper a novel wavefront correction algorithm is presented, which allows image restoration under coherent conditions. In most coherent imaging systems, especially in astronomy, the wavefront deformation is known. Using this information, the proposed algorithm allows a high quality restoration even in case of severe wavefront distortions. We present two versions of this algorithm, which are an evolution of the Gerchberg-Saxton and the Hybrid-Input-Output algorithm. The algorithm is verified on simulated and real microscopic images.

This paper describes an adaptable system which is able to perform manipulation operations (such as Peg-in-Hole or Laying-Down actions) with flexible objects. As such objects easily change their shape significantly during the execution of an action, traditional strategies, e.g, for solve path-planning problems, are often not applicable. It is therefore required to integrate visual tracking and shape reconstruction with a physical modeling of the materials and their deformations as well as action learning techniques. All these different submodules have been integrated into a demonstration platform, operating in real-time. Simulations have been used to bootstrap the learning of optimal actions, which are subsequently improved through real-world executions. To achieve reproducible results, we demonstrate this for casted silicone test objects of regular shape. Note to Practitioners - The aim of this work was to facilitate the setup of robot-based automation of delicate handling of flexible objects consisting of a uniform material. As examples, we have considered how to optimally maneuver flexible objects through a hole without colliding and how to place flexible objects on a flat surface with minimal introduction of internal stresses in the object. Given the material properties of the object, we have demonstrated in these two applications how the system can be programmed with minimal requirements of human intervention. Rather than being an integrated system with the drawbacks in terms of lacking flexibility, our system should be viewed as a library of new technologies that have been proven to work in close to industrial conditions. As a rather basic, but necessary part, we provide a technology for determining the shape of the object when passing on, e.g., a conveyor belt prior to being handled. The main technologies applicable for the manipulated objects are: A method for real-time tracking of the flexible objects during manipulation, a method for model-based offline prediction of the static deformation of grasped, flexible objects and, finally, a method for optimizing specific tasks based on both simulated and real-world executions.

In underwater environments, cameras need to be confined in an underwater housing, viewing the scene through a piece of glass. In case of flat port underwater housings, light rays entering the camera housing are refracted twice, due to different medium densities of water, glass, and air. This causes the usually linear rays of light to bend and the commonly used pinhole camera model to be invalid. When using the pinhole camera model without explicitly modeling refraction in Structure-from-Motion (SfM) methods, a systematic model error occurs. Therefore, in this paper, we propose a system for computing camera path and 3D points with explicit incorporation of refraction using new methods for pose estimation. Additionally, a new error function is introduced for non-linear optimization, especially bundle adjustment. The proposed method allows to increase reconstruction accuracy and is evaluated in a set of experiments, where the proposed method's performance is compared to SfM with the perspective camera model.

Spatial augmented reality is especially interesting for the design process of a car, because a lot of virtual content and corresponding real objects are used. One important issue in such a process is that the designer can trust the visualized colors on the real object, because design decisions are made on basis of the projection. In this paper, we present an interactive visualization technique which is able to exactly compute the RGB values for the projected image, so that the resulting colors on the real object are equally perceived as the real desired colors. Our approach computes the influences of the ambient light, the material, the pose and the color model of the projector to the resulting colors of the projected RGB values by using a physically based computation. This information allows us to compute the adjustment for the RGB values for varying projector positions at interactive rates. Since the amount of projectable colors does not only depend on the material and the ambient light, but also on the pose of the projector, our method can be used to interactively adjust the range of projectable colors by moving the projector to arbitrary positions around the real object. We further extend the mentioned method so that it is applicable to multiple projectors. All methods are evaluated in a number of experiments.

Head-mounted displays (HMDs) allow users to observe virtual environments (VEs) from an egocentric perspective. However, several experiments have provided evidence that egocentric distances are perceived as compressed in VEs relative to the real world. Recent experiments suggest that the virtual view frustum set for rendering the VE has an essential impact on the user's estimation of distances. In this article we analyze if distance estimation can be improved by calibrating the view frustum for a given HMD and user. Unfortunately, in an immersive virtual reality (VR) environment, a full per user calibration is not trivial and manual per user adjustment often leads to mini- or magnification of the scene. Therefore, we propose a novel per user calibration approach with optical see-through displays commonly used in augmented reality (AR). This calibration takes advantage of a geometric scheme based on 2D point - 3D line correspondences, which can be used intuitively by inexperienced users and requires less than a minute to complete. The required user interaction is based on taking aim at a distant target marker with a close marker, which ensures non-planar measurements covering a large area of the interaction space while also reducing the number of required measurements to five. We found the tendency that a calibrated view frustum reduced the average distance underestimation of users in an immersive VR environment, but even the correctly calibrated view frustum could not entirely compensate for the distance underestimation effects.

One challenge in computer vision is the joint reconstruction of deforming objects from colour and depth videos.So far, a lot of research has focused on deformation reconstruction based on colour images only, but as range cameras like the recently released Kinect become more and more common, the incorporation of depth information becomes feasible.In this article a new method is introduced to track object deformation in depth and colour image data.A NURBS based deformation function allows to decouple the geometrical object complexity from the complexity of the deformation itself, providing a low dimensional space to describe arbitrary 'realistic' deformations.While modelling the tracking objective as an analysis by synthesis problem, which is robust but usually computationally expensive, a set of optimisations is introduced, allowing a very fast calculation of the resulting error function.With a fast semi-global search a system is established that is capable of tracking complex deformations of large objects (6000 triangles and more) with more than 6Hz on a common desktop machine.The algorithm is evaluated using simulated and real data, showing the robustness and performance of the approach.

When using perspective cameras underwater, the underwater housing with its glass interface between water and air causes the light rays to change their direction due to refraction. In applications where geometrical properties of images are exploited without explicitly modeling refraction, i.e. when using the perspective pinhole camera model, this leads to a systematical error. This error is depending on the housing configuration like distance between camera and glass interface and angle between glass interface normal and optical axis. In this paper, we analyze the calibration of those parameters using a camera model explicitly considering refraction. The goal is to determine those parameters without the need of handling a calibration target underwater, which is cumbersome, if not impossible.

Spatial augmented reality is especially interesting for the design process of a car, because a lot of virtual content and corresponding real objects are used. One important issue in such a process is that the designer can trust the visualized colors on the real object, because design decisions are made on basis of the projection. In this article, we present an interactive visualization technique which is able to exactly compute the RGB values for the projected image, so that the resulting colors on the real object are equally perceived as the real desired colors. Our approach computes the influences of the ambient light, the material, the pose and the color model of the projector to the resulting colors of the projected RGB values by using a physically-based computation. This information allows us to compute the adjustment for the RGB values for varying projector positions at interactive rates. Since the amount of projectable colors does not only depend on the material and the ambient light, but also on the pose of the projector, our method can be used to interactively adjust the range of projectable colors by moving the projector to arbitrary positions around the real object. The proposed method is evaluated in a number of experiments.

Abstract If spatial augmented reality is used in the design process of a car, then one of the most important issues is that the virtual content is projected with a very high visual quality onto the real object, because based on this projection design decisions are made. Especially, the visualised colours on the real object should not be distinguishable from corresponding real reference colours. In this paper, we introduce a new approach for the augmentation of real objects which is able to match the requirements of a design process. We present a new rendering method with ray tracing which increases the visual quality of the projection images in comparison to existing methods. The desired values of these images have further to be adjusted according to the material, the ambient light and the local orientation of the projector. For this purpose, we develop a physically based computation which exactly determines the corresponding projection intensities for these values by using three‐dimensional lookup tables at every projector pixel. Since not all of the desired values can be represented with an intensity of the projector, an adjustment has to be computed for these values. Therefore, we conduct a user study with design experts who work in the automotive industry and use the results to propose a new adjustment method for such values. Finally, we compare our methods to existing procedures and conclude which ones are suitable for the design process of a car.

This paper introduces a reformulation of the extended Kalman Filter using the Gauss-Helmert model for least squares estimation. By proving the equivalence of both estimators it is shown how the methods of statistical analysis in least squares estimation can be applied to the prediction and update process in Kalman Filtering. Especially the efficient computation of the reliability (or redundancy) matrix allows the implementation of self supervising systems. As an application an unparameterized method for estimating the variances of the filters process noise is presented.

Abstract A growing number of applications depend on accurate and fast 3D scene analysis. Examples are model and lightfield acquisition, collision prevention, mixed reality and gesture recognition. The estimation of a range map by image analysis or laser scan techniques is still a time‐consuming and expensive part of such systems. A lower‐priced, fast and robust alternative for distance measurements are time‐of‐flight (ToF) cameras. Recently, significant advances have been made in producing low‐cost and compact ToF devices, which have the potential to revolutionize many fields of research, including computer graphics, computer vision and human machine interaction (HMI). These technologies are starting to have an impact on research and commercial applications. The upcoming generation of ToF sensors, however, will be even more powerful and will have the potential to become ‘ubiquitous real‐time geometry devices’ for gaming, web‐conferencing, and numerous other applications. This paper gives an account of recent developments in ToF technology and discusses the current state of the integration of this technology into various graphics‐related applications.

Parametric, gradient-based image alignment is used nowadays in many applications such as object tracking, image registration or camera calibration. In such processes intensity differences between a template and a warped image are minimised based on Newton-like optimisation algorithms. It has been known for a long time that pre-filtering the images under inspection and the use of coarse-to-fine strategies can somehow increase the convergence radius, but a general derivation is missing. We present a generic framework which relates parameter uncertainty with positions in the image’s scale space instead of heuristic isotropic smoothing to improve the convergence radius. Specifying parameter uncertainty is often more intuitive than selecting a good pyramid level and improves convergence particularly in settings where a single parameter’s influence (e.g. a rotation angle) varies largely across a patch. We show that the classical application of image pyramids in displacement estimation embeds into the novel formulation and demonstrate our approach on refinement of robust feature correspondences and homography estimation.

When rotating a pinhole camera, images are related by the infinite homography KRK-1, which is algebraically a conjugate rotation. Although being a very common image transformation, e.g. important for self-calibration or panoramic image mosaicing, it is not completely understood yet. We show that a conjugate rotation has 7 degrees of freedom (as opposed to 8 for a general homography) and give a minimal parameterization. To estimate the conjugate rotation, authors traditionally made use of point correspondences, which can be seen as local zero order Taylor approximations to the image transformation. Recently however, affine feature correspondences have become increasingly popular. We observe that each such affine correspondence now provides a local first order Taylor approximation, which has not been exploited in the context of geometry estimation before. Using those two novel concepts above, we finally show that it is possible to estimate a conjugate rotation from a single affine feature correspondence under the assumption of square pixels and zero skew. As a byproduct, the proposed algorithm directly yields rotation, focal length and principal point.

Recently real-time active 3D range cameras based on time-of-flight technology (PMD) have become available. Those cameras can be considered as a competing technique for stereo-vision based surface reconstruction. Since those systems directly yield accurate 3d measurements, they can be used for benchmarking vision based approaches, especially in highly dynamic environments. Therefore, a comparative study of the two approaches is relevant. In this work the achievable accuracy of the two techniques, PMD and stereo, is compared on the basis of patch-let estimation. As patchlet we define an oriented small planar 3d patch with associated surface normal. Least-squares estimation schemes for estimating patchlets from PMD range images as well as from a pair of stereo images are derived. It is shown, how the achivable accuracy can be estimated for both systems. Experiments under optimal conditions for both systems are performed and the achievable accuracies are compared. It has been found that the PMD system outperformed the stereo system in terms of achievable accuracy for distance measurements, while the estimation of normal direction is comparable for both systems.

We extend the successful 2D robust feature concept into the third dimension in that we produce a descriptor for a reconstructed 3D surface region. The descriptor is perspectively invariant if the region can locally be approximated well by a plane. We exploit depth and texture information, which is nowadays available in real-time from video of moving cameras, from stereo systems or PMD cameras (photonic mixer devices). By computing a normal view onto the surface we still keep the descriptiveness of similarity invariant features like SIFT while achieving in- variance against perspective distortions, while descriptiveness typically suffers when using affine invariant features. Our approach can be exploited for structure-from-motion, for stereo or PMD cameras, alignment of large scale reconstructions or improved video registration.

This work describes a visual markerless real-time tracking system for augmented reality applications. The system uses a firewire camera with a fisheye lens mounted at 10 fps. Visual tracking of 3D scene points is performed simultaneously with 3D camera pose estimation without any prior scene knowledge. All visual-geometric data is acquired using a structure-from-motion approach. The lens selection was driven by research results that show the superiority of a fisheye lens to a standard perspective lens for this approach. 2D features in the hemispherical image are tracked using a 2D point tracker. Based on the feature tracks, 3D camera ego-motion and 3D features are estimated.

We present a system for robust real time person tracking that integrates face detection, face color tracking and foot tracking in a uniform way by using a particle filter. The system is embedded in a complete immersive environment (3-sided CAVE with 1-sided stereo back projection). The person controls the visual environment by walking around inside.

We address the problem of using external rotation information with uncalibrated video sequences. The main problem addressed is, what is the benefit of the orientation information for camera calibration? It is shown that in case of a rotating camera the camera calibration problem is linear even in the case that all intrinsic parameters vary. For arbitrarily moving cameras the calibration problem is also linear but underdetermined for the general case of varying all intrinsic parameters. However, if certain constraints are applied to the intrinsic parameters the camera calibration can be computed linearly. It is analyzed which constraints are needed for camera calibration of freely moving cameras. Furthermore we address the problem of aligning the camera data with the rotation sensor data in time. We give an approach to align these data in case of a rotating camera.

No abstract available.

Abstract In this paper we present a novel approach for interactive rendering of virtual views from real image sequences.Combining the concepts of light fields, depth‐compensated image warping and view dependent texture mapping,this plenoptic modeling approach can handle large and complex scenes. A portable, handheld multi‐camera systemhas been developed that allows to record multiple image streams by simply walking around the scene. Theseimage streams are automatically calibrated and depth maps for all views are generated as input to the renderingstage. For rendering a view dependent warping surface is constructed on the fly and depth‐compensated imageinterpolation is applied with view‐dependent texture mapping. Rendering quality is scalable to allow fast previewand to achieve high‐end quality with the same approach. The system can handle large and geometrically complexscenes with hundreds of real images at interactive rates. Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Computer Graphics]: Viewing algorithms, I.4.1[Image Processing and Computer Vision]: Digitization and Image Capture, I.4.8 [Image Processing and ComputerVision]: Scene Analysis

This contribution addresses the problem of obtaining 3D models from image sequences. A 3D surface description of the scene is extracted completely from a set of uncalibrated camera images of the scene. No prior knowledge about the scene or about the camera is needed to build the 3D models. The only assumptions are the rigidity of the scene objects and opaque object surfaces. The modelling system described here uses a three-step approach. First, the camera pose and intrinsic parameters are calibrated by tracking salient feature points throughout the sequence. Next, consecutive images of the sequence are treated as stereoscopic image pairs, and dense correspondence maps are computed by area matching. Finally, dense and accurate depth maps are computed by linking together all correspondences over the viewpoints. The depth maps are converted to triangular surfaces meshes that are texture mapped for photo-realistic appearance. The feasibility of the approach has been tested on both real and synthetic data and is illustrated here on several outdoor image sequences. Copyright © 2000 John Wiley & Sons, Ltd.

We focus on the calibration of very long image sequences from a hand-held camera that samples the viewing sphere of a scene. View sphere sampling is important for plenoptic (image-based) modeling that captures the appearance of a scene by storing images from all possible directions. The plenoptic approach is appealing since it allows, in principle, fast scene rendering of scenes with complex geometry and surface reflections, without the need for an explicit geometrical scene model. However the acquired images have to be calibrated, and current approaches mostly use pre-calibrated acquisition systems. This limits the generality of the approach. We propose using an uncalibrated hand-held camera only. The image sequence is acquired by simply waving the camera around the scene objects, creating a zigzag scan path over the viewing sphere. We extend the sequential camera tracking of an existing structure-from-motion approach to the calibration of a mesh of viewpoints. Novel views are generated by piecewise mapping and interpolating the new image from the nearest viewpoints according to the viewpoint mesh. Local depth map estimates enhance the rendering process. Extensive experiments with ground truth data and hand-held sequences confirm the performance of our approach.

In this paper a new rectification method is proposed. The method is both simple and efficient and can deal with all possible camera motions. A minimal image size without any pixel loss is guaranteed. The only required information is the oriented fundamental matrix. The whole rectification process is carried out directly in the images. The idea consists of using a polar parametrization of the image around the epipole. The transfer between the images is obtained through the fundamental matrix. The proposed method has important advantages compared to the traditional rectification schemes. In some cases these approaches yield very large images or can not rectify at all. Even the recently proposed cylindrical rectification method can encounter problems in some cases. These problems are mainly due to the fact that the matching ambiguity is not reduced to half epipolar lines. Although this last method is more complex than the one proposed in this paper the resulting images are in general larger. The performance of the new approach is illustrated with some results on real image pairs.

3D shape models are often put together from several partial reconstructions. There are good algorithms available now to perform the necessary, precise registration automatically, but only after the partial reconstructions have been brought into approximate positions. This paper proposes a technique to do precisely that. Automatic 'crude registration' is illustrated for Euclidean and affine transformations between parts. The technique is based on the extraction and invariant characterisation of bitangent curve pairs. These can be found and matched efficiently. Registration of these curves yields the crude initialisation that more precise registration algorithms like ICP or mutual information maximisation can start from.

This paper addresses the problem of obtaining photo-realistic 3D models of a scene from images alone with a structure-from-motion approach. The 3D scene is observed from multiple viewpoints by freely moving a camera around the object. No restrictions on camera movement and interval camera parameters like zoom are imposed, as the camera pose and intrinsic parameters are calibrated from the sequence. The only restrictions on the scene content are the rigidity of the scene objects and opaque, piecewise smooth object surfaces. The approach operates independently of object scale and requires only a single low-cost consumer photo or video camera. The modeling system described here uses a three-step approach. First, the camera pose and intrinsic parameters are calibrated on-line by tracking salient feature points between the different views. Next, consecutive images of the sequence are treated as stereoscopic image pairs and dense correspondence maps are computed by area matching. Finally, dense and accurate depth maps are computed by linking together all correspondences over the viewpoints. The depth maps are converted to triangular surfaces meshes that are texture mapped for photo-realistic appearance. The resulting surface models are stored in VRML-format for easy exchange and visualization. The feasibility of the approach has been tested extensively and will be illustrated on several real scenes. In particular we will demonstrate the generation of realistic 3D models for a virtual exhibition of the archaeological excavation site in Sagalassos, Turkey.

This paper develops new image matching techniques for visual servoing based on affine invariants which allow one to deal with large viewpoint changes and that do not rely on specific markers. The only assumption is that there are some locally planar and unoccluded scene regions that have enough structure to be detected in the image. Those regions are classified by a set of illumination and viewpoint invariant features. The features represent the image in a very compact way and allow fast comparison and feature matching between quite different viewpoints. The matching procedure is embedded in a visual servoing system for a mobile robot. Experiments show its potential for navigation with large camera rotations and view point changes in a cluttered environment without the need for artificial landmarks.

The media and communications providers share an increasing interest in 3D models of people, objects, and scenes. The paper focuses on features that 3D acquisition systems ought to have in order to optimally serve these markets, where emphasis is on realistic visualisation. It is argued that 3D acquisition techniques developed for traditional applications such as visual inspection aren't necessarily the best option. Techniques should be developed that are dedicated to visualisation-specific requirements. This is exemplified with two systems that have been developed recently. One takes uncalibrated video data as input from which it generates a 3D model. A second system projects a grid of lines and gets dense 3D from a single image. This system needs some calibration, but the corresponding procedure is extremely simple. It can also be used to capture detailed, 3D scene dynamics.

In this paper the problem of obtaining 3D models from image sequences is addressed. The proposed method deals with uncalibrated monocular image sequences. No prior knowledge about the scene or about the camera is necessary to build the 3D models. The only assumptions are the rigidity of the scene objects and opaque object surfaces. The modeling system uses a 3-step approach. First, the camera pose and intrinsic parameters are calibrated by tracking salient feature points throughout the sequence. Next, consecutive images of the sequence are treated as stereoscopic image pairs and dense correspondence maps are computed by area matching. Finally, dense and accurate depth maps are computed by linking together all correspondences over the viewpoints. The depth maps are converted to triangular surfaces meshes that are texture mapped for photo-realistic appearance. The feasibility of the approach has been tested on both real and synthetic data and is illustrated here on several outdoor image sequences.

In this paper the feasibility of self-calibration in the presence of varying internal camera parameters is under investigation. A self-calibration method is presented which efficiently deals with all kinds of constraints on the internal camera parameters. Within this framework a practical method is proposed which can retrieve metric reconstruction from image sequences obtained with uncalibrated zooming/focusing cameras. The feasibility of the approach is illustrated on real and synthetic examples.

One of the key problems for robots and autonomous vehicles is the acquisition of 3D information about their environment. In this paper a flexible technique for 3D acquisition is proposed. This technique only requires an uncalibrated monocular camera. No prior knowledge about the scene or about the camera is necessary to build metric 3D models of the environment. In addition zoom and focus can be used freely. The feasibility of the approach has been tested on both real and synthetic data and is illustrated here on real image sequences.

An approach to automatically generate 3D polygonal models from stereoscopic image pairs of piecewise planar objects is presented. Dense disparity maps are computed by constrained epipolar block matching. Local surface orientations are computed from the quantized disparity map using a spline approximation under explicit consideration of quantization noise. The local surface orientation is then clustered into regions of similar surface orientation to find the dominant object planes. A photo realistic 3D polygonal model of the object is constructed by fitting planar polygons to the surfaces and by mapping original image texture to the model.

A stereoscopic scene analysis system for 3-D modeling of objects from stereoscopic image sequences is described. A dense map of 3-D surface points is obtained by image correspondence, object segmentation, interpolation, and triangulation. Emphasis is put on the accurate measurement of image correspondences from grey level images. The surface geometry of each scene object is approximated by a triangular wire-frame which stores the surface texture in texture maps. Sequence processing serves to track camera motion and to fuse surfaces from different view points into a consistent 3-D surface model. From the textured 3-D models, highly realistic image sequences from arbitrary view points can be synthesized using computer graphics techniques.>

A vision–based 3‐D scene analysis system is described that is capable to model complex real–world scenes like streets and buildings automatically from stereoscopic image pairs. Input to the system is a sequence of stereoscopic images taken with two standard CCD Cameras and TV lenses. The relative orientation of both cameras to each other is known by calibration. The camerapair is then moved throughout the scene and a long sequence of closely spaced views is recorded. Each of the stereoscopic image pairs is rectified and a dense map of 3‐D suface points is obtained by area correlation, object segmentation, interpolation, and triangulation. 3‐D camera motion relative to the scene coordinate system is tracked directly from the image sequence which allows to fuse 3‐D surface measurements from different viewpoints into a consistent 3‐D model scene. The surface geometry of each scene object is approximated by a triangular surface mesh which stores the suface texture in a texture map. From the textured 3‐D models, realistic looking image sequences from arbitrary view points can be synthesized using computer graphics.

The analysis of 3-D scenes consisting of nonrigid moving objects using 2-D image sequences is discussed. A parametric description of dynamic objects is extracted, and the time-variant scene parameters are estimated throughout the sequence by employing an analysis-by-synthesis approach. Images are synthesized from the parametric scene description and compared with the original images input to the camera. Frame differences between the synthesized and original images are evaluated to obtain an estimated scene parameter update. The analysis method is applied to videophone scenes in the form of a data compression algorithm, whereby the scene parameters are transmitted and the output sequence is synthesized at the receiver.>

Rigid 3D objects were modelled automatically from an image sequence taken by a camera that was rotated around the object. The image sequence was recorded using a calibrated camera which allows one to measure the camera positions and to estimate the true object size. The 3D object shape was obtained in two steps. The object silhouettes were employed to find the enclosing volume of the object. The volume was converted into a flexible surface representation and the 3D shape was refined based on the texture information of the object surface. Texture mapping was applied to generate a highly realistic 3D model of the object.>

A model based analysis by synthesis algorithm is presented. It combines image analysis and synthesis techniques to obtain a three dimensional scene description and to generate synthetic movies out of that scene description. A 3D model world is generated automatically out of TV image sequences containing 3D objects with naturally textured surfaces. The objects are projected into the 2D image domain and compared with subsequent images of the sequence to be analyzed. Differences between input and synthesized images serve to extract shape, motion and surface texture parameters and to adapt the 3D model scene. The model data base can be manipulated to generate synthetic movies with highly realistic, natural looking images.