In this paper, we introduce a novel multi view-based method for completing high-resolution 3D point clouds of partial object shapes obtained by mobile laser scanning (MLS) platforms. Our approach estimates both the geometry and color cues of the missing or incomplete object segments, by projecting the 3D input point cloud by multiple virtual cameras, and performing 2D inpainting in the image domains of the different views. In contrast to existing state-of-the-art methods, our method can generate point clouds consisting of a variable number of points, depending on the detailedness of the input measurement, which property highly facilitates the efficient processing of MLS data with inhomogeneous point density. For training and quantitative evaluation of the proposed method, we provide a new point cloud dataset that consists of both synthetic point clouds of four different street objects with accurate ground truth, and real MLS measurements of partially or fully scanned vehicles. The quantitative and qualitative experiments on the provided dataset demonstrate that our method surpasses state-of-the-art approaches in reconstructing the local fine geometric structures as well as in estimating the overall shape and color pattern of the objects.

3D vehicle shape completion is a key task in urban environment reconstruction, since the available sensors and 3D scanning procedures (such as Mobile Laser Scanning) in an outdoor city scene cannot usually extract the entire car shapes, which can be necessary for specifying their geometric properties for further analysis or realistic visualization. In this paper, we propose a novel multi-view based 3D object point cloud completion technique. In contrast to existing approaches, our method operates on 2D images formed by projecting the point cloud from several virtual camera positions around the object of interest. Both color and geometrical information is considered during the process, generating dense textured point clouds, displaying realistic patterns in the missing regions from the partial inputs. We present both quantitative and qualitative tests on various synthetic and real laser scanned vehicle point clouds, which demonstrate that our method surpasses existing state-of-the-art approaches. By applying it to vehicles from the Shapenet dataset, our approach outperforms recent techniques in terms of Earth Mover’s Distance (EMD) and Chamfer Distance (CD) by 43.8% and 12.17%, respectively.

In this paper we introduce a novel end-to-end blind inpainting algorithm for masonry wall images, performing the automatic detection and virtual completion of occluded or damaged wall regions. For this purpose, we propose a three-stage deep neural network that comprises a U-Net-based sub-network for wall segmentation into brick, mortar and occluded regions, which is followed by a two-stage adversarial inpainting model. The first adversarial network predicts the schematic mortar-brick pattern of the occluded areas based on the observed wall structure, providing in itself valuable structural information for archeological and architectural applications. Finally, the second adversarial network predicts the RGB pixel values yielding a realistic visual experience for the observer. While the three stages implement a sequential pipeline, they interact through dependencies of their loss functions admitting the consideration of hidden feature dependencies between the different network components. For training and testing the network a new dataset has been created, and an extensive qualitative and quantitative evaluation versus the state-of-the-art is given.

In this paper we propose an end-to-end, automatic, online camera-LIDAR calibration approach, for application in self driving vehicle navigation. The main idea is to connect the image domain and the 3D space by generating point clouds from camera data while driving, using a structure from motion (SfM) pipeline, and use it as the basis for registration. As a core step of the algorithm we introduce an object level alignment to transform the generated and captured point clouds into a common coordinate system. Finally, we calculate the correspondences between the 2D image domain and the 3D LIDAR point clouds, to produce the registration. We evaluated the method in various different real life traffic scenarios.

This paper presents new approaches for gait and activity analysis based on data streams of a rotating multibeam (RMB) Lidar sensor. The proposed algorithms are embedded into an integrated 4D vision and visualization system, which is able to analyze and interactively display real scenarios in natural outdoor environments with walking pedestrians. The main focus of the investigations is gait-based person reidentification during tracking and recognition of specific activity patterns, such as bending, waving, making phone calls, and checking the time looking at wristwatches. The descriptors for training and recognition are observed and extracted from realistic outdoor surveillance scenarios, where multiple pedestrians are walking in the field of interest following possibly intersecting trajectories; thus, the observations might often be affected by occlusions or background noise. Since there is no public database available for such scenarios, we created and published a new Lidar-based outdoor gait and activity data set on our website that contains point cloud sequences of 28 different persons extracted and aggregated from 35-min-long measurements. The presented results confirm that both efficient gait-based identification and activity recognition are achievable in the sparse point clouds of a single RMB Lidar sensor. After extracting the people trajectories, we synthesized a free-viewpoint video, in which moving avatar models follow the trajectories of the observed pedestrians in real time, ensuring that the leg movements of the animated avatars are synchronized with the real gait cycles observed in the Lidar stream.

In this paper we introduce a new deep learning based approach to detect and remove phantom objects from point clouds produced by mobile laser scanning (MLS) systems. The phantoms are caused by the presence of scene objects moving concurrently with the MLS platform, and appear as long, sparse but irregular point cloud segments in the measurements. We propose a new 3D CNN framework working on a voxelized column-grid to identify the phantom regions. We quantitatively evaluate the proposed model on real MLS test data, and compare it to two different reference approaches.

In this letter, we present a new approach for object classification in continuously streamed Lidar point clouds collected from urban areas. The input of our framework is raw 3-D point cloud sequences captured by a Velodyne HDL-64 Lidar, and we aim to extract all vehicles and pedestrians in the neighborhood of the moving sensor. We propose a complete pipeline developed especially for distinguishing outdoor 3-D urban objects. First, we segment the point cloud into regions of ground, short objects (i.e., low foreground), and tall objects (high foreground). Then, using our novel two-layer grid structure, we perform efficient connected component analysis on the foreground regions, for producing distinct groups of points, which represent different urban objects. Next, we create depth images from the object candidates, and apply an appearance-based preliminary classification by a convolutional neural network. Finally, we refine the classification with contextual features considering the possible expected scene topologies. We tested our algorithm in real Lidar measurements, containing 1485 objects captured from different urban scenarios.

In this paper, we introduce a probabilistic approach for extracting the complex hierarchical object structures from the digital images used by various vision applications. The proposed framework extends conventional marked point process (MPP) models by: 1) admitting object-subobject ensembles in parent-child relationships; and 2) allowing corresponding objects to form coherent object groups, by a Bayesian segmentation of the population. Different from earlier, highly domain specific attempts on MPP generalization, the proposed model is defined at an abstract level, providing clear interfaces for applications in various domains. We also introduce a global optimization process for the multi-layer framework for finding optimal entity configurations, considering the observed data, prior knowledge, and interactions between the neighboring and the hierarchically related objects. The proposed method is demonstrated in three different application areas: built in area analysis in remotely sensed images, traffic monitoring on airborne, and mobile laser scanning (Lidar) data and optical circuit inspection. A new benchmark database is published for the three test cases, and the model's performance is quantitatively evaluated.

In this paper we propose a general approach for registration of point clouds obtained by various mobile laser scanning technologies. Our method is able to robustly match measurements with significantly different density characteristic including the sparse and inhomogeneous instant 3D (I3D) data taken be self-driving cars, and the dense and regular point clouds captured by mobile mapping systems (MMS) for virtual city generation. The core steps of the algorithm are robust scan segmentation, abstract street object extraction, object based coarse transformation estimation in the Hough accumulator space, and point-level registration refinement. Experimental results are provided using three different sensors: Velodyne HDL64 and VLP16 I3D scanners, and a Riegl VMX450 MMS. Application examples are shown regarding self localization of autonomous cars through crossmodal I3D and MMS frame registration, IMU-less SLAM and change detection based on I3D data.

In this paper, we present a new object-based hierarchical model for the joint probabilistic extraction of vehicles and groups of corresponding vehicles-called traffic segments-in airborne light detection and ranging (Lidar) point clouds collected from dense urban areas. First, the 3-D point set is classified into terrain, vehicle, roof, vegetation, and clutter classes. Then, the points with the corresponding class labels and echo strength (i.e., intensity) values are projected to the ground. In the obtained 2-D class and intensity maps, we approximate the top view projections of vehicles by rectangles. Since our tasks are simultaneously the extraction of the rectangle population which describes the position, size, and orientation of the vehicles and grouping the vehicles into the traffic segments, we propose a hierarchical two-level marked point process (MPP) (L2MPP) model for the problem. The output vehicle and traffic segment configurations are extracted by an iterative stochastic optimization algorithm. We have tested the proposed method with real data of a discrete-return Lidar sensor providing up to four range measurements for each laser pulse. Using manually annotated ground-truth information on a data set containing 1009 vehicles, we provide quantitative evaluation results showing that the L2MPP model surpasses two earlier grid-based approaches, a 3-D point-cloud-based process and a single-layer MPP solution. The accuracy of the proposed method measured in F-rate is 97% at object level, 83% at pixel level, and 95% at group level.

In this paper we introduce a probabilistic approach for extracting complex hierarchical object structures from digital images. The proposed framework extends conventional Marked Point Process models by (i) admitting object-subobject ensembles in parent-child relationships and (ii) allowing corresponding objects to form coherent object groups. The proposed method is demonstrated in three application areas: optical circuit inspection, built in area analysis in aerial images, and traffic monitoring on airborne Lidar data.

In this paper, we propose a multiframe marked point process model of line segments and point groups for automatic target structure extraction and tracking in inverse synthetic aperture radar (ISAR) image sequences. To deal with scatterer scintillations and high speckle noise in the ISAR frames, we obtain the resulting target sequence by an iterative optimization process, which simultaneously considers the observed image data and various prior geometric interaction constraints between the target appearances in the consecutive frames. A detailed quantitative evaluation is performed on eight real ISAR image sequences of different carrier ships and airplane targets, using a test database containing 545 manually annotated frames.

In this paper, we introduce a Bayesian approach on multiple people localization in multicamera systems. First, pixel-level features are extracted, which are based on physical properties of the 2-D image formation process, and provide information about the head and leg positions of the pedestrians, distinguishing standing and walking people, respectively. Then, features from the multiple camera views are fused to create evidence for the location and height of people in the ground plane. This evidence accurately estimates the leg position even if either the area of interest is only a part of the scene or the overlap ratio of the silhouettes from irrelevant outside motions with the monitored area is significant. Using this information, we create a 3-D object configuration model in the real world. We also utilize a prior geometrical constraint, which describes the possible interactions between two pedestrians. To approximate the position of the people, we use a population of 3-D cylinder objects, which is realized by a marked point process. The final configuration results are obtained by an iterative stochastic energy optimization algorithm. The proposed approach is evaluated on two publicly available datasets, and compared to a recent state-of-the-art technique. To obtain relevant quantitative test results, a 3-D ground truth annotation of the real pedestrian locations is prepared, while two different error metrics and various parameter settings are proposed and evaluated showing the advantages of our proposed model.

In this paper, we introduce a new probabilistic method which integrates building extraction with change detection in remotely sensed image pairs. A global optimization process attempts to find the optimal configuration of buildings, considering the observed data, prior knowledge, and interactions between the neighboring building parts. We present methodological contributions in three key issues: 1) We implement a novel object-change modeling approach based on Multitemporal Marked Point Processes, which simultaneously exploits low-level change information between the time layers and object-level building description to recognize and separate changed and unaltered buildings. 2) To answer the challenges of data heterogeneity in aerial and satellite image repositories, we construct a flexible hierarchical framework which can create various building appearance models from different elementary feature-based modules. 3) To simultaneously ensure the convergence, optimality, and computation complexity constraints raised by the increased data quantity, we adopt the quick Multiple Birth and Death optimization technique for change detection purposes, and propose a novel nonuniform stochastic object birth process which generates relevant objects with higher probability based on low-level image features.

In this paper we introduce a probabilistic approach on multiple person localization using multiple calibrated camera views. People present in the scene are approximated by a population of cylinder objects in the 3-D world coordinate system, which is a realization of a Marked Point Process. The observation model is based on the projection of the pixels of the obtained motion masks in the different camera images to the ground plane and to other parallel planes with different height. The proposed pixel-level feature is based on physical properties of the 2-D image formation process and can accurately localize the leg position on the ground plane and estimate the height of the people, even if the area of interest is only a part of the scene, meanwhile silhouettes from irrelevant outside motions may significantly overlap with the monitored region in some of the camera views. We introduce an energy function, which contains a data term calculated from the extracted features and a geometrical constraint term modeling the distance between two people. The final configuration results (location and height) are obtained by an iterative stochastic energy optimization process, called the Multiple Birth and Death dynamics. The proposed approached is compared to a recent state-of-the-art technique in a publicly available dataset and its advantages are quantitatively demonstrated.

In this paper we introduce a probabilistic approach for optical quality checking of Solder Pastes (SP) in Printed Circuit Boards (PCB). Dealing with unregistered image inputs, the task is to address at the same time SP identification, and detecting special soldering errors, called scooping. For this reason we introduce a novel Hierarchical Marked Point Process (HMPP) framework, which is able to handle the paste and scooping extraction problems simultaneously, so that the SPs and the included scoops have a parent-child relationship. A global optimization process attempts to find the optimal configuration of entities, considering the observed data, prior knowledge, and interactions between the neighboring circuit elements. The proposed method is evaluated on a real PCB image set containing more than 3000 SPs and 600 scooping artifacts.

In this paper we propose a Multi-frame Marked Point Process model for automatic target detection and tracking in Inverse Synthetic Aperture Radar (ISAR) image sequences. For purposes of dealing with high ISAR noise, we obtain the optimal target sequence by an energy minimization process, which simultaneously considers the observed image data and prior geometric interaction constraints between the target appearances in the consecutive frames. Finally, a robust permanent scatterer detection step is introduced to support the target identification process. Evaluation is performed on real ISAR image sequences of ship targets.

In this paper we introduce a probabilistic approach of building extraction in remotely sensed images. To cope with data heterogeneity we construct a flexible hierarchical framework which can create various building appearance models from different elementary feature based modules. A global optimization process attempts to find the optimal configuration of buildings, considering simultaneously the observed data, prior knowledge, and interactions between the neighboring building parts. The proposed method is evaluated on various aerial image sets containing more than 500 buildings, and the results are matched against two state-of-the-art techniques.

In this paper we introduce new model for efficient building extraction and change detection in sparsely populated remotely sensed image pairs. In the two time layers interacting marked point processes recognize the buildings while a priori constraints and low level similarity information are involved in the model. The exploration is accelerated through a non-uniform object birth process, which proposes objects in the built-in regions with higher probability based on global statistical image features.

In this paper we introduce a new probabilistic method which integrates building extraction with change detection in remotely sensed image pairs. A global optimization process attempts to find the optimal configuration of buildings, considering the observed data, prior knowledge, and interactions between the neighboring building parts. The accuracy is ensured by a Bayesian object model verification, meanwhile the computational cost is significantly decreased by a non-uniform stochastic object birth process, which proposes relevant objects with higher probability based on low-level image features.

In this paper, we propose a probabilistic model for detecting relevant changes in registered aerial image pairs taken with the time differences of several years and in different seasonal conditions. The introduced approach, called the conditional mixed Markov model, is a combination of a mixed Markov model and a conditionally independent random field of signals. The model integrates global intensity statistics with local correlation and contrast features. A global energy optimization process ensures simultaneously optimal local feature selection and smooth observation-consistent segmentation. Validation is given on real aerial image sets provided by the Hungarian Institute of Geodesy, Cartography and Remote Sensing and Google Earth.

We propose a new Bayesian method for detecting the regions of object displacements in aerial image pairs. We use a robust but coarse 2-D image registration algorithm. Our main challenge is to eliminate the registration errors from the extracted change map. We introduce a three-layer Markov random field (L(3)MRF) model which integrates information from two different features, and ensures connected homogenous regions in the segmented images. Validation is given on real aerial photos.

In the paper we propose a novel multi-layer Mixed Markov model for detecting relevant changes in registered aerial images taken with significant time differences. The introduced approach combines global intensity statistics with local correlation and contrast features. A global energy optimization process simultaneously ensures optimal local feature selection and smooth, observation-consistent classification. Validation is given on real aerial photos.

In in this paper, we propose a new model regarding foreground and shadow detection in video sequences. The model works without detailed a priori object-shape information, and it is also appropriate for low and unstable frame rate video sources. Contribution is presented in three key issues: 1) we propose a novel adaptive shadow model, and show the improvements versus previous approaches in scenes with difficult lighting and coloring effects; 2) we give a novel description for the foreground based on spatial statistics of the neighboring pixel values, which enhances the detection of background or shadow-colored object parts; 3) we show how microstructure analysis can be used in the proposed framework as additional feature components improving the results. Finally, a Markov random field model is used to enhance the accuracy of the separation. We validate our method on outdoor and indoor sequences including real surveillance videos and well-known benchmark test sets.

In this paper, we give a probabilistic model for automatic change detection on airborne images taken with moving cameras. To ensure robustness, we adopt an unsupervised coarse matching instead of a precise image registration. The challenge of the proposed model is to eliminate the registration errors, noise and the parallax artifacts caused by the static objects having considerable height (buildings, trees, walls etc.) from the difference image. We describe the background membership of a given image point through two different features, and introduce a novel three-layer Markov random field (MRF) model to ensure connected homogenous regions in the segmented image.

We outline our latest methods and algorithmic solutions for detection of concurrent motions and registering cameras in real-life surveillance systems. We show that cameras can be registered in several rather unfavorable conditions, based on: (1) unpredictable motion without structured background or defined object shapes or (2) walking persons of undefined silhouettes and short detectable walking distances or (3) shadows of undefined structures in front of flickering background. These methods are tested in real-life sequences and we show that flexible 3D camera registration is possible even in bad lighting conditions and in the lack of any known structures or motions.

In the paper we introduce an algorithm for matching partially overlapping image-pairs where the object of interest is in motion, even if the motion is discontinuous and in an unstructured environment. In our previous work [Z. Szlavik et al, 2004] we have shown that by using co-motion statistics matching of overlapping views can be done and then the projective geometry can be estimated. Here, we show how to optimize searching for concurrently moving pixels. The robust algorithm we describe here finds point correspondences in two images by using entropy-based thresholding and without searching for any structures and without the need for tracking continuous motion. Our method makes it possible to (re)calibrate multicamera systems without human assistance.