Novel view synthesis in 360° scenes from extremely sparse input views is essential for applications like virtual reality and augmented reality. This paper presents a novel framework for novel view synthesis in extremely sparse-view cases. As typical structure-from-motion methods are unable to estimate camera poses in extremely sparse-view cases, we apply DUSt3R to estimate camera poses and generate a dense point cloud. Using the poses of estimated cameras, we densely sample additional views from the upper hemisphere space of the scenes, from which we render synthetic images together with the point cloud. Training 3D Gaussian Splatting model on a combination of reference images from sparse views and densely sampled synthetic images allows a larger scene coverage in 3D space, addressing the overfitting challenge due to the limited input in sparse-view cases. Retraining a diffusion-based image enhancement model on our created dataset, we further improve the quality of the point-cloud-rendered images by removing artifacts. We compare our framework with benchmark methods in cases of only four input views, demonstrating significant improvement in novel view synthesis under extremely sparse-view conditions for 360° scenes. The source code is available at https: //github.com/angchen-dev/hemiSparseGS.

The recently introduced 3D Gaussian Splatting and subsequent methods have achieved significantly reduced inference times for novel view synthesis. To reduce this rendering time even further, in this paper we propose four improvements which are fully compatible with the high-level Gaussian Splatting formulation and can thus be incorporated into most methods based on this paradigm. Most notably, we alter the way Gaussians are duplicated across tiles by allowing for non-square axis-aligned Gaussian bounding boxes whose sizes take into account the Gaussian's opacity information. Our experiments demonstrate that we can decrease the 3D Gaussian Splatting rendering times by up to a factor of almost 4.

Detecting surface defects is crucial for maintaining the integrity of critical infrastructure. Traditional RGB image-based methods are limited by their reliance on 2D information, which impairs accurate damage assessment. This paper introduces a novel approach that enhances defect detection and quantification, utilizing dense 3D reconstructions generated through techniques like photogrammetry or profilometry. We develop an improved robust spline fitting algorithm to estimate the undamaged surfaces from the 3D reconstructions. The residual distances between the observed and fitted surfaces are subsequently used to segment and quantify defects. By leveraging 3D data, our method resolves visual ambiguities and enables damage quantification using physically meaningful metrics. For 3D models based on optical sensing, our method complements RGB image-based defect detectors and classifiers, facilitating the fusion of visual and 3D information for a more comprehensive defect analysis. Validated on both synthetic and real-world datasets, our method demonstrates strong performance and practical feasibility.

Hyperspectral images (HSIs) play a pivotal role in fields, such as medical diagnosis and agriculture. However, it often contends with significant noise stemming from narrowband spectral filtering. Existing denoising techniques have their limitations: model-driven methods rely on manual priors and hyperparameters, while learning-based methods struggle to discern intrinsic noise patterns, as they require paired images with specific example noise for training, fail to capture critical noise distribution information, leading to unrobust denoising results. This work addresses the issue by presenting a degradation-noise-aware unfolding network (DNA-Net). Unlike training directly with the simulated noise, DNA-Net initially models general sparse and Gaussian noise through statistic distributions. It then explicitly represents image priors with a customized spectral transformer. The model is subsequently unfolded into an end-to-end (E2E) network, with hyperparameters adaptively estimated from noisy HSI and degradation models, effectively regulating each iteration. Furthermore, a novel U-shaped local-nonlocal–spectral transformer (U-LNSA) is introduced, simultaneously capturing spectral correlations, local features, and nonlocal dependencies. The integration of U-LNSA into DNA-Net establishes the first Transformer-based deep unfolding method for HSI denoising. Experimental results on synthetic and real noise validate DNA-Net’s superior performance over state-of-the-art (SOTA) methods. Moreover, the DNA-Net, trained exclusively on mixed Gaussian noise and impulse noise, demonstrates the ability to generalize to unseen noise present in real images. Code and models will be released at:https://github.com/NavyZeng/DNA-Net.

Hyperspectral images (HSIs) have extensive applications in various fields such as medicine, agriculture, and industry. Nevertheless, acquiring high signal-to-noise ratio HSI poses a challenge due to narrow-band spectral filtering. Consequently, the importance of HSI denoising is substantial, especially for snapshot hyperspectral imaging technology. While most previous HSI denoising methods are supervised, creating supervised training datasets for the diverse scenes, hyperspectral cameras, and scan parameters is impractical. In this work, we present Diff-Unmix, a self-supervised denoising method for HSI using diffusion denoising generative models. Specifically, Diff-Unmix addresses the challenge of recovering noise-degraded HSI through a fusion of Spectral Unmixing and conditional abundance generation. Firstly, it employs a learnable block-based spectral unmixing strategy, complemented by a pure transformer-based backbone. Then, we introduce a self-supervised generative diffusion network to enhance abundance maps from the spectral unmixing block. This network reconstructs noise-free Unmixing probability distributions, effectively mitigating noise-induced degradations within these components. Finally, the reconstructed HSI is reconstructed through unmixing reconstruction by blending the diffusion-adjusted abundance map with the spectral endmembers. Experimental results on both simulated and real-world noisy datasets show that Diff-Unmix achieves state-of-the-art performance.

We present a hybrid visual-inertial odometry system that relies on a state-of-the-art deep feature matching front-end and a traditional visual-inertial optimization back-end.More precisely, we develop a fully-fledged feature tracker based on the recent SuperPoint and LightGlue neural networks, that can be plugged directly to the estimation back-end of VINS-Mono.By default, this feature tracker returns extremely abundant matches.To bound the computational complexity of the back-end optimization, limiting the number of used matches is desirable.Therefore, we explore various methods to filter the matches while maintaining a high visual-inertial odometry performance.We run extensive tests on the EuRoC machine hall and Vicon room datasets, showing that our system achieves state-of-the-art odometry performance according relative pose errors.

When using a GNSS/INS sensor to provide poses to combine LiDAR scans over time, knowing the extrinsic calibration between this GNSS/INS sensor and the LiDAR device is essential. In this work we present a new calibration method that directly optimises an entropy-based loss function, starting from a decent, but suboptimal calibration, e.g. obtained using simple measurement tools such as a ruler and a protractor. Our method does not require placing any markers, nor imposes restrictions on the types of scenes. In particular, we do not rely on plane detection and planar features. Our experiments demonstrate that we can compensate for initial rotational errors up to the order of 4 degrees.

In this article, we propose a novel bilayer low-rankness measure and two models based on it to recover a low-rank (LR) tensor. The global low rankness of underlying tensor is first encoded by LR matrix factorizations (MFs) to the all-mode matricizations, which can exploit multiorientational spectral low rankness. Presumably, the factor matrices of all-mode decomposition are LR, since local low-rankness property exists in within-mode correlation. In the decomposed subspace, to describe the refined local LR structures of factor/subspace, a new low-rankness insight of subspace: a double nuclear norm scheme is designed to explore the so-called second-layer low rankness. By simultaneously representing the bilayer low rankness of the all modes of the underlying tensor, the proposed methods aim to model multiorientational correlations for arbitrary N -way ( N ≥ 3 ) tensors. A block successive upper-bound minimization (BSUM) algorithm is designed to solve the optimization problem. Subsequence convergence of our algorithms can be established, and the iterates generated by our algorithms converge to the coordinatewise minimizers in some mild conditions. Experiments on several types of public datasets show that our algorithm can recover a variety of LR tensors from significantly fewer samples than its counterparts.

Low-rank tensor completion has been widely used in computer vision and machine learning. This paper develops a novel multimodal core tensor factorization (MCTF) method combined with a tensor low-rankness measure and a better nonconvex relaxation form of this measure (NC-MCTF). The proposed models encode low-rank insights for general tensors provided by Tucker and T-SVD and thus are expected to simultaneously model spectral low-rankness in multiple orientations and accurately restore the data of intrinsic low-rank structure based on few observed entries. Furthermore, we study the MCTF and NC-MCTF regularization minimization problem and design an effective block successive upper-bound minimization (BSUM) algorithm to solve them. Theoretically, we prove that the iterates generated by the proposed models converge to the set of coordinatewise minimizers. This efficient solver can extend MCTF to various tasks such as tensor completion. A series of experiments including hyperspectral image (HSI), video and MRI completion confirm the superior performance of the proposed method.

Matching aerial images might be challenging when they contain a large number of repetitive patterns. In this paper, we propose a feature-matching method that exploits the use of Affine Oriented FAST and Rotated BRIEF (AORB) as key-point detector and feature descriptor and not accurate GPS (Global Position System) data to achieve a reliable feature matching of nadir UAV images that contain a large number of repetitive patterns. The proposed method assumes that the set of correct matches between two images only differ in a 2D translation. Experimental results show that the proposed method is able to correctly match pairs of very challenging images containing a large number of repetitive patterns.

We present Quasar, a new programming framework that handles many complex aspects in the design of smart vision systems on embedded platforms, such as parallelization, data flow management, scheduling and load balancing. Quasar, as a high-level programming language, is nearly hardware-agnostic, has a low barrier of entry and is therefore well suited for algorithm design and rapid prototyping. Through several benchmarks and application use cases we demonstrate that programs written in Quasar have a performance that is on a par with (or better than) hand-tuned CUDA and OpenACC code while the development requires much less time and is future-proof.

High Dynamic Range (HDR) displays can show images with higher color contrast levels and peak luminosities than the common Low Dynamic Range (LDR) displays. However, most existing video content is recorded and/or graded in LDR format. To show this LDR content on HDR displays, a dynamic range expansion by using an Inverse Tone Mapped Operator (iTMO) is required. In addition to requiring human intervention for tuning, most of the iTMOs don't consider artistic intentions inherent to the HDR domain. Furthermore, the quality of their results decays with peak brightness above 1000 nits. In this paper, we propose a fully-automatic inverse tone mapping operator based on mid-level mapping. This allows expanding LDR images into HDR with peak brightness over 1000 nits, preserving the artistic intentions inherent to the HDR domain. We assessed our results using full-reference objective quality metrics as HDR- VDP-2.2 and DRIM. Experimental results demonstrate that our proposed method outperforms the current state of the art.

A linear, or 1D, camera is a type of camera that sweeps a linear sensor array over the scene, rather than capturing the scene using a single impression on a 2D sensor array. They are often used in satellite imagery, industrial inspection, or hyperspectral imaging. In satellite imaging calibration is often done through a collection of ground points for which the 3D locations are known. In other applications, e.g. hyperspectral imaging, such known points are not available and annotating many different points is onerous. Hence we will use a checkerboard for calibration. The state-of-the-art method for linear camera calibration with a checkerboard becomes unstable when the checkerboards are parallel to the image plane. Our proposed method1yields more accurate camera calibrations without suffering from this shortcoming.

High Dynamic Ranges (HDR) displays can show images with higher color contrast levels and peak luminosities than the commonly used Low Dynamic Range (LDR) displays. Although HDR displays are still expensive, they are reaching the consumer market in the coming years. Unfortunately, most video content is recorded and/or graded in LDR format. Typically, dynamic range expansion by using an Inverse Tone Mapped Operator (iTMO) is required to show LDR content in HDR displays. The most common type of artifact derived from dynamic range expansion is false contouring, which negatively affects the overall image quality. In this paper, we propose a new fast iterative false-contour removal method for inverse tone mapped HDR content. We consider the false-contour removal as a signal reconstruction problem, and we solve it using an iterative Projection Onto Convex Sets (POCS) minimization algorithm. Unlike most other false-contour removal techniques, we define reconstruction constraints taking into account the iTMO used. Experimental results demonstrate the effectiveness of the proposed method to remove false contours while preserving details in the image. In order speed-up the execution time, the proposed method was implemented to run on a GPU. We were able to show that it can be used to remove false contours in real-time from an inverse tone mapped High-definition HDR video sequences at 24 fps.

In this paper we present a novel approach to quickly obtain detailed 3D reconstructions of large scale environments. The method is based on the consecutive registration of 3D point clouds generated by modern lidar scanners such as the Velodyne HDL-32e or HDL-64e. The main contribution of this work is that the proposed system specifically deals with the problem of sparsity and inhomogeneity of the point clouds typically produced by these scanners. More specifically, we combine the simplicity of the traditional iterative closest point (ICP) algorithm with the analysis of the underlying surface of each point in a local neighbourhood. The algorithm was evaluated on our own collected dataset captured with accurate ground truth. The experiments demonstrate that the system is producing highly detailed 3D maps at the speed of 10 sensor frames per second.

In this letter, we present an efficient image denoising method combining quadtree-based nonlocal means (NLM) and locally adaptive principal component analysis. It exploits nonlocal multiscale self-similarity better, by creating sub-patches of different sizes using quadtree decomposition on each patch. To achieve spatially uniform denoising, we propose a local noise variance estimator combined with denoiser based on locally adaptive principal component analysis. Experimental results demonstrate that our proposed method achieves very competitive denoising performance compared with state-of-the-art denoising methods, even obtaining better visual perception at high noise levels.

Non-local self-similarity based image denoising depends strictly on similarity measure. The denoising performance is determined based on the ability to reliably find sufficient number of similar patches. In this paper, we propose a rotation invariant similarity measure to fully exploit the image non-local self-similarity. Instead of using image patches, we employ local frequency descriptors, that are rotation invariant and robust to noise, to measure the similarity. Thus, both translational and rotational similarity can be handled even at high noise level. The comparative experimental results show that the proposed method is effective as a rotation invariant similarity measure, and it can consistently improve the performance of non-local means algorithm to achieve better denoising results.

Wavelet-based demosaicing techniques have the advantage of being computationally relatively fast, while having a reconstruction performance that is similar to state-of-the-art techniques. Because the demosaicing rules are linear, it is fairly simple to integrate denoising into the demosaicing. In this paper, we present a method that performs joint denoising and demosaicing, using a Gaussian Scale Mixture (GSM) prior model, thereby modeling the local edge direction as a hidden variable. The results indicate that this technique offers a better reconstruction performance (in PSNR sense and visually) than sequential demosaicing and denoising. On a recent GPU, our algorithm takes 3.5 s for reconstructing a 12 megapixel RAW digital camera image.

This paper presents a two-stage denoising method for hyper-spectral image (HSI) by combining kernel principal component analysis (KPCA) and total variation (TV). In the first stage, we use KPCA denoising to reduce spectrally uncorre-lated noise. In the second stage, the information content is largely separated from the remaining noise by means of principal component analysis (PCA). The remaining noise is then efficiently removed by fast primal-dual TV denoising in low-energy PCA channels. Experimental results on simulated and real HSIs are very encouraging.

In the field of image denoising, the non-local means (NLMS) filter is a conceptually simple, yet powerful technique. This filter exploits non-local, i.e. spatially repetitive, structure in natural images to estimate noise-free structure. In contrast, a wide variety of image restoration problems have been solved exploiting local smoothness of natural images, e.g. by enforcing sparsity of images when subjected to a multi-resolution transform. In this paper we introduce the prior knowledge of non-local repetitiveness of image structures into a broad multi-resolution image restoration framework. The proposed framework allows the power of the NLMS filter, supplemented by multi-resolution sparsity, to be extended for a wide variety of image restoration problems, such as demosaicing, deconvolution, reconstruction from insufficient measurements,... in a conceptually simple way.

In this paper, we present a first-order primal-dual algorithm for tackling the joint demosaicking and deconvolution problem. The proposed algorithm exploits the sparsity of both discrete gradient (TV) and shearlet coefficients as prior knowledge. In order to deal with this sparsity across the color channels, we first decorrelate the signals in color space before sparsifying them spatially, resulting in a separable transform. We demonstrate that this approach yields better results than employing group sparsity strategies. We propose to update the decorrelation operator during the image reconstruction, this approach will result in a significant improvement in PSNR. By relaxing the sparsity of the chrominance signals, we obtain both better objective and subjective image quality compared to other state-of-the-art demosaicking and deconvolution algorithms. Also, color artifacts due to demosaicking are suppressed very well.

In this paper, we present a new method for High Dynamic Range (HDR) reconstruction based on a set of multiple photographs with different exposure times. While most existing techniques take a deterministic approach by assuming that the acquired low dynamic range (LDR) images are noise-free, we explicitly model the photon arrival process by assuming sensor data corrupted by Poisson noise. Taking the noise characteristics of the sensor data into account leads to a more robust way to estimate the non-parametric camera response function (CRF) compared to existing techniques. To further improve the HDR reconstruction, we adopt the split-Bregman framework and use Total Variation for regularization. Experimental results on real camera images and ground-truth data show the effectiveness of the proposed approach.

Parallel Imaging MRI (pMRI) and Compressive Sensing (CS) are two reconstruction techniques that have recently been applied to increase MRI performance. In this paper we demonstrate that a combined analysis of the pMRI and CS problems leads to a conceptually simple, yet effective technique that outperforms independent approaches to both reconstruction problems. We argue that the proposed technique is also naturally resilient to noise, due to its relation to the MAP image denoising formulation. A modified Basis Pursuit (BP) formulation of the CS-MRI problem allows it to handle the pMRI problem at the same time. We also present an exact solution to this BP problem, using the split Bregman technique, with discrete shearlet transform (DST) regularization. The DST is an excellent choice for natural image applications, due to its optimal sparsity property. Results show that this Compressive Parallel Sensing (COMPASS) reconstruction algorithm outperforms more traditional MRI reconstruction algorithms in both pMRI and CS experiments.

In this paper, we derive a novel robust image alignment technique that performs joint geometric and photometric registration in the total least square sense. The main idea is to use the total least square metrics instead of the ordinary least square metrics, which is commonly used in the literature. While the OLS model indicates that the target image may contain noise and the reference image should be noise-free, this puts a severe limitation on practical registration problems. By introducing the TLS model, which allows perturbations in both images, we can obtain mutually consistent parameters. Experimental results show that our method is indeed much more consistent and accurate in presence of noise compared to existing registration algorithms.

In high dynamic range (HDR) imaging, multiple photographs with different exposure times are combined into a radiance map, which reflects the radiance in real-life scenes. This involves recovering the response function of the imaging process. The technique proposed by Debevec and Malik is a well-known HDR image synthesis algorithm, but the computational complexity is relatively high, which limits the possible image size and the reconstruction quality. In this paper we present an improved joint optimization technique for estimating the camera response function (CRF) and the radiance map and a new sequential two-step optimization technique, which first estimates the CRF and then reconstructs the radiance map, resulting in better visual results and a higher SNR in remarkably less computation time.

In this paper we present a novel method for reconstructing low-resolution text images. Unlike other conventional interpolation methods, the unknown pixel value is not estimated based on its local surrounding neighbourhood, but on the whole text image. In particularly, we exploit the repetitive behaviour of the characters. A great advantage of our proposed approach is that we have more information at our disposal, which leads to a better reconstruction of the interpolated image. Results show the effectiveness of our proposed method and its superiority at very large magnifications to traditional interpolation methods.

In this paper we present a novel method for interpolating images and we introduce the concept of non-local interpolation. Unlike other conventional interpolation methods, the estimation of the unknown pixel values is not only based on its local surrounding neighbourhood, but on the whole image (non-locally). In particularly, we exploit the repetitive character of the image. A great advantage of our proposed approach is that we have more information at our disposal, which leads to better estimates of the unknown pixel values. Results show the effectiveness of non-local interpolation and its superiority at very large magnifications to other interpolation methods.

In this paper we present a method for interpolating images that also preserves sharp edge information. We concentrate on tackling blurred edges by mapping level curves of the image. Level curves or isophotes are spatial curves with constant intensity. The mapping of these intensities can be seen as a local contrast enhancement problem, therefore we can use contrast enhancement techniques coupled with additional constraints for the interpolation problem. A great advantage of this approach is that the shape of the level set contours is preserved and no explicit edge detection is needed here. Results show an improvement in visual quality: edges are sharper and ringing effects are removed.

The huge amount of incoming synthetic aperture radar (SAR) data nowadays demands the need for automatic image registration. Due the presence of speckle noise and the huge size of SAR images, registering SAR images is more difficult than traditional image registration methods. In This work we present an automatic hierarchical registration method, based on polar block matching, that can handle the registration of SAR images efficiently. Results show us that our proposed registration method outperforms the manual registration and that our method is very robust against speckle noise. An additional advantage of our registration method is that there is no need for a despeckling preprocessing step.

In this paper we point out that the general principle "do not compute what you do not need to compute" can be applied easily and successfully within a MPEG audio decoding strategy. More specifically, we will discuss the problem of eliminating costly computation cycles being wasted at processing useless zero-valued data. Hence, the title: "do not zero-pute". At first, this may all sound somewhat obvious or trivial. Indeed, this can be true in many cases, but experience gathered in various teaching related projects during several academic years has also lead us to believe the opposite. Moreover, a survey of the existing literature quickly reveals that the approach discussed below has not been investigated and documented properly. Although we will only illustrate our optimization approach by discussing the MPEG-audio layer II decoding process in detail, we hope the reader will be able to apply, extend, and implement the basic principles presented here within many other applications.