We introduce a structure-from-motion method specifically designed to process the raw micro-images captured by plenoptic 2.0 cameras. Unlike traditional monocular cameras, plenoptic cameras incorporate a micro-lens array between the sensor and the main lens, capturing depth information at the expense of a more complex set of parameters to evaluate. Instead of simply integrating their projection model into the classical structure-from-motion pipeline, our contribution identifies the pinhole cameras-driven constraints and takes advantage of the inherent disparity information present in plenoptic cameras. This facilitates a robust initialization of the reconstruction. Our method shortcuts two of the limitations of the classical structure-from-motion: the ambiguity found in scenes captured with low angular disparity and the scale ambiguity. It enables the reconstruction of scenes captured by multiple uncalibrated plenoptic cameras, without using any calibration pattern or subaperture view extraction step. Our method undergoes experimental validation on both natural and synthetic datasets, showing a 10% error accuracy for relative pose estimation, which is comparable to calibration-pattern based methods. The results are robust to coarse initialization. Contrary to classical structure-from-motion, it is able to reconstruct scenes with parallel facing cameras. It also shows greater accuracy than reconstruction methods based on pinhole camera conversion.

Immersive video is gaining relevance across various fields, but its integration into real applications remains limited due to the technical challenges of depth estimation. Generating accurate depth maps is essential for 3D rendering, yet high-quality algorithms can require hundreds of seconds to produce a single frame. While real-time depth estimation solutions exist — particularly monocular deep learning-based methods and active sensors such as time-of-flight or plenoptic cameras — their depth accuracy and multiview consistency are often insufficient for depth image-based rendering (DIBR) and immersive video applications. This highlights the persistent challenge of jointly achieving real-time performance and high-quality, correlated depth across views. This paper introduces eGoRG, a GPU-accelerated depth estimation algorithm based on MPEG DERS, which employs graph cuts to achieve high-quality results. eGoRG contributes a novel GPU-based graph cuts stage, integrating block-based push-relabel acceleration and a simplified alpha expansion method. These optimizations deliver quality comparable to leading graph-cut approaches while greatly improving speed. Evaluation on an MPEG multiview dataset and a static NeRF dataset demonstrates the algorithm’s effectiveness across different scenarios. • The proposal is a novel GPU-accelerated depth estimation algorithm based on graph cuts. • Algorithm-dependent strategies are introduced to maximize the quality–time trade-off. • Depth results are comparable to high-performing graph-cut approaches while being substantially faster. • The method is training-free and can process dynamic scenes. • The algorithm is a good trade-off between quality and processing time achieving near real-time results.

Neural Radiance Fields (NeRF) demonstrate impressive capabilities in rendering novel views of specific scenes by learning an implicit volumetric representation from posed RGB images without any depth information. View synthesis is the computational process of synthesizing novel images of a scene from different viewpoints, based on a set of existing images. One big problem is the need for a large number of images in the training datasets for neural network-based view synthesis frameworks. The challenge of data augmentation for view synthesis applications has not been addressed yet. NeRF models require comprehensive scene coverage in multiple views to accurately estimate radiance and density at any point. In cases without sufficient coverage of scenes with different viewing directions, cannot effectively interpolate or extrapolate unseen scene parts. In this paper, we introduce a new pipeline to tackle this data augmentation problem using depth data. We use MPEG's Depth Estimation Reference Software and Reference View Synthesizer to add novel non-existent views to the training sets needed for the NeRF framework. Experimental results show that our approach improves the quality of the rendered images using NeRF's model. The average quality increased by 6.4 dB in terms of Peak Signal-to-Noise Ratio (PSNR), with the highest increase being 11 dB. Our approach not only adds the ability to handle the sparsely captured multiview content to be used in the NeRF framework, but also makes NeRF more accurate and useful for creating high-quality virtual views.

Neural Radiance Fields (NeRF) have demonstrated exceptional performance in generating novel views of scenes by learning implicit volumetric representations from calibrated RGB images, without depth information. A major limitation is the need for large training datasets in neural network-based view synthesis frameworks. The challenge of effective data augmentation for view synthesis remains unresolved. NeRF models require extensive scene coverage from multiple views to accurately estimate radiance and density. Insufficient coverage reduces the model’s ability to interpolate or extrapolate unseen parts of the scene effectively. In this paper, we propose a novel pipeline to address this data augmentation issue using depth map information. We use depth image-based rendering (DIBR) to overcome the lack of enough views for training NeRF. Experimental results indicate that our approach enhances the quality of rendered images using the NeRF framework, achieving an average peak signal-to-noise ratio (PSNR) increase of 7.2 dB, with a maximum improvement of 12 dB.

In the field of hyperspectral imaging, accurate and reliable data analysis is essential for many applications, including medicine, remote sensing, and material science. White calibration is a critical step in this process, as it accounts for deviations in the light source intensity and spectral distribution. Nevertheless, it is important to note that the specific geometry of the scene plays a crucial role in the white calibration, affecting the angle of incidence of the light and, subsequently, the measured spectral response. By taking surface normals and depth information into account during white calibration, we can ensure that subsequent hyperspectral images are accurately calibrated and comparable, leading to more robust and meaningful data analysis. For that matter, in this paper, we demonstrate a methodology to fuse hyperspectral and depth information, and how this fusion can help to correct different geometrical properties of the analyzed sample. A series of laboratory experiments were conducted on samples with different geometries and surface properties. Specifically, the error dispersion of the spectral signatures was reduced from 12 % to less than 4 %, a substantial improvement that highlights the potential of the proposed method.

Over the last two decades, augmented reality (AR) has led to the rapid development of new interfaces in various fields of social and technological application domains. One such domain is medicine, and to a higher extent surgery, where these visualization techniques help to improve the effectiveness of preoperative and intraoperative procedures. Following this trend, this paper presents SLIMBRAIN, a real-time acquisition and processing AR system suitable to classify and display brain tumor tissue from hyperspectral (HS) information. This system captures and processes HS images at 14 frames per second (FPS) during the course of a tumor resection operation to detect and delimit cancer tissue at the same time the neurosurgeon operates. The result is represented in an AR visualization where the classification results are overlapped with the RGB point cloud captured by a LiDAR camera. This representation allows natural navigation of the scene at the same time it is captured and processed, improving the visualization and hence effectiveness of the HS technology to delimit tumors. The whole system has been verified in real brain tumor resection operations.

Hyperspectral (HS) imaging (HSI) techniques have demonstrated to be useful in the medical field to characterize tissues without any contact and without ionizing the patient. Besides, HSI combined with supervised machine learning (ML) algorithms have proven to be an effective technique to assist neurosurgeons to resect brain tumors. This research looks at the effects of hyperparameter optimization on two common supervised ML algorithms used for brain tumor classification: support vector machines (SVM) and random forest (RF). Correctly classifying brain tumor with HS data containing low spatial and spectral information can be challenging. To tackle this problem, this study has applied hyperparameter optimization techniques on SVM and RF with 10 brain images of patients suffering from glioblastoma multiforme (GBM) with non-mutated isocitrate dehydrogenase (IDH) enzymes. These captures have 409x217 spatial resolution and 25 normalized reflectance wavelengths gathered from 665 to 960 nm with a HS snapshot camera. Results show how this work has been able to obtain 98,60% of weighted area under the curve (AUC) on the test score by employing naive optimizations like grid search (GS) or random search (RS) and even more complex methods based on Bayesian optimization (BO). Not only the weighted AUC of SVM has been improved by 8%, but BO have also enhanced the AUC of the tumor class by 22.50% in comparison with non-optimized SVM models in the state-of-the-art, achieving AUC values of 95,49% on the tumor class. Furthermore, these improvements have been illustrated with classification maps to demonstrate the importance of hyperparameter optimization on SVM to clearly classify brain tumor, whereas non-optimized models from previous studies are unable to detect the tumor.