Reconstructing the surrounding surface geometry from recorded driving sequences poses a significant challenge due to the limited image overlap and complex topology of urban environments. SoTA neural implicit surface reconstruction methods often struggle in such setting, either failing due to small vision overlap or exhibiting suboptimal performance in accurately reconstructing both the surface and fine structures. To address these limitations, we introduce J-NeuS, a novel hybrid implicit surface reconstruction method for large driving sequences with outward facing camera poses. J-NeuS leverages cross-representation uncertainty estimation to tackle ambiguous geometry caused by limited observations. Our method performs joint optimization of two radiance fields in addition to guided sampling achieving accurate reconstruction of large areas along with fine structures in complex urban scenarios. Extensive evaluation on major driving datasets demonstrates the superiority of our approach in reconstructing large driving sequences with limited image overlap, outperforming concurrent SoTA methods.

Diabetic macular edema (DME) and age-related macular degeneration (AMD) are major causes of vision impairment and blindness. While many classification applications for these diseases achieve high performance, they often overlook the crucial aspect of dataset and image quality, leading to potential erroneous predictions. This study emphasizes the importance of data quality in medical image classification, specifically for retinal imaging. We propose an Optical Coherence Tomography (OCT) image quality evaluation model using the pre-trained ARNIQA (leArning distoRtion maNifold for Image Quality Assessment) model to accurately identify retinal diseases autonomously. Our methodology includes a three-class classification system utilizing two Convolutional Neural Network (CNN) models, ResNet50 and Xception, applied to three datasets: the original dataset, a subset of high-quality images, and a subset of low-quality images. Using a Tunisian OCT dataset of 2887 images, we demonstrate the efficacy of our approach, achieving $100 \%$ accuracy with highquality images.

Neural implicit surface representation methods have recently shown impressive 3D reconstruction results. However, existing solutions struggle to reconstruct driving scenes due to their large size, highly complex nature and their limited visual observation overlap. Hence, to achieve accurate reconstructions, additional supervision data such as LiDAR, strong geometric priors, and long training times are required. To tackle such limitations, we present ViiNeuS, a new hybrid implicit surface learning method that efficiently initializes the signed distance field to reconstruct large driving scenes from 2D street view images. ViiNeuS’s hybrid architecture models two separate implicit fields: one representing the volumetric density of the scene, and another one representing the signed distance to the surface. To accurately reconstruct urban outdoor driving scenarios, we introduce a novel volume-rendering strategy that relies on self-supervised probabilistic density estimation to sample points near the surface and transition progressively from volumetric to surface representation. Our solution permits a proper and fast initialization of the signed distance field without relying on any geometric prior on the scene, compared to concurrent methods. By conducting extensive experiments on four outdoor driving datasets, we show that ViiNeuS can learn an accurate and detailed 3D surface representation of various urban scene while being two times faster to train compared to previous state-of-the-art solutions.

International audience

Diabetic macular edema (DME) and Age-Related Macular Degeneration (AMD) are two of the most common disorders that can cause blindness in a population and primarily cause retinal degradation. The application of multiple deep learning algorithms on Optical Coherence Tomography OCT) images to detect these disorders demonstrates excellent performance. However, because these algorithms include black box features, medical professionals are hesitant to fully trust the results. To address these challenges, we present a modified convolutional neural network based on the xception architecture for diagnosing DME and AMD using optical coherence tomography (OCT) images. To demonstrate the model’s transparency and trustworthiness, we used the Grad-CAM technique, which incorporates Explainable AI into the research and improves model interpretability. This technique assists medical specialists in demystifying deep learning algorithms and obtaining more information about the critical areas in OCT images used for prediction. The proposed model achieved an accuracy of 99.87%, a precision of 99.67%, and a recall of 98.29% on a dataset of 934 images.

Data from different modalities, such as infrared and visible images, can offer complementary information, and integrating such information can significantly enhance the capabilities of a system to perceive and recognize its surroundings. Thus, multi-modal object detection has widespread applications, particularly in challenging weather conditions like low-light scenarios. The core of multi-modal fusion lies in developing a reasonable fusion strategy, which can fully exploit the complementary features of different modalities while preventing a significant increase in model complexity. To this end, this paper proposes a novel lightweight cross-fusion module named Channel-Patch Cross Fusion (CPCF), which leverages Channel-wise Cross-Attention (CCA), Patch-wise Cross-Attention (PCA) and Adaptive Gating (AG) to encourage mutual rectification among different modalities. This process simultaneously explores commonalities across modalities while maintaining the uniqueness of each modality. Furthermore, we design a versatile intermediate fusion framework that can leverage CPCF to enhance the performance of multi-modal object detection. The proposed method is extensively evaluated on multiple public multi-modal datasets, namely FLIR, LLVIP, and DroneVehicle. The experiments indicate that our method yields consistent performance gains across various benchmarks and can be extended to different types of detectors, further demonstrating its robustness and generalizability. Our codes are available athttps://github.com/Superjie13/CPCF_Multispectral.

Retinal pathologies like choroidal neovascularization (CNV), drusen, and diabetic macular edema (DME) can give rise to microvascular alterations in the retina, ultimately resulting in vision impairment. The manual detection of these diseases poses a significant challenge and necessitates specialized medical expertise. To address this challenge, our study introduces novel deep learning methods for the detection of these ocular pathologies automatically and based on optical coherence tomography (OCT) scans. In our experimental setup, we utilized a dataset comprising 6000 OCT images sourced from the publicly available Kaggle dataset. Through comprehensive evaluations, our study revealed that the implementation of a bilinear convolutional neural network (B-CNN) yielded the highest classification score, surpassing the accuracy achieved by alternative models. Furthermore, when compared to other deep learning networks, our proposed approach showcased superior performance in the early diagnosis of these three ocular diseases.

Multi-modal scene parsing is a prevalent topic in robotics and autonomous driving since the knowledge of different modalities can complement each other. Recently, the success of self-attention-based methods has demonstrated the effectiveness of capturing long-range dependencies. However, the tremendous cost dramatically limits the application of this idea in multi-modal fusion. To alleviate this problem, this paper designs a multimodal additive-attention cross-fusion block (AC) and an efficient AC variant (EAC) to effectively capture global awareness among different modalities. Moreover, a simple yet efficient transformer-based trans-context block (TC) is also presented to incorporate contextual information. Based on the above components, we propose a light hybrid cross-fusion network (HCFNet), which can explore long-range dependencies of multi-modal information while keeping local details. Finally, we conduct comprehensive experiments and analyses on both indoor (NYUv2-13, -40) and outdoor (Cityscapes-11) datasets. Experimental results show that the proposed HCFNet outperforms current start-of-the-art methods with mIoU scores of 66.9% and 51.5% on NYUv2-13 and -40 class settings, respectively. Our model also shows a competitive mIoU score of 80.6% on the Cityscapes-11 dataset. The code will be available at https://github.com/Superjie13/HCFNet.

Few-shot segmentation presents a significant challenge for semantic scene understanding under limited supervision. Namely, this task targets at generalizing the segmentation ability of the model to new categories given a few samples. In order to obtain complete scene information, we extend the RGB-centric methods to take advantage of complementary depth information. In this paper, we propose a two-stream deep neural network based on metric learning. Our method, known as RDNet, learns class-specific prototype representations within RGB and depth embedding spaces, respectively. The learned prototypes provide effective semantic guidance on the corresponding RGB and depth query image, leading to more accurate performance. Moreover, we build a novel outdoor scene dataset, known as Cityscapes-3i, using labeled RGB images and depth images from the Cityscapes dataset. We also perform ablation studies to explore the effective use of depth information in few-shot segmentation tasks. Experiments on Cityscapes-3ishow that our method achieves excellent results with visual and complementary geometric cues from only a few labeled examples.

Deep learning-based image understanding techniques require a large number of labeled images for training. Few-shot semantic segmentation, on the contrary, aims at generalizing the segmentation ability of the model to new categories given only a few labeled samples. To tackle this problem, we propose a novel prototypical network (MAPnet) with multiscale feature attention. To fully exploit the representative features of target classes, we firstly extract rich contextual information of labeled support images via a multiscale feature enhancement module. The learned prototypes from support features provide further semantic guidance on the query image. Then we adaptively integrate multiple similarity-guided probability maps by attention mechanism, yielding an optimal pixel-wise prediction. Furthermore, the proposed method was validated on the PASCAL-5idataset in terms of 1-way N-shot evaluation. We also test the model with weak annotations, including scribble and bounding box annotations. Both the qualitative and quantitative results demonstrate the advantages of our approach over other state-of-the-art methods.

This paper deals with new augmentation methods for an unconventional imaging modality sensitive to the physics of the observed scene called polarimetry. In nature, polarized light is obtained by reflection or scattering. Robotics applications in urban environments are subject to many obstacles that can be specular and therefore provide polarized light. These areas are prone to segmentation errors using standard modalities but could be solved using information carried by the polarized light. Deep Convolutional Neural Networks (DCNNs) have shown excellent segmentation results, but require a significant amount of data to achieve best performances. The lack of data is usually overcomed by using augmentation methods. However, unlike RGB images, polarization images are not only scalar (intensity) images and standard augmentation techniques cannot be applied straightforwardly. We propose enhancing deep learning models through a regularized augmentation procedure applied to polarimetric data in order to characterize scenes more effectively under challenging conditions. We subsequently observe an average of 18.1 % improvement in IoU between not augmented and regularized training procedures on real world data.

Monocular depth estimation is a recurring subject in the field of computer vision. Its ability to describe scenes via a depth map while reducing the constraints related to the formulation of perspective geometry tends to favor its use. However, despite the constant improvement of algorithms, most methods exploit only colorimetric information. Consequently, robustness to events to which the modality is not sensitive to, like specularity or transparency, is neglected. In response to this phenomenon, we propose using polarimetry as an input for a self-supervised monodepth network. Therefore, we propose exploiting polarization cues to encourage accurate reconstruction of scenes. Furthermore, we include a term of polarimetric regularization to state-of-the-art method to take specific advantage of the data. Our method is evaluated both qualitatively and quantitatively demonstrating that the contribution of this new information as well as an enhanced loss function improves depth estimation results, especially for specular areas.

We present a new method for image-only camera relocalisation composed of a fast image indexing retrieval step followed by pose refinement based on ICP (Iterative Closest Point). The first step aims to find an initial pose for the query by evaluating images similarity with low dimensional global deep descriptors. Subsequently, we predict with a fully convolutional deep encoder-decoder neural network a dense depth map from the image query. We use this depth map to create a local point cloud and refine the initial query pose using an ICP algorithm.We demonstrate the effectiveness of our new approach on various indoor scenes. Compared to learned pose regression methods, our proposal can be used on multiple scenes without the need of a specific weights-setup for each scene, while showing equivalent results.

We propose a new approach for outdoor large scale image based localization that can deal with challenging scenarios like cross-season, cross-weather, day/night and long-term localization. The key component of our method is a new learned global image descriptor, that can effectively benefit from scene geometry information during training. At test time, our system is capable of inferring the depth map related to the query image and use it to increase localization accuracy. We are able to increase recall@1 performances by 2.15% on cross-weather and long-term localization scenario and by 4.24% points on a challenging winter/summer localization sequence versus state-of-the-art methods. Our method can also use weakly annotated data to localize night images across a reference dataset of daytime images.

This article reviews the current state of automatic classification methodologies to identify Diabetic Macular Edema (DME) versus normal subjects based on Spectral Domain OCT (SD-OCT) data. Addressing this classification problem has valuable interest since early detection and treatment of DME play a major role to prevent eye adverse effects such as blindness. The main contribution of this article is to cover the lack of a public dataset and benchmark suited for classifying DME and normal SD-OCT volumes, providing our own implementation of the most relevant methodologies in the literature. Subsequently, 6 different methods were implemented and evaluated using this common benchmark and dataset to produce reliable comparison.

Visual saliency is an attention mechanism which helps to focus on regions of interest instead of processing the whole image or video data. Detecting salient objects in still images has been widely addressed in literature with several formulations and methods. However, visual saliency detection in videos has attracted little attention, although motion information is an important aspect of visual perception. A common approach for obtaining a spatio-temporal saliency map is to combine a static saliency map and a dynamic saliency map. In this paper, we extend a recent saliency detection approach based on principal component analysis (PCA) which have shwon good results when applied to static images. In particular, we explore different strategies to include temporal information into the PCA-based approach. The proposed models have been evaluated on a publicly available dataset which contain several videos of dynamic scenes with complex background, and the results show that processing the spatio-tempral data with multilinear PCA achieves competitive results against state-of-the-art methods.

Timber defects detection is one of the important topics in machine vision applications, since the number and severity of defects determine the quality of the wood and consequently its price. In this paper we propose a method to detect wood defects such as cracks and knots. Firstly we create a dictionary based on the bag-of-words approach in a training step. The dictionary is obtained either using LBP and SURF features alone or with a combination of both features. In the second step an image processing pipeline which associates contrast enhancement, entropy maximization and image filtering is used to detect the potential defect regions and we proposed to use SVM classifier to detect knots and cracks. The proposed algorithm is evaluated on two different datasets which have knots and cracks as groundtruth. The experimental results show that our method achieves a precision of 0.92 and 0.91, and a recall of 0.94 and 0.96 for the Epicea and Pine datasets respectively with multiple features based dictionary.

Visual saliency detection is an important step in many computer vision applications, since it reduces further processing steps to regions of interest. Saliency detection in still images is a well-studied topic. However, videos scenes contain more information than static images, and this additional temporal information is an important aspect of human perception. Therefore, it is necessary to include motion information in order to obtain spatio-temporal saliency map for a dynamic scene. In this paper, we introduce a new spatio-temporal saliency detection method for dynamic scenes based on dynamic textures computed with local binary patterns. In particular, we extract local binary patterns descriptors in two orthogonal planes (LBP-TOP) to describe temporal information, and color features are used to represent spatial information. The obtained three maps are finally fused into a spatio-temporal saliency map. The algorithm is evaluated on a dataset with complex dynamic scenes and the results show that our proposed method outperforms state-of-art methods.

We present an adaptive weight based superpixel segmentation method for the goal of creating mesh representation that respects the 3D scene structure. We propose a new fusion framework which employs both dense optical flow and color images to compute the probability of boundaries. The main contribution of this work is that we introduce a new color and optical flow pixel-wise weighting model that takes into account the non-linear error distribution of the depth estimation from optical flow. Experiments show that our method is better than the other state-of-art methods in terms of smaller error in the final produced mesh.

Visual saliency is an important research topic in computer vision applications, which helps to focus on regions of interest instead of processing the whole image. Detecting visual saliency in still images has been widely addressed in literature. However, visual saliency detection in videos is more complicated due to additional temporal information. A spatio-temporal saliency map is usually obtained by the fusion of a static saliency map and a dynamic saliency map. The way both maps are fused plays a critical role in the accuracy of the spatio-temporal saliency map. In this paper, we evaluate the performances of different fusion techniques on a large and diverse dataset and the results show that a fusion method must be selected depending on the characteristics, in terms of color and motion contrasts, of a sequence. Overall, fusion techniques which take the best of each saliency map (static and dynamic) in the final spatio-temporal map achieve best results.

A hybrid framework of probabilistic atlas and statistical shape and appearance model (SSAM) is proposed to achieve 3D prostate segmentation. An initial 3D segmentation of the prostate is obtained by registering the probabilistic atlas to the test dataset with deformable Demons registration. The initial results obtained are used to initialize multiple SSAMs corresponding to the apex, central and base regions of the prostate gland to incorporate local variabilities. Multiple mean parametric models of shape and appearance are derived from principal component analysis of prior shape and intensity information of the prostate from the training data. The parameters are then modified with the prior knowledge of the optimization space to achieve 2D segmentation. The 2D labels are registered to the 3D labels generated using probabilistic atlas to constrain the pose variation and generate valid 3D shapes. The proposed method achieves a mean Dice similarity coefficient value of 0.89±0.11 and mean Hausdorff distance of 3.05±2.25 mm when validated with 15 prostate volumes of a public dataset in a leave-one-out validation framework.

This paper presents a novel method to identify the 2D axial Magnetic Resonance (MR) slice from a pre-acquired MR prostate volume that closely corresponds to the 2D axial Transrectal Ultrasound (TRUS) slice obtained during prostate biopsy. The shape-context representations of the segmented prostate contours in both the imaging modalities are used to establish point correspondences using Bhattacharyya distance. Thereafter, Chi-square distance is used to find the prostate shape similarities between the MR slices and the TRUS slice. Normalized mutual information and correlation coefficient between the TRUS and MR slices are computed to find the information theoretic similarities between the TRUS-MR slices. The maximum of the weighted likelihood function of the afore-mentioned statistical similarity measures finally yields the MR slice that closely resembles the TRUS slice acquired during the biopsy procedure. The method is evaluated for 20 patient datasets and close matches with the ground truth are obtained for 16 cases.

This work presents a robust technique for tracking a set of detected points on a human face. Facial features can be manually selected or automatically detected. We present a simple and efficient method for detecting facial features such as eyes and nose in a color face image. We then introduce a tracking method which, by employing geometric constraints based on knowledge about the configuration of facial features, avoid the loss of points caused by error accumulation and tracking drift. Experiments with different sequences and comparison with other tracking algorithms, show that the proposed method gives better results with a comparable processing time.