Camera localization is the process of automatically determining the position and orientation of a camera with respect to its 3D environment based on the images it captures. Camera localization methods, including classical structure-based techniques and modern deep neural networks, often encounter limitations in visually complex environments when relying only on RGB images. Some of these limitations can be overcome by processing RGB-D images, provided that the available color and depth cues are properly integrated. Building upon CoAtNet, a successful hybrid Transformer neural model, this paper introduces CoAtXNet: a dual-stream architecture for jointly processing RGB color and depth channels. CoAtXNet interrelates two CoAtNet networks working in tandem through a straightforward cross-attention mechanism that intertwines the final Transformer stages. This integration of independent data streams yields enhanced feature representations. Experiments on the well-known 7-Scenes and 12-Scenes RGB-D camera localization datasets show that CoAtXNet achieves highly competitive performance, improving average translation and orientation accuracy over several recent localization methods. In addition to quantitative comparisons, we analyze the behavior of CoAtXNet under progressively degraded RGB appearance. The results show that the proposed model is less sensitive to appearance degradation than simpler RGB-D fusion baselines, suggesting that the cross-attention mechanism allows the model to place greater emphasis on geometric structures when RGB reliability decreases. An implementation of the proposed method is publicly available.

Camera localization refers to the process of automatically determining the position and orientation of a camera within its 3D environment from the images it captures. Traditional camera localization methods often rely on Convolutional Neural Networks, which are effective at extracting local visual features but struggle to capture long-range dependencies critical for accurate localization. In contrast, Transformer-based approaches model global contextual relationships appropriately, although they often lack precision in fine-grained spatial representations. To bridge this gap, we introduce CoHAtNet, a novel Convolutional Hybrid-Attention Network that tightly integrates convolutional and self-attention mechanisms. Unlike previous hybrid models that stack convolutional and attention layers separately, CoHAtNet embeds local features extracted via Mobile Inverted Bottleneck Convolution blocks directly into the Value component of the self-attention mechanism of Transformers. This yields a hybrid self-attention block capable of dynamically capturing both local spatial detail and global semantic context within a single attention layer. Additionally, CoHAtNet enables modality-level fusion by processing RGB and depth data jointly in a unified pipeline, allowing the model to leverage complementary appearance and geometric cues throughout. Extensive evaluations have been conducted on two widely-used camera localization datasets: 7-Scenes (RGB-D) and Cambridge Landmarks (RGB). Experimental results show that CoHAtNet achieves state-of-the-art performance in both translation and orientation accuracy. These results highlight the effectiveness of our hybrid design in challenging indoor and outdoor environments. This makes CoHAtNet a strong candidate for end-to-end camera localization tasks.

Abstract Analyzing large volumes of high-dimensional data poses significant challenges. Dimensionality reduction aims to reveal the most prominent properties of data by embedding them into a low-dimensional representation. Spectral dimensionality reduction methods using kernel matrices have been proven to yield optimal results. Online versions of those methods are desirable to incrementally project new data without recomputing the whole embedding from the complete dataset. In addition, integrating different spectral methods may have a synergistic effect. This paper presents an online dimensionality reduction method based on deep neural networks that integrates embeddings optimized by statistical approximation of neighborhoods and induced by different spectral methods through stacking ensemble learning. In particular, the proposed method first applies a self-supervised stage in order to train a set of deep encoders based on the embeddings induced by different spectral methods applied to a given input dataset. Those basis encoders are optimized and then integrated through a metamodel constituted by a fully connected network. A supervised and an unsupervised approach have been designed depending on whether the final aim is to enforce topological preservation or cluster induction. The proposed method has been experimentally validated on well-known image datasets and compared to some of the most relevant dimensionality reduction techniques by using widely-used quality measures.

Estimating a depth map and, at the same time, predicting the 3D pose of an object from a single 2D color image is a very challenging task. Depth estimation is typically performed through stereo vision by following several time-consuming stages, such as epipolar geometry, rectification and matching. Alternatively, when stereo vision is not useful or applicable, depth relations can be inferred from a single image as studied in this paper. More precisely, deep learning is applied in order to solve the problem of estimating a depth map from a single image. Then, that map is used for predicting the 3D pose of the main object depicted in the image. The proposed model consists of two successive neural networks. The first network is based on a Generative Adversarial Neural network (GAN). It estimates a dense depth map from the given color image. A Convolutional Neural Network (CNN) is then used to predict the 3D pose from the generated depth map through regression. The main difficulty to jointly estimate depth maps and 3D poses using deep networks is the lack of training data with both depth and viewpoint annotations. This contribution assumes a cross-domain training procedure with 3D CAD models corresponding to objects appearing in real images in order to render depth images from different viewpoints. These rendered images are then used to guide the GAN network to learn the mapping from the image domain to the depth domain. By exploiting the dataset as a source of training data, the proposed model outperforms state-of-the-art models on the PASCAL 3D+ dataset. The code of the proposed model is publicly available at https://github.com/SaddamAbdulrhman/Depth-and-Viewpoint-Estimation/tree/master.

Most of the variational optical flow methods are based on the well-known brightness constancy assumption or high-order constancy assumptions to implement the data term in the optimization energy function. Unfortunately, any variation in the lighting within the scene violates the brightness constancy constraint; in turn, the gradient constancy assumption does not work properly with large illumination changes. This paper proposes an illumination-robust constancy based on a robust texture descriptor rather than the brightness constancy. Thus, the similarity function used as a data term was obtained from extracting texture features through the local directional pattern descriptor for two consecutive frames within the duality total variational optical flow algorithm. In addition, a weighted nonlocal term that depends on both the color similarity and the occlusion state of pixels is integrated during the optimization process to increase the accuracy of the resulting flow field. The experimental results show a qualitative comparison with the proposed approach and yield state-of-the-art results on the KITTI, Midleburry, and MPI-sintel data sets.

The limited depth-of-field of some cameras prevents them from capturing perfectly focused images when the imaged scene covers a large distance range. In order to compensate for this problem, image fusion has been exploited for combining images captured with different camera settings, thus yielding a higher quality all-in-focus image. Since most current approaches for image fusion rely on maximizing the spatial frequency of the composed image, the fusion process is sensitive to noise. In this paper, a new algorithm for computing the all-in-focus image from a sequence of images captured with a low depth-of-field camera is presented. The proposed approach adaptively fuses the different frames of the focus sequence in order to reduce noise while preserving image features. The algorithm consists of three stages: 1) focus measure; 2) selectivity measure; 3) and image fusion. An extensive set of experimental tests has been carried out in order to compare the proposed algorithm with state-of-the-art all-in-focus methods using both synthetic and real sequences. The obtained results show the advantages of the proposed scheme even for high levels of noise.

Variational optical flow techniques allow the estimation of flow fields from spatio-temporal derivatives. They are based on minimizing a functional that contains a data term and a regularization term. Recently, numerous approaches have been presented for improving the accuracy of the estimated flow fields. Among them, tensor voting has been shown to be particularly effective in the preservation of flow discontinuities. This paper presents an adaptation of the data term by using anisotropic stick tensor voting in order to gain robustness against noise and outliers with significantly lower computational cost than (full) tensor voting. In addition, an anisotropic complementary smoothness term depending on directional information estimated through stick tensor voting is utilized in order to preserve discontinuity capabilities of the estimated flow fields. Finally, a weighted non-local term that depends on both the estimated directional information and the occlusion state of pixels is integrated during the optimization process in order to denoise the final flow field. The proposed approach yields state-of-the-art results on the Middlebury benchmark.

Visual attention is the cognitive process of selectively focusing on certain areas of a visual scene while ignoring the others. It is a desirable capability for intelligent video surveillance systems, as it allows them to control the aim of mobile cameras or to selectively process the most relevant parts of the captured images. This paper proposes an adaptation of a well-known biologically-inspired visual attention model in order to increase its computational efficiency without sacrificing its accuracy, and shows that the latter can be further improved through a supervised training stage that fine-tunes the model to the particular application scope in which the system is being utilized. Experimental results and comparisons with previous visual attention techniques are shown and discussed.

This paper proposes two alternative formulations to reduce the high computational complexity of tensor voting, a robust perceptual grouping technique used to extract salient information from noisy data. The first scheme consists of numerical approximations of the votes, which have been derived from an in-depth analysis of the plate and ball voting processes. The second scheme simplifies the formulation while keeping the same perceptual meaning of the original tensor voting: The stick tensor voting and the stick component of the plate tensor voting must reinforce surfaceness, the plate components of both the plate and ball tensor voting must boost curveness, whereas junctionness must be strengthened by the ball component of the ball tensor voting. Two new parameters have been proposed for the second formulation in order to control the potentially conflictive influence of the stick component of the plate vote and the ball component of the ball vote. Results show that the proposed formulations can be used in applications where efficiency is an issue since they have a complexity of order O(1). Moreover, the second proposed formulation has been shown to be more appropriate than the original tensor voting for estimating saliencies by appropriately setting the two new parameters.

An inherent problem of unsupervised texture segmentation is the absence of previous knowledge regarding the texture patterns present in the images to be segmented. A new efficient methodology for unsupervised image segmentation based on texture is proposed. It takes advantage of a supervised pixel-based texture classifier trained with feature vectors associated with a set of texture patterns initially extracted through a clustering algorithm. Therefore, the final segmentation is achieved by classifying each image pixel into one of the patterns obtained after the previous clustering process. Multi-sized evaluation windows following a top-down approach are applied during pixel classification in order to improve accuracy. The proposed technique has been experimentally validated on MeasTex, VisTex and Brodatz compositions, as well as on complex ground and aerial outdoor images. Comparisons with state-of the-art unsupervised texture segmenters are also provided.

This paper presents a new method for robust color image segmentation based on tensor voting, a robust perceptual grouping technique used to extract salient information from noisy data. First, an adaptation of tensor voting to both image denoising and robust edge detection is applied. Second, pixels in the filtered image are classified into likely-homogeneous and likely-inhomogeneous by means of the edginess maps generated in the first step. Third, the likely-homosgeneous pixels are segmented through an efficient graph-based segmenter. Finally, a modified version of the same graph-based segmenter is applied to the likely-inhomogeneous pixels in order to obtain the final segmentation. Experiments show that the proposed algorithm has a better performance than the state-of-the-art.

Images taken with different focus settings are used in shape-from-focus to reconstruct the depth map of a scene. A problem when acquiring images with different focus settings is the shift of image features due to changes in magnification. This paper shows that those changes affect the shape-from-focus performance and that the final reconstruction can be improved by compensating for that shift. The proposed scheme takes into account the effects due to magnification changes between near and far focused images and it is able to determine the depth of the scene points with higher accuracy than traditional techniques. Experimental results of the application of the proposed method are shown.

This paper presents a new efficient technique for supervised pixel-based texture classification. The proposed scheme first performs a selection process that automatically determines a subset of prototypes that characterize each texture class based on the outcome of a multichannel Gabor wavelet filter bank. Then, every image pixel is classified into one of the given texture classes by using a K-NN classifier fed with the prototypes determined previously. The proposed technique is compared to previous texture classifiers by using both Brodatz and real outdoor textured images.

This paper presents a new method for color edge detection based on the tensor voting framework, a robust perceptual grouping technique used to extract salient information from noisy data. The tensor voting framework is adapted to encode color information via tensors in order to propagate them into a neighborhood through a voting process specifically designed for color edge detection by taking into account perceptual color differences, region uniformity and edginess according to a set of intuitive perceptual criteria. Perceptual color differences are estimated by means of an optimized version of the CIEDE2000 formula, while uniformity and edginess are estimated by means of saliency maps obtained from the tensor voting process. Experiments show that the proposed algorithm is more robust and has a similar performance in precision when compared with the state-of-the-art.

This paper defines a new methodology for evaluating edge detectors through measurements on edginess maps instead of on binary edge maps as previous methodologies do. These measurements avoid possible bias introduced by the application-dependent process of generating binary edge maps from edginess maps. The features of completeness, discriminability, precision and robustness, which a general-purpose edge detector must comply with, are introduced. The R, DS, P and FAR-measurements in addition to PSNR applied to the edginess maps are defined to assess the performance of edge detection. The R, DS, P and FAR-measurements can be seen as generalizations of previously proposed measurements on binary edge maps. Well-known and state-of-the-art edge detectors have been compared by means of the new proposed metrics. Results show that it is difficult for an edge detector to comply with all the proposed features.

This paper presents a pixel-based texture classifier oriented to the identification of texture models that can be present in an input image, given a set of models known in advance. The proposed methodology is based on the integration of texture features generated by texture methods that belong to different families, which are evaluated over multiple windows of different sizes. This is a novelty with respect to the current texture classifiers, which are based on specific families of texture methods evaluated over single windows of a size defined empirically. Experiments show that this integration strategy produces better results than classical texture classifiers based on specific families of texture methods.

This work presents a new algorithm for determining the trajectory of a mobile robot and, simultaneously, creating a detailed volumetric 3D model of its workspace. The algorithm exclusively utilizes information provided by a single stereo vision system, avoiding thus the use both of more costly laser systems and error-prone odometry. Six-degrees-of-freedom egomotion is directly estimated from images acquired at relatively close positions along the robot's path. Thus, the algorithm can deal with both planar and uneven terrain in a natural way, without requiring extra processing stages or additional orientation sensors. The 3D model is based on an octree that encapsulates clouds of 3D points obtained through stereo vision, which are integrated after each egomotion stage. Every point has three spatial coordinates referred to a single frame, as well as true-color components. The spatial location of those points is continuously improved as new images are acquired and integrated into the model.

This paper presents a new iterative algorithm for automatically generating a hierarchical clustering of the objects contained in a complex 3D scene. The proposed object-oriented representation is shown to be advantageous over octrees, a traditional scene-oriented hierarchical representation, for accelerating extensively-used tasks such as minimum distance computation. Experimental results with large synthetic 3D scenes are presented.

This paper proposes a new coordination algorithm for efficiently exploring an unknown environment with a team of mobile robots. The proposed technique subsequently applies a well-known unsupervised clustering algorithm (k-means) in order to fairly divide the remaining unknown space into as many disjoint regions as available robots. Each robot is primarily responsible for exploring its assigned region and can help other robots on its way through. Unknown space is dynamically repartitioned as new areas are discovered by the team, balancing thus the overall workload among team members and naturally leading to greater dispersion over the environment and thus faster broad coverage than with previous greedy-like approaches, which guide robots based on maximum profit strategies that simply trade off between distance to the closest frontiers and amount of unknown cells likely to be discovered from them.

This paper presents an efficient technique for generating adaptive triangular meshes from range images. The algorithm consists of two stages. First, a user-defined number of points is adaptively sampled from the given range image. Those points are chosen by taking into account the surface shapes represented in the range image in such a way that points tend to group in areas of high curvature and to disperse in low-variation regions. This selection process is done through a noniterative, inherently parallel algorithm in order to gain efficiency. Once the image has been subsampled, the second stage applies a two and one half-dimensional Delaunay triangulation to obtain an initial triangular mesh. To favor the preservation of surface and orientation discontinuities (jump and crease edges) present in the original range image, the aforementioned triangular mesh is iteratively modified by applying an efficient edge flipping technique. Results with real range images show accurate triangular approximations of the given range images with low processing times.

In this paper a system for off-line traffic sign detection is shown. Matlab-image-processing toolbox is used for this purpose. The vision-based traffic sign detection module developed in this work manages 172/spl times/352 color images in RGB (red, green, blue) format. The first step in the algorithm is to obtain the gradient image and its vertical edge projection. In a second step, a color and shape analysis is performed.

This paper presents a new technique for combining multiple texture feature extraction methods in order to classify the pixels of an input image into a set of texture models of interest. The problem of integrating multiple texture methods for classification purposes is cast as a collaborative decision making problem. Each texture method is considered to be an expert that gives an opinion about the membership of every input image pixel to each texture model, along with a conviction about that judgement. A conviction measure based on the Kullback J-divergence between texture models is proposed, along with an arbitration mechanism that combines those convictions by taking into account conflicts that may occur when different experts disagree with a similar strength. The proposed technique is compared to previous pixel-based texture classifiers by using real textured images.

A supervised pixel-based classifier for identifying the presence of a given set of texture patterns of interest in a complex textured image is described. The proposed technique integrates the outcome of multiple texture feature extraction methods belonging to different families. In this way, it yields lower classification rates than previous texture classifiers based on specific families of texture methods. Experimental results with real outdoor images are presented.

This paper describes an algorithm to implement image filtering and enhancement operations by processing adaptive triangular meshes that represent gray-level images. Experimental results show that these operations are significantly more efficient when they are performed upon triangular meshes than by sequentially processing all the pixels contained in the given images.

This paper presents a technique for lossy compression of dense range images. Two separate compression schemes are applied. The first scheme (geometric compression) reduces redundant geometric information by generating an adaptive 3D triangular mesh that approximates the shapes present in the original range image. Geometric compression is used for obtaining an efficient representation of the range image that allows further processing. The second compression scheme (topological compression) encodes the connectivity information contained in the triangular mesh. Topological compression is used for generating a compact representation suitable to be stored or transmitted. Both compression schemes avoid costly iterative optimization algorithms. Results with real range images are presented.

Explores the utilization of adaptive triangular meshes representing gray-level images to accelerate basic image processing operations. In particular two algorithms for thresholding and labeling gray-level images represented with adaptive triangular meshes are described and evaluated. Experimental results with real gray-level images show that it is possible to perform faster by working in the 3D geometric domain than by sequentially processing all the pixels contained in the original images.

Presents a technique for approximating range images by means of adaptive triangular meshes with a bounded approximation error and without applying optimization. This approach consists of three stages. In the first stage, every pixel of the given range image is mapped to a 3D point defined in a reference frame associated with the range sensor. Then, those 3D points are mapped to a 3D curvature space. In the second stage, the points contained in this curvature space are triangulated through a 3D Delaunay algorithm, giving rise to a tetrahedronization of them. In the last stage, an iterative process starts digging the external surface of the previous tetrahedronization, removing those triangles that do not fulfill the given approximation error. In this way, successive fronts of triangular meshes are obtained in both range image space and curvature space. This iterative process is applied until a triangular mesh in the range image space fulfilling the given approximation error is obtained. Experimental results are presented.

This paper presents a new method for the incremental integration of overlapped range images. It is assumed that frame transformations between all pairs of views can be reliably computed. This method progressively merges each new sensed range image with the current reconstructed model. The proposed method consists of three stages. In the first stage the overlapped regions are identified. Then, from the overlapped regions, the points that define the integration boundaries are projected over a reference plane and are triangulated over that 2D space by means of a constrained Delaunay algorithm. Finally, the obtained triangulation is back projected to the 3D range image space. These new triangular meshes represent the sewing between the meshes to be integrated. In this way a new single triangular mesh which will be used to merge with futures range images is generated. Experimental results are presented.

This paper presents an iterative algorithm for approximating gray-scale images with adaptive triangular meshes ensuring a given tolerance. At each iteration, the algorithm applies a non-iterative adaptive meshing technique. In this way, this technique converges faster than traditional mesh refinement algorithms. The performance of the proposed technique is studied in terms of compression ratio and speed, comparing it with an optimization-based mesh refinement algorithm.

Describes an efficient technique for computing a hierarchical representation of the objects contained in a complex 3D scene. First, an adjacency graph keeping the costs of grouping the different pairs of objects in the scene is built. Then the minimum spanning tree (MST) of that graph is determined. A binary clustering tree (BCT) is obtained from the MST. Finally, a merging stage joins the adjacent nodes in the BCT which have similar costs. The final result is an n-ary tree which defines an intuitive clustering of the objects of the scene at different levels of abstraction. Experimental results with synthetic 3D scenes are presented.

A new technique is presented for determining the positions where a range sensor should be located to acquire the surfaces of a complex scene. The algorithm consists of two stages. The first stage applies a voting scheme that considers occlusion edges. Most of the surfaces of the scene are recovered through views computed in that way. Then, the second stage fills up remaining holes through a scheme based on visibility analysis. By leaving the more expensive visibility computations at the end of the exploration process, efficiency is increased. 1 Introduction The automatic reconstruction of 3D objects (scenes in general) through range images is gaining popularity in computer vision and robotics owing to the variety of applications that can benefit from it, including world modeling [2], reverse engineering and object segmentation [3] or recognition. Two basic tasks must be addressed in order to solve that problem. First, an exploration process is necessary for determining the pos...

Presents a technique for determining a small set of views that allow the observation and acquisition of the surfaces of the objects present in a target scene through a range sensor that moves over a sphere containing that scene. No a priori knowledge about the shape of those objects is assumed. Instead of applying costly visibility analysis techniques from the beginning as in most previous approaches, a two-stage algorithm is proposed. The first stage is responsible for getting the majority of object surfaces through a voting scheme based on occlusion edges. Then, a second stage applies visibility analysis to fill holes left by the first stage due to self-occlusions.

The paper presents an efficient algorithm for generating adaptive triangular meshes from dense range images. The proposed technique consists of two stages. First, a quadrilateral mesh is generated from the given range image. The points of this mesh adapt to the surface shapes represented in the range image by grouping in areas of high curvature and dispersing in low-variation regions. The second stage splits each quadrilateral cell obtained before into two triangles. Between the two possible flips, it is chosen the one whose diagonal's direction is closest to the orientation of the discontinuities present in that cell. Both stages avoid costly iterative optimization techniques. Results with real range images are presented. They show low CPU times and accurate triangular approximations of the given images.

This paper proposes a fast technique for generating adaptive quadrilateral meshes from range images with no optimization. The obtained meshes adapt to the features of the input images by concentrating points in areas of high curvature and by dispersing them in low variation regions. This leads to more accurate approximations of the given range images than when uniform sampling with the same number of points is applied. Experimental results with real range images representing both free-form and polyhedral and cylindrical objects are presented.

An efficient algorithm for extracting planar and curved surface patches that express distinctive parts of the objects contained in a given range image is presented. The proposed technique does not directly segment the range image but a triangular approximation of it obtained through a fast adaptive randomized sampling algorithm. This intermediate representation allows us to avoid the processing of all the individual points of the range image in the segmentation phase.

This paper describes the use of a previously developed surface model for representing 3D geometric information that is subject to uncertainty. The proposed model is based on a geometric data fusion technique that allows the efficient approximation of arbitrary triangular meshes in space with smooth surfaces. This paper shows how the shape modifiers defined in the model can be used to represent both global and local uncertainty associated with the input data. The possibility of handling scattered 3D points, taking into account their corresponding uncertainties, makes this technique a suitable tool for modeling free-form surfaces acquired through sensing, as well as for integrating 3D data obtained from multiple sensors with different associated confidences.

This paper describes and evaluates an efficient technique that allows the fast generation of 3D triangular meshes from range images avoiding optimization procedures. Such a tool is advantageous in order to integrate range imagery into world models based on scattered representations. Furthermore, this technique can also be used as a fast preprocessing stage of registration, segmentation or recognition algorithms, owing to its abstraction capabilities that tend to eliminate redundant information. The proposed method has two stages: 1) the vertices of the mesh are computed through adaptive randomized sampling of the range image based on curvature estimations; and 2) the mesh is generated by triangulating the sampled vertices through an efficient 2 1/2 D Delaunay algorithm. The sampling process concentrates points in areas of large curvature and tends to preserve surface and orientation discontinuities. Multiresolution representations are supported in a natural way. The proposed technique is evaluated with several real range images that include both free-form and polyhedral surfaces.