State Space Models (SSMs) have recently emerged as an alternative to Vision Transformers (ViTs) due to their unique ability of modeling global relationships with linear complexity. SSMs are specifically designed to capture spatially proximate relationships of image patches. However, they fail to identify relationships between conceptually related yet not adjacent patches. This limitation arises from the non-causal nature of image data, which lacks inherent directional relationships. Additionally, current vision-based SSMs are highly sensitive to transformations such as rotation. Their predefined scanning directions depend on the original image orientation, which can cause the model to produce inconsistent patch-processing sequences after rotation. To address these limitations, we introduce Spectral VMamba, a novel approach that effectively captures the global structure within an image by leveraging spectral information derived from the graph Laplacian of image patches. Through spectral decomposition, our approach encodes patch relationships independently of image orientation, achieving patch traversal rotation invariance with our Rotational Feature Normalizer (RFN) module. Our experiments on classification tasks show that Spectral VMamba outperforms the leading SSM models in vision, such as VMamba, while maintaining invariance to rotations and a providing a similar runtime efficiency. The implementation is available at: https://github.com/Sahardastani/spectral_vmamba.git.

State Space Models (SSMs) have emerged as efficient alternatives to Vision Transformers (ViTs), with VMamba standing out as a pioneering architecture designed for vision tasks. However, their generalization performance degrades significantly under distribution shifts. To address this limitation, we propose TRUST (Test-Time Refinement using Uncertainty-Guided SSM Traverses), a novel test-time adaptation (TTA) method that leverages diverse traversal permutations to generate multiple causal perspectives of the input image. Model predictions serve as pseudo-labels to guide updates of the Mamba-specific parameters, and the adapted weights are averaged to integrate the learned information across traversal scans. Altogether, TRUST is the first approach that explicitly leverages the unique architectural properties of SSMs for adaptation. Experiments on seven benchmarks show that TRUST consistently improves robustness and outperforms existing TTA methods.

In recent years there has been a resurgence of interest in our community in the shape analysis of 3D objects repre-sented by surface meshes, their voxelized interiors, or surface point clouds. In part, this interest has been stimulated by the increased availability of RGBD cameras, and by applications of computer vision to autonomous driving, medical imaging, and robotics. In these settings, spectral co-ordinates have shown promise for shape representation due to their ability to incorporate both local and global shape properties in a manner that is qualitatively invariant to iso-metric transformations. Yet, surprisingly, such coordinates have thus far typically considered only local surface positional or derivative information. In the present article, we propose to equip spectral coordinates with medial (object width) information, so as to enrich them. The key idea is to couple surface points that share a medial ball, via the weights of the adjacency matrix. We develop a spectral feature using this idea, and the algorithms to compute it. The incorporation of object width and medial coupling has direct benefits, as illustrated by our experiments on object classification, object part segmentation, and surface point correspondence.

Brain surface analysis is essential to neuroscience, however, the complex geometry of the brain cortex hinders computational methods for this task. The difficulty arises from a discrepancy between 3D imaging data, which is represented in Euclidean space, and the non-Euclidean geometry of the highly-convoluted brain surface. Recent advances in machine learning have enabled the use of neural networks for non-Euclidean spaces. These facilitate the learning of surface data, yet pooling strategies often remain constrained to a single fixed-graph. This paper proposes a new learnable graph pooling method for processing multiple surface-valued data to output subject-based information. The proposed method innovates by learning an intrinsic aggregation of graph nodes based on graph spectral embedding. We illustrate the advantages of our approach with in-depth experiments on two large-scale benchmark datasets. The ablation study in the paper illustrates the impact of various factors affecting our learnable pooling method. The flexibility of the pooling strategy is evaluated on four different prediction tasks, namely, subject-sex classification, regression of cortical region sizes, classification of Alzheimer's disease stages, and brain age regression. Our experiments demonstrate the superiority of our learnable pooling approach compared to other pooling techniques for graph convolutional networks, with results improving the state-of-the-art in brain surface analysis.

Learning similarity is a key aspect in medical image analysis, particularly in recommendation systems or in uncovering the interpretation of anatomical data in images. Most existing methods learn such similarities in the embedding space over image sets using a single metric learner. Images, however, have a variety of object attributes such as color, shape, or artifacts. Encoding such attributes using a single metric learner is inadequate and may fail to generalize. Instead, multiple learners could focus on separate aspects of these attributes in subspaces of an overarching embedding. This, however, implies the number of learners to be found empirically for each new dataset. This work, Dynamic Subspace Learners, proposes to dynamically exploit multiple learners by removing the need of knowing apriori the number of learners and aggregating new subspace learners during training. Furthermore, the visual interpretability of such subspace learning is enforced by integrating an attention module into our method. This integrated attention mechanism provides a visual insight of discriminative image features that contribute to the clustering of image sets and a visual explanation of the embedding features. The benefits of our attention-based dynamic subspace learners are evaluated in the application of image clustering, image retrieval, and weakly supervised segmentation. Our method achieves competitive results with the performances of multiple learners baselines and significantly outperforms the classification network in terms of clustering and retrieval scores on three different public benchmark datasets. Moreover, our method also provides an attention map generated directly during inference to illustrate the visual interpretability of the embedding features. These attention maps offer a proxy-labels, which improves the segmentation accuracy up to 15% in Dice scores when compared to state-of-the-art interpretation techniques.

Domain Adaption tasks have recently attracted substantial attention in computer vision as they improve the transferability of deep network models from a source to a target domain with different characteristics. A large body of state-of-the-art domain-adaptation methods was developed for image classification purposes, which may be inadequate for segmentation tasks. We propose to adapt segmentation networks with a constrained formulation, which embeds domain-invariant prior knowledge about the segmentation regions. Such knowledge may take the form of anatomical information, for instance, structure size or shape, which can be known a priori or learned from the source samples via an auxiliary task. Our general formulation imposes inequality constraints on the network predictions of unlabeled or weakly labeled target samples, thereby matching implicitly the prediction statistics of the target and source domains, with permitted uncertainty of prior knowledge. Furthermore, our inequality constraints easily integrate weak annotations of the target data, such as image-level tags. We address the ensuing constrained optimization problem with differentiable penalties, fully suited for conventional stochastic gradient descent approaches. Unlike common two-step adversarial training, our formulation is based on a single segmentation network, which simplifies adaptation, while improving training quality. Comparison with state-of-the-art adaptation methods reveals considerably better performance of our model on two challenging tasks. Particularly, it consistently yields a performance gain of 1-4% Dice across architectures and datasets. Our results also show robustness to imprecision in the prior knowledge. The versatility of our novel approach can be readily used in various segmentation problems, with code available publicly.

Recently, dense connections have attracted substantial attention in computer vision because they facilitate gradient flow and implicit deep supervision during training. Particularly, DenseNet that connects each layer to every other layer in a feed-forward fashion and has shown impressive performances in natural image classification tasks. We propose HyperDenseNet, a 3-D fully convolutional neural network that extends the definition of dense connectivity to multi-modal segmentation problems. Each imaging modality has a path, and dense connections occur not only between the pairs of layers within the same path but also between those across different paths. This contrasts with the existing multi-modal CNN approaches, in which modeling several modalities relies entirely on a single joint layer (or level of abstraction) for fusion, typically either at the input or at the output of the network. Therefore, the proposed network has total freedom to learn more complex combinations between the modalities, within and in-between all the levels of abstraction, which increases significantly the learning representation. We report extensive evaluations over two different and highly competitive multi-modal brain tissue segmentation challenges, iSEG 2017 and MRBrainS 2013, with the former focusing on six month infant data and the latter on adult images. HyperDenseNet yielded significant improvements over many state-of-the-art segmentation networks, ranking at the top on both benchmarks. We further provide a comprehensive experimental analysis of features re-use, which confirms the importance of hyper-dense connections in multi-modal representation learning. Our code is publicly available.

This paper aims at evaluating the effect of spinal surgery on the torso shape appearance of adolescent patients. Current methods that assess the surgical outcome on the trunk shape are limited to its global asymmetry or rely on unreliable manual measurements. We introduce a novel framework to evaluate pre- to postoperative local asymmetry changes using a spectral representation of the torso shape, more specifically, the Laplacian spectrum (eigenvalues and eigenvectors) of a graph. We conduct a statistical analysis on the eigenvalues to efficiently select the spectral space and determine the significant components between preop and postop groups. On the selected eigenvectors, we propose a local analysis based on the concept of Euler characteristic to detect their local maxima and minima, which are then used to compute local left-right (L-R) asymmetries of torso shape. On 49 patients with a thoracic spinal deformity, the method captures significant pre- to postoperative changes of asymmetry at the waist, shoulder blades, shoulders, and breasts. We have evaluated average correction rates for L-R asymmetry of the waist height (67%), shoulder-blade height (64%) and depth (67%), lateral offset between shoulder and neck (61%), and breast height (52%). Spectral torso shape analysis provides a novel approach to quantify the surgical correction of the scoliotic trunk from local shape asymmetry. The proposed method could help the surgeon to understand the impact of different spinal surgery strategies on the postoperative appearance and choose the one that should provide better patient's satisfaction.

Existing methods for surface matching are limited by the tradeoff between precision and computational efficiency. Here, we present an improved algorithm for dense vertex-to-vertex correspondence that uses direct matching of features defined on a surface and improves it by using spectral correspondence as a regularization. This algorithm has the speed of both feature matching and spectral matching while exhibiting greatly improved precision (distance errors of 1.4 percent). The method, FOCUSR, incorporates implicitly such additional features to calculate the correspondence and relies on the smoothness of the lowest-frequency harmonics of a graph Laplacian to spatially regularize the features. In its simplest form, FOCUSR is an improved spectral correspondence method that nonrigidly deforms spectral embeddings. We provide here a full realization of spectral correspondence where virtually any feature can be used as an additional information using weights on graph edges, but also on graph nodes and as extra embedded coordinates. As an example, the full power of FOCUSR is demonstrated in a real-case scenario with the challenging task of brain surface matching across several individuals. Our results show that combining features and regularizing them in a spectral embedding greatly improves the matching precision (to a submillimeter level) while performing at much greater speed than existing methods.

Cardiac fibers, as well as their local arrangement in laminar sheets, have a complex spatial variation of their orientation that has an important role in mechanical and electrical cardiac functions. In this paper, a statistical atlas of this cardiac fiber architecture is built for the first time using human datasets. This atlas provides an average description of the human cardiac fiber architecture along with its variability within the population. In this study, the population is composed of ten healthy human hearts whose cardiac fiber architecture is imaged ex vivo with DT-MRI acquisitions. The atlas construction is based on a computational framework that minimizes user interactions and combines most recent advances in image analysis: graph cuts for segmentation, symmetric log-domain diffeomorphic demons for registration, and log-Euclidean metric for diffusion tensor processing and statistical analysis. Results show that the helix angle of the average fiber orientation is highly correlated to the transmural depth and ranges from -41° on the epicardium to +66° on the endocardium. Moreover, we find that the fiber orientation dispersion across the population (±13°) is lower than for the laminar sheets (±31°) . This study, based on human hearts, extends previous studies on other mammals with concurring conclusions and provides a description of the cardiac fiber architecture more specific to human and better suited for clinical applications. Indeed, this statistical atlas can help to improve the computational models used for radio-frequency ablation, cardiac resynchronization therapy, surgical ventricular restoration, or diagnosis and followups of heart diseases due to fiber architecture anomalies.

Thin Plate Splines are often used in image registration to model deformations. Its physical analogy involves a thin lying sheet of metal that is deformed and forced to pass through a set of control points. The Thin Plate Spline equation minimizes that thin plate bending energy. Rather than using Euclidean distances between control points for image deformation, we are using geodesic distances for image segmentation. Control points become seed points and force the thin plate to pass through given heights. Intuitively, the thin plate surface in the vicinity of a seed point within a region should have similar heights. The minimally bended thin plate actually gives a "confidence" map telling what the closest seed point is for every surface point. The Thin Plate Spline has a closed-form solution which is fast to compute and global optimal. This method shows comparable results to the Graph Cuts method.

In the short time since publication of Boykov and Jolly's seminal paper [2001], graph cuts have become well established as a leading method in 2D and 3D semi-automated image segmentation. Although this approach is computationally feasible for many tasks, the memory overhead and supralinear time complexity of leading algorithms results in an excessive computational burden for high-resolution data. In this paper, we introduce a multilevel banded heuristic for computation of graph cuts that is motivated by the well-known narrow band algorithm in level set computation. We perform a number of numerical experiments to show that this heuristic drastically reduces both the running time and the memory consumption of graph cuts while producing nearly the same segmentation result as the conventional graph cuts. Additionally, we are able to characterize the type of segmentation target for which our multilevel banded heuristic yields different results from the conventional graph cuts. The proposed method has been applied to both 2D and 3D images with promising results.