Self-supervised learning (SSL) methods have achieved remarkable success in learning image representations allowing invariances in them — but therefore discarding transformation information that some computer vision tasks actually require. While recent approaches attempt to address this limitation by learning equivariant features using linear operators in feature space, they impose restrictive assumptions that constrain flexibility and generalization. We introduce a weaker definition for the transformation relation between image and feature space denoted as equivariance-coherence. We propose a novel SSL auxillary task that learns equivariance-coherent representations through intermediate transformation reconstruction, which can be integrated with existing joint embedding SSL methods. Our key idea is to reconstruct images at intermediate points along transformation paths, e.g. when training on 30° rotations, we reconstruct the 10° and 20° rotation states. Reconstructing intermediate states requires the transformation information used in augmentations, rather than suppressing it, and therefore fosters features containing the augmented transformation information. Our method decomposes feature vectors into invariant and equivariant parts, training them with standard SSL losses and reconstruction losses, respectively. We demonstrate substantial improvements on synthetic equivariance benchmarks while maintaining competitive performance on downstream tasks requiring invariant representations. The approach seamlessly integrates with existing SSL methods (iBOT, DINOv2) and consistently enhances performance across diverse tasks, including segmentation, detection, depth estimation, and video dense prediction. Our framework provides a practical way for augmenting SSL methods with equivariant capabilities while preserving invariant performance.

Parameter-Efficient Fine-Tuning (PEFT) methods have transformed the approach to fine-tuning large models for downstream tasks by enabling the adjustment of significantly fewer parameters than those in the original model matrices. In this work, we study the "very low rank regime", where we fine-tune the lowest amount of parameters per linear layer for each considered PEFT method. We propose ${1\!{\text {I}}}{\text{LoRa}}$ (Summation Low-Rank Adaptation), a compute, parameter and memory efficient fine-tuning method which uses the feature sum as fixed compression and a single trainable vector as decompression. Differently from state-of-the-art PEFT methods like LoRA, VeRA, and the recent MoRA, ${1\!{\text {I}}}{\text{LoRa}}$ uses fewer parameters per layer, reducing the memory footprint and the computational cost. We extensively evaluate our method against state-of-the-art PEFT methods on multiple fine-tuning tasks, and show that our method not only outperforms them, but is also more parameter, memory and computationally efficient. Moreover, thanks to its memory efficiency, ${1\!{\text {I}}}{\text{LoRa}}$ allows to fine-tune more evenly across layers, instead of focusing on specific ones (e.g. attention layers), improving performance further.

Cell tracking is a key computational task in live-cell microscopy, but fully automated analysis of high-throughput imaging requires reliable and, thus, uncertainty-aware data analysis tools, as the amount of data recorded within a single experiment exceeds what humans are able to overlook. We here propose and benchmark various methods to reason about and quantify uncertainty in linear assignment-based cell tracking algorithms. Our methods take inspiration from statistics and machine learning, leveraging two perspectives on the cell tracking problem explored throughout this work: Considering it as a Bayesian inference problem and as a classification problem. Our methods admit a framework-like character in that they equip any frame-to-frame tracking method with uncertainty quantification. We demonstrate this by applying it to various existing tracking algorithms including the recently presented Transformer-based trackers. We demonstrate empirically that our methods yield useful and well-calibrated tracking uncertainties.

The growing integration of machine learning (ML) and artificial intelligence (AI) models into high-stakes domains such as healthcare and scientific research calls for models that are not only accurate but also interpretable. Among the existing explainable methods, counterfactual explanations offer interpretability by identifying minimal changes to inputs that would alter a model’s prediction, thus providing deeper insights. However, current counterfactual generation methods suffer from critical limitations, including gradient vanishing, discontinuous latent spaces, and an overreliance on the alignment between learned and true decision boundaries. To overcome these limitations, we propose LeapFactual, a novel counterfactual explanation algorithm based on conditional flow matching. LeapFactual generates reliable and informative counterfactuals, even when true and learned decision boundaries diverge. LeapFactual is not limited to models with differentiable loss functions. It can even handle human-in-the-loop systems, expanding the scope of counterfactual explanations to domains that require the participation of human annotators, such as citizen science. We provide extensive experiments on benchmark and real-world datasets highlighting that LeapFactual generates accurate and in-distribution counterfactual explanations that offer actionable insights. We observe, for instance, that our reliable counterfactual samples with labels aligning to ground truth can be beneficially used as new training data to enhance the model. The proposed method is diversely applicable and enhances scientific knowledge discovery as well as non-expert interpretability.

Magnetic Resonance Imaging (MRI) is a powerful imaging technique widely used for visualizing structures within the human body and in other fields such as plant sciences. However, there is a demand to develop fast 3D-MRI reconstruction algorithms to show the fine structure of objects from under-sampled acquisition data, i.e., k-space data. This emphasizes the need for efficient solutions that can handle limited input while maintaining high-quality imaging. In contrast to previous methods only using 2D, we propose a 3D MRI reconstruction method that leverages a regularized 3D diffusion model combined with optimization method. By incorporating diffusion-based priors, our method improves image quality, reduces noise, and enhances the overall fidelity of 3D MRI reconstructions. We conduct comprehensive experiments analysis on clinical and plant science MRI datasets. To evaluate the algorithm effectiveness for under-sampled k-space data, we also demonstrate its reconstruction performance with several undersampling patterns, as well as with in- and out-of-distribution pre-trained data. In experiments, we show that our method improves upon tested competitors.

Monocular depth estimation (MDE) is a challenging task in computer vision, often hindered by the cost and scarcity of high-quality labeled datasets. We tackle this challenge using auxiliary datasets from related vision tasks for an alternating training scheme with a shared decoder built on top of a pre-trained vision foundation model, while giving a higher weight to MDE. Through extensive experiments we demonstrate the benefits of incorporating various in-domain auxiliary datasets and tasks to improve MDE quality on average by ~ 11 %. Our experimental analysis shows that auxiliary tasks have different impacts, confirming the importance of task selection, highlighting that quality gains are not achieved by merely adding data. Remarkably, our study reveals that using semantic segmentation datasets as Multi-Label Dense Classification (MLDC) often results in additional quality gains. Lastly, our method significantly improves the data efficiency for the considered MDE datasets, enhancing their quality while reducing their size by at least 80%. This paves the way for using auxiliary data from related tasks to improve MDE quality despite limited availability of high-quality labeled data. Code is available at https://jugit.fz-juelich.de/ias-8/mdeaux.

Abstract In deep learning, using larger training datasets usually leads to more accurate models. However, simply adding more but redundant data may be inefficient, as some training samples may be more informative than others. We propose Focal Sampling, a method that biases SGD (Stochastic Gradient Descent) towards samples that are found to be more important after a few training epochs, by sampling them more often for the rest of the training. In contrast to state-of-the-art, our approach requires less computational overhead to estimate sample importance, as it computes estimates once during training using the prediction probabilities, and does not require restarting training. In the experimental evaluation, we see that our learning technique trains faster than state-of-the-art and can achieve higher test accuracy, especially when datasets are not well balanced or when using multiple data augmentations. Lastly, results suggest that our approach has intrinsic balancing properties and that balancing datasets based on class importance, rather than by number of samples, can achieve higher test accuracy. Code is available at https://jugit.fz-juelich.de/ias-8/sgd_biased .

VRoot is an immersive extended reality reconstruction tool for root system architectures from 3D volumetric scans of soil columns. We have conducted a laboratory user study to assess the performance of new users with our software in comparison to established software. We utilize a plant model to derive a synthetic root architecture, providing a baseline for reconstruction. This demo showcases the processes and techniques contributing to exact and efficient manual root architecture reconstruction in Virtual Reality. The extraction task typically is the sparse graph-structure extraction from a 3D magnetic-resonance imaging (MRI) data set. We visualize the RSA directly within the MRI and offer selection-set-based methods of adapting and augmenting the root architecture. This application is in productive use at our partner institute, where it is used to analyze complex root images.

In deep learning, using larger training datasets usually leads to more accurate models. However, simply adding more but redundant data may be inefficient, as some training samples may be more informative than others. We propose to bias SGD (Stochastic Gradient Descent) towards samples that are found to be more important after a few training epochs, by sampling them more often for the rest of training.In contrast to state-of-the-art, our approach requires less computational overhead to estimate sample importance, as it computes estimates once during training using the prediction probabilities, and does not require that training be restarted.In the experimental evaluation, we see that our learning technique trains faster than state-of-the-art and can achieve higher test accuracy, especially when datasets are not well balanced. Lastly, results suggest that our approach has intrinsic balancing properties. Code is available at https://github.com/AlessioQuercia/sgd_biased.

The retrieval of sun-induced fluorescence (SIF) from hyper-spectral imagery is an ill-posed problem that has been tackled in different ways. We present a novel retrieval method combining semi-supervised deep learning with an existing spectral fitting method. A validation study with in-situ SIF measurements shows high sensitivity of the deep learning method to SIF changes even though systematic shifts deteriorate its absolute prediction accuracy. A detailed analysis of diurnal SIF dynamics and SIF prediction in topographically variable terrain highlights the benefits of this deep learning approach.

In many remote sensing applications the measured radiance needs to be corrected for atmospheric effects to study surface properties such as reflectance, temperature or emission features. The correction often applies radiative transfer to simulate atmospheric propagation, a time-consuming step usually done offline. In principle, an efficient machine learning (ML) model can accelerate the simulation step. This is the goal pursued here in the context of solar-induced fluorescence (SIF) emitted by vegetation around the O2-A band using the spaceborne DESIS and airborne HyPlant spectrometers. We present an ML simulator of at-sensor radiances trained on synthetic spectra and describe its performance in detail. The simulator is fast and accurate, constituting a promising alternative to a full-fledged, lengthy radiative transfer code for SIF retrieval in the O2-A band with DESIS and HyPlant.

In this contribution we introduce an almost lossless affine 2D image transformation method. To this end we extend the theory of the well-known Chirp-z transform to allow for fully affine transformation of general n-dimensional images. In addition we give a practical spatial and spectral zero-padding approach dramatically reducing losses of our transform, where usual transforms introduce blurring artifacts due to sub-optimal interpolation. The proposed method improves the mean squared error by approx. a factor of 1800 compared to the commonly used linear interpolation, and by a factor of 250 to the best competitor. We derive the transform from basic principles with special attention to implementation details and supplement this paper with python code for 2D images. In demonstration experiments we show the superior image quality compared to usual approaches, when using our method. However runtimes are considerably larger than when using toolbox algorithms.

Reliable measurements of Sun Induced Fluorescence (SIF) require a good instrument characterization as well as a complex processing chain. In this paper, we summarize the state of the art SIF retrieval methods and measurements platforms for field phenotyping. Furthermore, we use HyScreen, hyperspectral-imaging system for top of canopy measurements of SIF, as an example of the instrument requirements, data process, and data validation needed to obtain reliable measurements of SIF.

Plant growth is a dynamic process, and the precise course of events during early plant development is of major interest for plant research. In this work, we investigate the growth of rosette plants by processing time-lapse videos of growing plants, where we use Nicotiana tabacum (tobacco) as a model plant. In each frame of the video sequences, potential leaves are detected using a leaf-shape model. These detections are prone to errors due to the complex shape of plants and their changing appearance in the image, depending on leaf movement, leaf growth, and illumination conditions. To cope with this problem, we employ a novel graph-based tracking algorithm which can bridge gaps in the sequence by linking leaf detections across a range of neighboring frames. We use the overlap of fitted leaf models as a pairwise similarity measure, and forbid graph edges that would link leaf detections within a single frame. We tested the method on a set of tobacco-plant growth sequences, and could track the first leaves of the plant, including partially or temporarily occluded ones, along complete sequences, demonstrating the applicability of the method to automatic plant growth analysis. All seedlings displayed approximately the same growth behavior, and a characteristic growth signature was found.

High-angular resolution diffusion imaging (HARDI) is a magnetic resonance technique estimating the direction of self-diffusion of water molecules in biological tissue. HARDI encodes at each pixel (voxel) the orientation distribution function (ODF) of water diffusion molecules, i.e. the probability distribution function of finding a water molecule which moved in a certain direction during the observation time. As a consequence ODF images differ from usual gray scale images with respect to their underlying geometry as well as with respect to their error distribution. We present a Bayesian estimator for ODF images considering these differences. To this end, we derive a likelihood function based on the Rician distribution of the NMR signals and propose prior distributions considering ODFs as Riemanian manifolds. Utilizing properties of spherical harmonics and the square root representation of ODFs allows us to effectively reconstruct and regularize ODF images in one step within this Riemannian framework. Experiments demonstrate the merits of our approach on synthetic as well as on real data.

Relations between deterministic (e.g. variational or PDE based methods) and Bayesian inference have been known for a long time. However, a classification of deterministic approaches into those methods which can be handled within a Bayesian framework and those with no such statistical counterpart is still missing in literature. After providing such taxonomy, we present a Bayesian framework for embedding the former ones into a statistical context allowing to equip them with advantages of probabilistic estimation theory. A stochastic point of view allows us (1) to learn influence functions and derivative filter, (2) adapt diffusion and regularization approaches to changes in the image characteristics (e.g. varying noise levels), and (3) to estimate error bounds on the solution. For the latter ones we present alternative learning schemes also allowing their parameters to be related to the image statistics such that hand tuning becomes dispensable. We demonstrate that a statistical point of view on diffusion and regularization schemes leads to image denoising performances comparable with state of the art Markov random field approaches while being computationally much more effective.

We extend estimation of range flow to handle brightness changes in image data caused by inhomogeneous illumination. Standard range flow computes 3D velocity fields using both range and intensity image sequences. Toward this end, range flow estimation combines a depth change model with a brightness constancy model. However, local brightness is generally not preserved when object surfaces rotate relative to the camera or the light sources, or when surfaces move in inhomogeneous illumination. We describe and investigate different approaches to handle such brightness changes. A straightforward approach is to prefilter the intensity data such that brightness changes are suppressed, for instance, by a highpass or a homomorphic filter. Such prefiltering may, though, reduce the signal-to-noise ratio. An alternative novel approach is to replace the brightness constancy model by 1) a gradient constancy model, or 2) by a combination of gradient and brightness constancy constraints used earlier successfully for optical flow, or 3) by a physics-based brightness change model. In performance tests, the standard version and the novel versions of range flow estimation are investigated using prefiltered or nonprefiltered synthetic data with available ground truth. Furthermore, the influences of additive Gaussian noise and simulated shot noise are investigated. Finally, we compare all range flow estimators on real data.

Diffusion tensor magnetic resonance imaging (DT-MRI) is a non-invasive imaging technique allowing to estimate the molecular self-diffusion tensors of water within surrounding tissue. Due to the low signal-to-noise ratio of magnetic resonance images, reconstructed tensor images usually require some sort of regularization in a post-processing step. Previous approaches are either suboptimal with respect to the reconstructing or regularization step. This paper presents a Bayesian approach for simultaneous reconstructing and regularization of DT-MR images that allows to resolve the disadvantages of previous approaches. To this end, estimation theoretical concepts are generalized to tensor valued images that are considered as Riemannian manifolds. Doing so allows us to derive a maximum a posterior estimator of the tensor image that considers both the statistical characteristics of the Rician noise occurring in MR images as well as the nonlinear structure of tensor valued images. Experiments on synthetic data as well as real DT-MRI data validate the advantage of considering both statistical as well as geometrical characteristics of DT-MRI.

In this paper, we present a new and efficient method to implement robust smoothing of low-level signal features: B-spline channel smoothing. This method consists of three steps: encoding of the signal features into channels, averaging of the channels, and decoding of the channels. We show that linear smoothing of channels is equivalent to robust smoothing of the signal features if we make use of quadratic B-splines to generate the channels. The linear decoding from B-spline channels allows the derivation of a robust error norm, which is very similar to Tukey's biweight error norm. We compare channel smoothing with three other robust smoothing techniques: nonlinear diffusion, bilateral filtering, and mean-shift filtering, both theoretically and on a 2D orientation-data smoothing task. Channel smoothing is found to be superior in four respects: It has a lower computational complexity, it is easy to implement, it chooses the global minimum error instead of the nearest local minimum, and it can also be used on nonlinear spaces, such as orientation space.

Many sensing techniques and image processing applications are characterized by noisy, or corrupted, image data. Anisotropic diffusion is a popular, and theoretically well understood, technique for denoising such images. Diffusion approaches however require the selection of an "edge stopping" function, the definition of which is typically ad hoc. We exploit and extend recent work on the statistics of natural images to define principled edge stopping functions for different types of imagery. We consider a variety of anisotropic diffusion schemes and note that they compute spatial derivatives at fixed scales from which we estimate the appropriate algorithm-specific image statistics. Going beyond traditional work on image statistics, we also model the statistics of the eigenvalues of the local structure tensor. Novel edge-stopping functions are derived from these image statistics giving a principled way of formulating anisotropic diffusion problems in which all edge-stopping parameters are learned from training data.