Being able to learn on weakly labeled data and provide interpretability are two of the main reasons why attention-based deep multiple instance learning (ABMIL) methods have become particularly popular for classification of histopathological images. Such image data usually come in the form of gigapixel-sized whole-slide-images (WSI) that are cropped into smaller patches (instances). However, the sheer volume of the data poses a practical big data challenge: All the instances from one WSI cannot fit the GPU memory of conventional deep-learning models. Existing solutions compromise training by relying on pre-trained models, strategic selection of instances, sub-sampling, or self-supervised pre-training. We propose a training strategy based on gradient accumulation that enables direct end-to-end training of ABMIL models without being limited by GPU memory. We conduct experiments on both QMNIST and Imagenette to investigate the performance and training time and compare with the conventional memory-expensive baseline as well as a recent sampled-based approach. This memory-efficient approach, although slower, reaches performance indistinguishable from the memory-expensive baseline.

Infant engagement during guided play is a reliable indicator of early learning outcomes, psychiatric issues and familial wellbeing. An obstacle to using such information in real-world scenarios is the need for a domain expert to assess the data. We show that an end-to-end Deep Learning approach can perform well in automatic infant engagement detection from a single video source, without requiring a clear view of the face or the whole body. To tackle the problem of explainability in learning methods, we evaluate how four common attention mapping techniques can be used to perform subjective evaluation of the network’s decision process and identify multimodal cues used by the network to discriminate engagement levels. We further propose a quantitative comparison approach, by collecting a human attention baseline and evaluating its similarity to each technique.

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI, which typically works well only for small displacements. This points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g., rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, containing remote sensing images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI, as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at github.com/MIDA-group/globalign.

We propose contrastive coding to learn shared, dense image representations, referred to as CoMIRs (Contrastive Multimodal Image Representations). CoMIRs enable the registration of multimodal images where existing registration methods often fail due to a lack of sufficiently similar image structures. CoMIRs reduce the multimodal registration problem to a monomodal one, in which general intensity-based, as well as feature-based, registration algorithms can be applied. The method involves training one neural network per modality on aligned images, using a contrastive loss based on noise-contrastive estimation (InfoNCE). Unlike other contrastive coding methods, used for, e.g., classification, our approach generates image-like representations that contain the information shared between modalities. We introduce a novel, hyperparameter-free modification to InfoNCE, to enforce rotational equivariance of the learnt representations, a property essential to the registration task. We assess the extent of achieved rotational equivariance and the stability of the representations with respect to weight initialization, training set, and hyperparameter settings, on a remote sensing dataset of RGB and near-infrared images. We evaluate the learnt representations through registration of a biomedical dataset of bright-field and second-harmonic generation microscopy images; two modalities with very little apparent correlation. The proposed approach based on CoMIRs significantly outperforms registration of representations created by GAN-based image-to-image translation, as well as a state-of-the-art, application-specific method which takes additional knowledge about the data into account. Code is available at: https://github.com/MIDA-group/CoMIR.

The authors propose a novel HEp‐2 cell image classifier to improve the automation process of patients' serum evaluation. The authors' solution builds on the recent progress in deep learning based image classification. They propose an ensemble approach using multiple state‐of‐the‐art architectures. They incorporate additional texture information extracted by an improved version of local binary patterns maps, LBP‐maps, which enables to create a very effective cell image classifier. This innovative combination is trained on three publicly available datasets and its general applicability is demonstrated through the evaluation on three independent test sets. The presented results show that their approach leads to a general improvement of performance on average on the three public datasets.

Intensity-based image registration approaches rely on similarity measures to guide the search for geometric correspondences with high affinity between images. The properties of the used measure are vital for the robustness and accuracy of the registration. In this study a symmetric, intensity interpolationfree, affine registration framework based on a combination of intensity and spatial information is proposed. The excellent performance of the framework is demonstrated on a combination of synthetic tests, recovering known transformations in the presence of noise, and real applications in biomedical and medical image registration, for both 2D and 3D images. The method exhibits greater robustness and higher accuracy than similarity measures in common use, when inserted into a standard gradientbased registration framework available as part of the open source Insight Segmentation and Registration Toolkit (ITK). The method is also empirically shown to have a low computational cost, making it practical for real applications. Source code is available.

Local binary pattern (LBP) descriptors have been popular in texture classification in recent years. They were introduced as descriptors of local image texture and their histograms are shown to be well performing texture features. In this paper we introduce two new LBP descriptors, αLBP and its improved variant IαLBP. We evaluate their performance in classification by comparing them with some of the existing LBP descriptors - LBP, ILBP, shift LBP (SLBP) and with one ternary descriptor - LTP. The texture descriptors are evaluated on three datasets - KTH-TIPS2b, UIUC and Virus texture dataset. The novel descriptor outperforms the other descriptors on two datasets, KTH-TIPS2b and Virus, and is tied for first place with ILBP on the UIUC dataset.

We present a method for simulating lower quality images starting from higher quality ones, based on acquired image pairs from different optical setups. The method does not require estimates of point (or line) spread functions of the system, but utilizes the relative transfer function derived from images of real specimen of interest in the observed application. Thanks to the use of a larger number of real specimen, excellent stability and robustness of the method is achieved. The intended use is exploring the influence of image quality on features and classification accuracy in pattern recognition based screening tasks. Visual evaluation of the obtained images strongly confirms usefulness of the method. The approach is quantitatively evaluated by observing stability of feature values, proven useful for PAP-smear classification, between synthetic and real images from seven different microscope setups. The evaluation shows that features from the synthetically generated lower resolution images are as similar to features from real images at that resolution, as features from two different images of the same specimen, taken at the same low resolution, are to each other.

We present four novel point-to-set distances defined for fuzzy or gray-level image data, two based on integration over α-cuts and two based on the fuzzy distance transform. We explore their theoretical properties. Inserting the proposed point-to-set distances in existing definitions of set-to-set distances, among which are the Hausdorff distance and the sum of minimal distances, we define a number of distances between fuzzy sets. These set distances are directly applicable for comparing gray-level images or fuzzy segmented objects, but also for detecting patterns and matching parts of images. The distance measures integrate shape and intensity/membership of observed entities, providing a highly applicable tool for image processing and analysis. Performance evaluation of derived set distances in real image processing tasks is conducted and presented. It is shown that the considered distances have a number of appealing theoretical properties and exhibit very good performance in template matching and object classification for fuzzy segmented images as well as when applied directly on gray-level intensity images. Examples include recognition of hand written digits and identification of virus particles. The proposed set distances perform excellently on the MNIST digit classification task, achieving the best reported error rate for classification using only rigid body transformations and a kNN classifier.

Cancer diagnosis is based on visual examination under a microscope of tissue sections from biopsies. But whereas pathologists rely on tissue stains to identify morphological features, automated tissue recognition using color is fraught with problems that stem from image intensity variations due to variations in tissue preparation, variations in spectral signatures of the stained tissue, spectral overlap and spatial aliasing in acquisition, and noise at image acquisition. We present a blind method for color decomposition of histological images. The method decouples intensity from color information and bases the decomposition only on the tissue absorption characteristics of each stain. By modeling the charge-coupled device sensor noise, we improve the method accuracy. We extend current linear decomposition methods to include stained tissues where one spectral signature cannot be separated from all combinations of the other tissues' spectral signatures. We demonstrate both qualitatively and quantitatively that our method results in more accurate decompositions than methods based on non-negative matrix factorization and independent component analysis. The result is one density map for each stained tissue type that classifies portions of pixels into the correct stained tissue allowing accurate identification of morphological features that may be linked to cancer.

We propose a registration method to find affine transformations between 3D objects by constructing and solving an overdetermined system of polynomial equations. We utilize voxel coverage information for more precise object boundary description. An iterative solution enables us to easily adjust the method to recover e.g. rigid-body and similarity transformations. Synthetic tests show the advantage of the voxel coverage representation, and reveal the robustness properties of our method against different types of segmentation errors. The method is tested on a real medical CT volume.

SRμCT images of paper and pulp fiber materials are characterized by a low signal to noise ratio. De-noising is therefore a common preprocessing step before segmentation into fiber and background components. We suggest a de-noising method based on total variation minimization using a modified Spectral Conjugate Gradient algorithm. Quantitative evaluation performed on synthetic 3D data and qualitative evaluation on real 3D paper fiber data confirm appropriateness of the suggested method for the particular application.

We present a novel method that provides an accurate and precise estimate of the length of the boundary (perimeter) of an object by taking into account gray levels on the boundary of the digitization of the same object. Assuming a model where pixel intensity is proportional to the coverage of a pixel, we show that the presented method provides error-free measurements of the length of straight boundary segments in the case of nonquantized pixel values. For a more realistic situation, where pixel values are quantized, we derive optimal estimates that minimize the maximal estimation error. We show that the estimate converges toward a correct value as the number of gray levels tends toward infinity. The method is easy to implement; we provide the complete pseudocode. Since the method utilizes only a small neighborhood, it is very easy to parallelize. We evaluate the estimator on a set of concave and convex shapes with known perimeters, digitized at increasing resolution. In addition, we provide an example of applicability of the method on real images, by suggesting appropriate preprocessing steps and presenting results of a comparison of the suggested method with other local approaches.

Binary mathematical morphology can be computed by thresholding a distance transform, provided that the distance transform is a metric. Here we show that the polar distance transform is a metric and use it for morphological operations. The polar distance transform varies with the spatial coordinates of the image, resulting in spatially-variant morphology. In this distance transform each pixel is related to an image origin. We prefer angular propagation over radial, thus we construct structuring elements that are elongated in the angular direction, which is useful when circular segments are handled. We show an example where segments of annual rings on a log end face are connected using mathematical morphology based on the polar distance transform.