Learning medical visual representations from image-report pairs through joint learning has garnered increasing research attention due to its potential for transferring acquired knowledge to various downstream medical tasks. Previous works have predominantly focused on instance-wise or token-wise cross-modal alignment, often neglecting the importance of pathological-level consistency. This paper presents a novel framework PLACE that promotes the Pathological-Level Alignment and enriches the fine-grained details via Correlation Exploration without additional human annotations. Specifically, we propose a novel pathological-level cross-modal alignment (PCMA) approach to maximize the consistency of pathology observations from both images and reports. To facilitate this, a Visual Pathology Observation Extractor is introduced to extract visual pathological observation representations from localized tokens. The PCMA module operates independently of any external disease annotations, enhancing the generalizability and robustness of our methods. Furthermore, we design a proxy task that enforces the model to identify correlations among image patches, thereby enriching the fine-grained details crucial for various downstream tasks. Experimental results demonstrate that our proposed framework achieves new state-of-the-art performance on multiple downstream tasks, including classification, image-to-text retrieval, semantic segmentation, object detection and report generation.

To help address the occlusion problem in panoptic segmentation and image understanding, this paper proposes a new large-scale dataset named COCO-OLAC (COCO Occlusion Labels for All Computer Vision Tasks), which is derived from the COCO dataset by manually labelling images into three perceived occlusion levels. Using COCO-OLAC, we systematically assess and quantify the impact of occlusion on panoptic segmentation on samples having different levels of occlusion. Comparative experiments with SOTA panoptic models demonstrate that the presence of occlusion significantly affects performance, with higher occlusion levels resulting in notably poorer performance. Additionally, we propose a straightforward yet effective method as an initial attempt to leverage the occlusion annotation using contrastive learning to render a model that learns a more robust representation capturing different severities of occlusion. Experimental results demonstrate that the proposed approach boosts the performance of the baseline model and achieves SOTA performance on the proposed COCO-OLAC dataset.1

We introduce LYSTO, the Lymphocyte Assessment Hackathon, which was held in conjunction with the MICCAI 2019 Conference in Shenzhen (China). The competition required participants to automatically assess the number of lymphocytes, in particular T-cells, in images of colon, breast, and prostate cancer stained with CD3 and CD8 immunohistochemistry. Differently from other challenges setup in medical image analysis, LYSTO participants were solely given a few hours to address this problem. In this paper, we describe the goal and the multi-phase organization of the hackathon; we describe the proposed methods and the on-site results. Additionally, we present post-competition results where we show how the presented methods perform on an independent set of lung cancer slides, which was not part of the initial competition, as well as a comparison on lymphocyte assessment between presented methods and a panel of pathologists. We show that some of the participants were capable to achieve pathologist-level performance at lymphocyte assessment. After the hackathon, LYSTO was left as a lightweight plug-and-play benchmark dataset on grand-challenge website, together with an automatic evaluation platform.

Radiology report generation (RRG) has gained increasing research attention because of its huge potential to mitigate medical resource shortages and aid the process of disease decision making by radiologists. Recent advancements in Radiology Report Generation (RRG) are largely driven by improving a model's capabilities in encoding single-modal feature representations, while few studies explicitly explore the cross-modal alignment between image regions and words. Radiologists typically focus first on abnormal image regions before composing the corresponding text descriptions, thus cross-modal alignment is of great importance to learn a RRG model which is aware of abnormalities in the image. Motivated by this, we propose a Class Activation Map guided Attention Network (CAMANet) which explicitly promotes cross-modal alignment by employing aggregated class activation maps to supervise cross-modal attention learning, and simultaneously enrich the discriminative information. Experimental results demonstrate that CAMANet outperforms previous SOTA methods on two commonly used RRG benchmarks.

We introduce a new backdoor attack against a deep-learning video rebroadcast detection network. In addition to the difficulties of working with video signals rather than still images, injecting a backdoor into a deep learning model for rebroadcast detection presents the additional problem that the backdoor must survive the digital-to-analog and analog-to-digital conversion associated to video rebroadcast. To cope with this problem, we have built a backdoor attack that works by varying the average luminance of video frames according to a predesigned sinusoidal function. In this way, robustness against geometric transformation is automatically achieved, together with a good robustness against luminance transformations associated to display and recapture, like Gamma correction and white balance. Our experiments demonstrate the effectiveness of the proposed backdoor attack, especially when the attack is carried out by also corrupting the labels of the attacked training samples.

The rise of intelligent vehicle systems will lead to more human-machine interactions and so there is a need to create a bridge between the system and the actions and behaviours of the people inside the vehicle. In this paper, we propose a dual camera setup to monitor the actions and behaviour of vehicle passengers and a deep learning architecture which can utilise video data to classify a range of actions. The method incorporates two different views as input to a 3D convolutional network and uses transfer learning from other action recognition data. The performance of this method is evaluated using an in-vehicle dataset, which contains video recordings of people performing a range of common in-vehicle actions. We show that the combination of transfer learning and using dual viewpoints in a 3D action recognition network offers an increase in classification accuracy of action classes with distinct poses, e.g. mobile phone use and sleeping, whilst it does not apply as well for classifying those actions with small movements, such as talking and eating.

Driving is a safety critical task that requires a high levels of attention and workload from the driver. Despite this, people often also perform secondary tasks such as eating or using a mobile phone, which increase workload levels and divert cognitive and physical attention from the primary task of driving. If a vehicle is aware that the driver is currently under high workload, the vehicle functionality can be changed in order to minimize any further demand. Traditionally, workload measurements have been performed using intrusive means such as physiological sensors. Another approach may be to monitor workload online using readily available and robust sensors accessible via the vehicle's Controller Area Network (CAN). In this paper, we present details of the Warwick-JLR Driver Monitoring Dataset (DMD) collected for this purpose, and to announce its publication for driver monitoring research. The collection protocol is briefly introduced, followed by statistical analysis of the dataset to describe its structure. Finally, the public release of the dataset, for use in both driver monitoring and data mining research, is announced.

We focus on optimising the Active Shape Model (ASM) with several extensions. The modification is threefold. First, we tackle the over-constraint problem and obtain an optimal shape with minimum energy considering both the shape prior and the salience of local features, based on statistical theory: a compact closed form solution to the optimal shape is deduced. Second, we enhance the ASM searching method by modelling and removing the variations of local appearance presented in the training data. Third, we speed up the convergence of shape fitting by integrating information from multi-scale local features simultaneously. Experiments show significant improvement brought by these modifications, i.e., optimal shape against standard relaxation methods dealing with inadequate training samples; enhanced searching method against standard gradient descent methods in searching accuracy; multi-scale local features against popular coarse-to-fine strategies in convergence speed.

The importance of memory performance and capacity is a growing concern for high performance computing laboratories around the world. It has long been recognized that improvements in processor speed exceed the rate of improvement in dynamic random access memory speed and, as a result, memory access times can be the limiting factor in high performance scientific codes. The use of multi-core processors exacerbates this problem with the rapid growth in the number of cores not being matched by similar improvements in memory capacity, increasing the likelihood of memory contention. In this paper, we present WMTools, a lightweight memory tracing tool and analysis framework for parallel codes, which is able to identify peak memory usage and also analyse per-function memory use over time. An evaluation of WMTools, in terms of its effectiveness and also its overheads, is performed using nine established scientific applications/benchmark codes representing a variety of programming languages and scientific domains. We also show how WMTools can be used to automatically generate a parameterized memory model for one of these applications, a two-dimensional non-linear magnetohydrodynamics application, Lare2D. Through the memory model we are able to identify an unexpected growth term which becomes dominant at scale. With a refined model we are able to predict memory consumption with under 7% error.

Input/Output (I/O) operations can represent a significant proportion of the run-time of parallel scientific computing applications. Although there have been several advances in file format libraries, file system design and I/O hardware, a growing divergence exists between the performance of parallel file systems and the compute clusters that they support. In this paper, we document the design and application of the RIOT I/O toolkit (RIOT) being developed at the University of Warwick with our industrial partners at the Atomic Weapons Establishment and Sandia National Laboratories. We use the toolkit to assess the performance of three industry-standard I/O benchmarks on three contrasting supercomputers, ranging from a mid-sized commodity cluster to a large-scale proprietary IBM BlueGene/P system. RIOT provides a powerful framework in which to analyse I/O and parallel file system behaviour—we demonstrate, for example, the large file locking overhead of IBM's General Parallel File System, which can consume nearly 30% of the total write time in the FLASH-IO benchmark. Through I/O trace analysis, we also assess the performance of HDF-5 in its default configuration, identifying a bottleneck created by the use of suboptimal Message Passing Interface hints. Furthermore, we investigate the performance gains attributed to the Parallel Log-structured File System (PLFS) being developed by EMC Corporation and the Los Alamos National Laboratory. Our evaluation of PLFS involves two high-performance computing systems with contrasting I/O backplanes and illustrates the varied improvements to I/O that result from the deployment of PLFS (ranging from up to 25× speed-up in I/O performance on a large I/O installation to 2× speed-up on the much smaller installation at the University of Warwick).

Natural images contain considerable redundancy, some of which is successfully captured using recently developed directional wavelets. In this paper, an affine symmetric image model is considered. It provides a flexible scheme to exploit geometric redundancy. A patch of texture from an image is rotated, scaled and sheared to approximate other similar parts in the image, revealing the self-similarity relation. The general scheme is derived as follows. A texture model is required that identifies structural patterns. Then the affine symmetry is exploited between structural textures at a local level, the objective being to find the minimum residual error by estimating the affine transform relating two patches of texture. Having developed a local model, the methodology is extended to the whole image to estimate the global affine relation. This global model is further developed in a multiresolution framework for multiscale analysis, by which the self similarity of the image is exploited across space and scale. The multiresolution model can be applied to a series of practical problems. Experimental evaluation demonstrates the effectiveness of the approach in affine invariant texture segmentation and image approximation.

A general purpose block-to-block affine transformation estimator is described. The estimator is based on Fourier slice analysis and Fourier spectral alignment. It shows encouraging performance in terms of both speed and accuracy compared to existing methods. The key elements of its success are attributed to the ability to: 1) locate an arbitrary number of affine invariant points in the spectrum that latch onto significant structural features; 2) match the estimated invariant points with the target spectrum by the slicewise phase-correlation; and 3) use affine invariant points to directly compute all linear parameters of the full affine transform by spectral alignment. Experimental results using a wide range of textures are presented. Potential applications include affine invariant image segmentation, registration, affine symmetric image coding, and motion analysis.

The increasing availability of multi-core and multiprocessor architectures provides new opportunities for improving the performance of many computer simulations. Markov chain Monte Carlo (MCMC) simulations are widely used for approximate counting problems, Bayesian inference and as a means for estimating very high-dimensional integrals. As such MCMC has found a wide variety of applications infields including computational biology and physics, financial econometrics, machine learning and image processing. This paper presents a new method for reducing the run-time of Markov chain Monte Carlo simulations by using SMP machines to speculatively perform iterations in parallel, reducing the runtime of MCMC programs whilst producing statistically identical results to conventional sequential implementations. We calculate the theoretical reduction in runtime that may be achieved using our technique under perfect conditions, and test and compare the method on a selection of multi-core and multi-processor architectures. Experiments are presented that show reductions in runtime of 35% using two cores and 55% using four cores.

Classical shape analysis methods use principal component analysis to reduce the dimensionality of shape spaces. The basic assumption behind these methods is that the subspace corresponding to the major modes of variation for a particular class of shapes is linearised. This may not necessarily be the case in practice. In this paper, we present a novel method for extraction of the intrinsic parameters of multiple shape classes in an unsupervised manner. The proposed method is based on learning the global structure of shape manifolds using diffusion maps. We demonstrate that the method is effective in separating the members of different shape classes after embedding them into a low-dimensional Euclidean space. 1

Many natural textures comprise structural patterns and show strong self-similarity. We use affine symmetry to segment an image into self-similar regions; that is a patch of texture (blocks from a uniformly partitioned image) can be transformed to other similar patches by warping. If the texture image contains multiple regions, we then cluster patches into a number of classes such that the overall warping error is minimized. Discovering the optimal clusters is not trivial and known methods are computationally intensive due to the affine transformation. We demonstrate efficient segmentation of structural textures without affine computation. The algorithm uses Fourier Slice Analysis to obtain a spectral contour signature. Experimental evaluation on structural textures shows encouraging results and application on natural images demonstrates identification of texture objects.

In this paper, a multiresolution volumetric texture segmentation (M-VTS) algorithm is presented. The method extracts textural measurements from the Fourier domain of the data via subband filtering using an orientation pyramid (Wilson and Spann, 1988). A novel Bhattacharyya space, based on the Bhattacharyya distance, is proposed for selecting the most discriminant measurements and producing a compact feature space. An oct tree is built of the multivariate features space and a chosen level at a lower spatial resolution is first classified. The classified voxel labels are then projected to lower levels of the tree where a boundary refinement procedure is performed with a three-dimensional (3-D) equivalent of butterfly filters. The algorithm was tested with 3-D artificial data and three magnetic resonance imaging sets of human knees with encouraging results. The regions segmented from the knees correspond to anatomical structures that can be used as a starting point for other measurements such as cartilage extraction.

This paper presents a vascular representation and segmentation algorithm based on a multiresolution Hermite model (MHM). A two-dimensional Hermite function intensity model is developed which models blood vessel profiles in a quad-tree structure over a range of spatial resolutions. The use of a multiresolution representation simplifies the image modeling and allows for a robust analysis by combining information across scales. Estimation over scale also reduces the overall computational complexity. As well as using MHM for vessel labelling, the local image modeling can accurately represent vessel directions, widths, amplitudes, and branch points which readily enable the global topology to be inferred. An expectation-maximization (EM) type of optimization scheme is used to estimate local model parameters and an information theoretic test is then applied to select the most appropriate scale/feature model for each region of the image. In the final stage, Bayesian stochastic inference is employed for linking the local features to obtain a description of the global vascular structure. After a detailed description and analysis of MHM, experimental results on two standard retinal databases are given that demonstrate its comparative performance. These show MHM to perform comparably with other retinal vessel labelling methods.

An estimation method for approximating the lighting of a multi-view scene is developed. It is assumed that a set of scene patches can be obtained with estimates of their normals and depths from a given camera. The effect of lighting on the scene is modelled as multiplicative and additive bias-fields, represented by spherical harmonic (SH) basis functions. Parameters of a weighted sum of SHs to a given order are sought by minimising the entropy of the patch colours as the bias is taken out. The method performs a gradient descent using the entropy as a loss function. The entropy is estimated by sampling using a Parzen window estimator which allows its change with respect to the SH weights to be calculated analytically. We illustrate our estimator on 2D retrospective shading correction and then define Phong illumination as a bias field estimation problem and its continuous generalisation. Results on simple modelled scenes lit by one or more Phong point light sources without scattering are presented. We discuss how the lighting estimation could be extended to handle shadows and propose a model for estimation of parametric BRDFs in arbitrary lighting using the same framework. 1

A novel image-dependent representation, warplets, based on self-similarity of regions is introduced. The representation is well suited to the description and segmentation of images containing textures and oriented patterns, such as fingerprints. An affine model of an image as a collection of self-similar image blocks is developed and it is shown how textured regions can be represented by a single prototype block together with a set of transformation coefficients. Images regions are aligned to a set of dictionary blocks and their variability captured by PCA analysis. The block-to-block transformations are found by Gaussian mixture modelling of the block spectra and a least-squares estimation. Clustering in the April domain can be used to determine a April dictionary. Experimental results on a variety of images demonstrate the potential of the use of April for segmentation and coding.

This paper introduces a method of image segmentation that partitions an image into regions exhibiting self-similarity. An affine texture de-scription which models an image as a collection of self-similar image blocks is developed. Transformations between prototype and target blocks are estimated by Gaussian Mixture Modelling of block spectra and least-squares estimation. An appropriate set of prototype blocks (codebook) is determined for a given an image by clustering in a fea-ture subspace of discrete samples from an affine symmetry group. The new method is well suited to the description and segmentation of images containing textures, such as fingerprints and medical images and could be used as a basis for indexing and searching in CBIR applications. Ex-perimental results are presented which demonstrate the potential of the method on widely varying images. µ

The computational complexity of a texture classification algorithm is limited by the dimensionality of the feature space. Although finding the optimal feature subset is a NP-hard problem [1], a feature selection algorithm that can reduce the dimensionality of problem is often desirable. In this paper, we report work on a feature selection algorithm for texture classification using two subband filtering methods: a full wavelet packet decomposition and a Gabor type decomposition. The value of a cost function associated with a subband (feature) is used as a measure of relevance of that subband for classification purposes. This leads to a fast feature selection algorithm which ranks the features according to their measure of relevance. Experiments on a range of test images and both filtering methods provide results that are promising. 1

We describe a method for inferring tree-like vascular structures from 2D imagery. A Markov chain Monte Carlo (MCMC) algorithm is employed to sample from the posterior distribution given local feature estimates, derived from likelihood maximisation for a Gaussian intensity profile. A multiresolution scheme, in which coarse scale estimates are used to initialise the algorithm for finer scales, has been implemented and used to model retinal images. Results are presented to show the effectiveness of the method.

This paper describes an estimation and representation method for object structure in 3D image data. A windowed Fourier transform is used to estimate the parameters of 3D local structures (lines, planes and spheres) across a number of window sizes or scales. We present the estimation method and describe a decision process which aims to find the simplest object description. Results of the method on test data and blood vessels from magnetic resonance imaging are presented. 1

This paper describes a novel approach to tissue classification using three-dimensional (3D) derivative features in the volume rendering pipeline. In conventional tissue classification for a scalar volume, tissues of interest are characterized by an opacity transfer function defined as a one-dimensional (1D) function of the original volume intensity. To overcome the limitations inherent in conventional 1D opacity functions, we propose a tissue classification method that employs a multidimensional opacity function, which is a function of the 3D derivative features calculated from a scalar volume as well as the volume intensity. Tissues of interest are characterized by explicitly defined classification rules based on 3D filter responses highlighting local structures, such as edge, sheet, line, and blob, which typically correspond to tissue boundaries, cortices, vessels, and nodules, respectively, in medical volume data. The 3D local structure filters are formulated using the gradient vector and Hessian matrix of the volume intensity function combined with isotropic Gaussian blurring. These filter responses and the original intensity define a multidimensional feature space in which multichannel tissue classification strategies are designed. The usefulness of the proposed method is demonstrated by comparisons with conventional single-channel classification using both synthesized data and clinical data acquired with CT (computed tomography) and MRI (magnetic resonance imaging) scanners. The improvement in image quality obtained using multichannel classification is confirmed by evaluating the contrast and contrast-to-noise ratio in the resultant volume-rendered images with variable opacity values.

In this paper we focus on using local 3D structure for segmentation. A tensor descriptor is estimated for each neighbourhood, i.e. for each voxel in the data set. The tensors are created from a combination of the outputs form a set of 3D quadrature filters. The shape of the tensors describe locally the structure of the neighbourhood in terms of how much it is like a plane, a line, and a sphere. We apply this to segmentation of bone from Computer Tomography data (CT). Traditional methods are based purely on gray-level value discrimination and have difficulties in recovering thin bone structures due to so called partial voluming, a problem which is present in all such sampled data. We illuminate the partial voluming problem by showing that thresholding creates complicated artifacts even if the signal is densely enough sampled and can be perfectly reconstructed. The unwanted effects of thresholding can be reduced by a change of the signal basis. We show that by using additional local structure information can significantly reduce the degree of sampling artifacts. Evaluation of the method on a clinical case is presented, the segmentation of a human skull from a CT volume. The method shows that many of the thin bone structures which disappear in a pure thresholding can be recovered.

Deals with the design of filter kernels having specified radial and angular frequency responses based on combined optimization and frequency sampling. This is used to generate small-radius, low-pass, and edge-detection kernels for multiresolution pyramids. The performance of the new kernels in estimating orientation is shown to be significantly better than that of other commonly used pyramid kernels.>

The problem of estimating feature orientation from noisy image data is addressed using a multiresolution technique. It is shown that by an appropriate choice of representation of orientation, it is possible to employ simple linear smoothing methods to reduce estimation noise. The smoothing is done using a combination of scale-space recursive filtering and iterative estimation, giving significant improvements in estimated orientations at low computational cost. Applications to enhancement and segmentation are presented.