Identifying predictive covariates, which forecast individual treatment effectiveness, is crucial for decision-making across different disciplines such as personalized medicine. These covariates, referred to as biomarkers, are extracted from pretreatment data, often within randomized controlled trials, and should be distinguished from prognostic biomarkers, which are independent of treatment assignment. Our study focuses on discovering predictive imaging biomarkers, specific image features, by leveraging pretreatment images to uncover new causal relationships. Unlike laborintensive approaches relying on handcrafted features prone to bias, we present a novel task of directly learning predictive features from images. We propose an evaluation protocol to assess a model's ability to identify predictive imaging biomarkers and differentiate them from purely prognostic ones by employing statistical testing and a comprehensive analysis of image feature attribution. We explore the suitability of deep learning models originally developed for estimating the conditional average treatment effect (CATE) for this task, which have been assessed primarily for their precision of CATE estimation while overlooking the evaluation of imaging biomarker discovery. Our proof-of-concept analysis demonstrates the feasibility and potential of our approach in discovering and validating predictive imaging biomarkers from synthetic outcomes and real-world image datasets. Our code is available at https://github.com/MIC-DKFZ/predictive_image_biomarker_analysis.

This work aims to improve the efficiency of text-to-image diffusion models. While diffusion models use computationallyexpensive UNet-based denoising operations in ev-ery generation step, we identify that not all operations are equally relevant for the final output quality. In par-ticular, we observe that UNet layers operating on high-res feature maps are relatively sensitive to small pertur-bations. In contrast, low-res feature maps influence the semantic layout of the final image and can often be per-turbed with no noticeable change in the output. Based on this observation, we propose Clockwork Diffusion, a method that periodically reuses computation from preceding denoising steps to approximate low-res feature maps at one or more subsequent steps. For multiple base-lines, and for both text-to-image generation and image editing, we demonstrate that Clockwork leads to compa-rable or improved perceptual scores with drastically re-duced computational complexity. As an example, for Sta-ble Diffusion vI.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and CLIP change. We re-lease code at https://github.com/Qualcomm-AI-research/clockwork-diffusion

Neural video compression algorithms are nearly competitive with hand-crafted codecs in terms of rate-distortion performance and subjective quality. However, many neural codecs are inflexible black boxes, and give users little to no control over the reconstruction quality and bitrate. In this work, we present a flexible neural video codec that combines ideas from variable-bitrate codecs and region-of-interest-based coding. By conditioning our model on a global rate-distortion tradeoff parameter and a region-of-interest (ROI) mask, we obtain dynamic control over the per-frame bitrate and the reconstruction quality in the ROI at test time. The resulting codec enables practical use cases such as coding under bitrate constraints with fixed ROI quality, while taking a negligible hit in performance compared to a fixed-rate model. We find that our codec performs best on sequences with complex motion, where we substantially outperform non-ROI codecs in the region of interest with Bjøntegaard-Delta rate savings exceeding 60%.

In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094.

Detecting Out-of-Distribution (OoD) data is one of the greatest challenges in safe and robust deployment of machine learning algorithms in medicine. When the algorithms encounter cases that deviate from the distribution of the training data, they often produce incorrect and over-confident predictions. OoD detection algorithms aim to catch erroneous predictions in advance by analysing the data distribution and detecting potential instances of failure. Moreover, flagging OoD cases may support human readers in identifying incidental findings. Due to the increased interest in OoD algorithms, benchmarks for different domains have recently been established. In the medical imaging domain, for which reliable predictions are often essential, an open benchmark has been missing. We introduce the Medical-Out-Of-Distribution-Analysis-Challenge (MOOD) as an open, fair, and unbiased benchmark for OoD methods in the medical imaging domain. The analysis of the submitted algorithms shows that performance has a strong positive correlation with the perceived difficulty, and that all algorithms show a high variance for different anomalies, making it yet hard to recommend them for clinical practice. We also see a strong correlation between challenge ranking and performance on a simple toy test set, indicating that this might be a valuable addition as a proxy dataset during anomaly detection algorithm development.

Neural Processes (NPs) are a family of conditional generative models that are able to model a distribution over functions, in a way that allows them to perform predictions at test time conditioned on a number of context points. A recent addition to this family, Convolutional Conditional Neural Processes (ConvCNP), have shown remarkable improvement in performance over prior art, but we find that they sometimes struggle to generalize when applied to time series data. In particular, they are not robust to distribution shifts and fail to extrapolate observed patterns into the future. By incorporating a Gaussian Process into the model, we are able to remedy this and at the same time improve performance within distribution. As an added benefit, the Gaussian Process reintroduces the possibility to sample from the model, a key feature of other members in the NP family.

Accurate assessment of pulmonary emphysema is crucial to assess disease severity and subtype, to monitor disease progression, and to predict lung cancer risk. However, visual assessment is time-consuming and subject to substantial inter-rater variability while standard densitometry approaches to quantify emphysema remain inferior to visual scoring. We explore if machine learning methods that learn from a large dataset of visually assessed CT scans can provide accurate estimates of emphysema extent and if methods that learn from emphysema extent scoring can outperform algorithms that learn only from emphysema presence scoring. Four Multiple Instance Learning classifiers, trained on emphysema presence labels, and five Learning with Label Proportions classifiers, trained on emphysema extent labels, are compared. Performance is evaluated on 600 low-dose CT scans from the Danish Lung Cancer Screening Trial and we find that learning from emphysema presence labels, which are much easier to obtain, gives equally good performance to learning from emphysema extent labels. The best performing Multiple Instance Learning and Learning with Label Proportions classifiers, achieve intra-class correlation coefficients around 0.90 and average overall agreement with raters of 78% and 79% compared to an inter-rater agreement of 83%.

Optimal surface methods are a class of graph cut methods posing surface estimation as an n-ary ordered labeling problem. They are used in medical imaging to find interacting and layered surfaces optimally and in low order polynomial time. Representing continuous surfaces with discrete sets of labels, however, leads to discretization errors and, if graph representations are made dense, excessive memory usage. Limiting memory usage and computation time of graph cut methods are important and graphs that locally adapt to the problem has been proposed as a solution. Min-marginal energies computed using dynamic graph cuts offer a way to estimate solution uncertainty and these uncertainties have been used to decide where graphs should be adapted. Adaptive graphs, however, introduce extra parameters, complexity, and heuristics. We propose a way to use min-marginal energies to estimate continuous solution labels that does not introduce extra parameters and show empirically on synthetic and medical imaging datasets that it leads to improved accuracy. The increase in accuracy was consistent and in many cases comparable with accuracy otherwise obtained with graphs up to eight times denser, but with proportionally less memory usage and improvements in computation time.

We present a new three-dimensional coupled optimal surface graph-cut algorithm to segment the wall of the carotid artery bifurcation from Magnetic Resonance (MR) images. The method combines the search for both inner and outer borders into a single graph cut and uses cost functions that integrate information from multiple sequences. Our approach requires manual localization of only three seed points indicating the start and end points of the segmentation in the internal, external, and common carotid artery. We performed a quantitative validation using images of 57 carotid arteries. Dice overlap of 0.86 ± 0.06 for the complete vessel and 0.89 ± 0.05 for the lumen compared to manual annotation were obtained. Reproducibility tests were performed in 60 scans acquired with an interval of 15 ± 9 days, showing good agreement between baseline and follow-up segmentations with intraclass correlations of 0.96 and 0.74 for the lumen and complete vessel volumes respectively.

We present a fast and robust atlas-based algorithm for labeling airway trees, using geodesic distances in a geometric tree-space. Possible branch label configurations for an unlabeled airway tree are evaluated using distances to a training set of labeled airway trees. In tree-space, airway tree topology and geometry change continuously, giving a natural automatic handling of anatomical differences and noise. A hierarchical approach makes the algorithm efficient, assigning labels from the trachea and downwards. Only the airway centerline tree is used, which is relatively unaffected by pathology. The algorithm is evaluated on 80 segmented airway trees from 40 subjects at two time points, labeled by three medical experts each, testing accuracy, reproducibility and robustness in patients with chronic obstructive pulmonary disease (COPD). The accuracy of the algorithm is statistically similar to that of the experts and not significantly correlated with COPD severity. The reproducibility of the algorithm is significantly better than that of the experts, and negatively correlated with COPD severity. Evaluation of the algorithm on a longitudinal set of 8724 trees from a lung cancer screening trial shows that the algorithm can be used in large scale studies with high reproducibility, and that the negative correlation of reproducibility with COPD severity can be explained by missing branches, for instance due to segmentation problems in COPD patients. We conclude that the algorithm is robust to COPD severity given equally complete airway trees, and comparable in performance to that of experts in pulmonary medicine, emphasizing the suitability of the labeling algorithm for clinical use.

While graphs with continuous node attributes arise in many applications, state-of-the-art graph kernels for comparing continuous-attributed graphs suffer from a high runtime complexity; for instance, the popular shortest path kernel scales as $\mathcal{O}(n^4)$, where $n$ is the number of nodes. In this paper, we present a class of path kernels with computational complexity $\mathcal{O}(n^2 (m + \delta^2))$, where $\delta$ is the graph diameter and $m$ the number of edges. Due to the sparsity and small diameter of real-world graphs, these kernels scale comfortably to large graphs. In our experiments, the presented kernels outperform state-of-the-art kernels in terms of speed and accuracy on classification benchmark datasets.