Over the past five years, artificial intelligence (AI) has introduced new models and methods for addressing the challenges associated with the broader adoption of AI models and systems in medicine. This paper reviews recent advances in AI for medical image and video analysis, outlines emerging paradigms, highlights pathways for successful clinical translation, and provides recommendations for future work. Hybrid Convolutional Neural Network (CNN) Transformer architectures now deliver state-of-the-art results in segmentation, classification, reconstruction, synthesis, and registration. Foundation and generative AI models enable the use of transfer learning to smaller datasets with limited ground truth. Federated learning supports privacy-preserving collaboration across institutions. Explainable and trustworthy AI approaches have become essential to foster clinician trust, ensure regulatory compliance, and facilitate ethical deployment. Together, these developments pave the way for integrating AI into radiology, pathology, and wider healthcare workflows.

In medical image segmentation, convolutional neural networks (CNNs) and transformers are dominant. For CNNs, given the local receptive fields of convolutional layers, long-range spatial correlations are captured through consecutive convolutions and pooling. However, as the computational cost and memory footprint can be prohibitively large, 3D models can only afford fewer layers than 2D models with reduced receptive fields and abstract levels. For transformers, although long-range correlations can be captured by multi-head attention, its quadratic complexity with respect to input size is computationally demanding. Therefore, either model may require input size reduction to allow more filters and layers for better segmentation. Nevertheless, given their discrete nature, models trained with patch-wise training or image downsampling may produce suboptimal results when applied on higher resolutions. To address this issue, here we propose the resolution-robust HNOSeg-XS architecture. We model image segmentation by learnable partial differential equations through the Fourier neural operator which has the zero-shot super-resolution property. By replacing the Fourier transform by the Hartley transform and reformulating the problem in the frequency domain, we created the HNOSeg-XS model, which is resolution robust, fast, memory efficient, and extremely parameter efficient. When tested on the BraTS'23, KiTS'23, and MVSeg'23 datasets with a Tesla V100 GPU, HNOSeg-XS showed its superior resolution robustness with fewer than 34.7k model parameters. It also achieved the overall best inference time (<0.24 s) and memory efficiency (<1.8 GiB) compared to the tested CNN and transformer models. The code repository is available at https://github.com/IBM/multimodal-3d-image-segmentation.

Maintaining and, ultimately, increasing vegetation coverage is likely the most impactful approach to globally capture carbon. Biomass is a crucial parameter for quantifying carbon stored in vegetation, and estimating it poses challenges as statistical models need to be customized to specific biomes. This study compares the prediction of aboveground biomass using various regression methods that were locally fitted in three distinct study sites located in Texas and Louisiana, USA. These sites (biomes) had average aboveground biomass densities of 4.1, 17.3, and 94.6 Mg/ha. The predictions obtained from these localized models were then compared to those derived from a general model that pooled data from all three sites together. Optical and radar imagery acquired from Sentinel satellites were used as predictors, while biomass density from GEDI served as the reference. In most experiments, Random Forest scored best, and the results indicate that the biome-specific models exhibited slightly higher accuracy. Specifically, the root mean square error (RMSE) values for the biome-specific models were 8.8, 16.8, and 54.8 Mg/ha, respectively. In comparison, the general model exhibited approximately 1 Mg/ha higher RMSE. The results indicate that the locally fitted models tailored to specific biomes generally outperformed the general model tested.

Soil organic carbon (SOC) sequestration is the transfer and storage of atmospheric carbon dioxide in soils, which plays an important role in climate change mitigation. SOC concentration can be improved by proper land use, thus it is beneficial if SOC can be estimated at a regional or global scale. As multispectral satellite data can provide SOC-related information such as vegetation and soil properties at a global scale, estimation of SOC through satellite data has been explored as an alternative to manual soil sampling. Although existing studies show promising results, they are mainly based on pixel-based approaches with traditional machine learning methods, and convolutional neural networks (CNNs) are uncommon. To study the use of CNNs on SOC remote sensing, here we propose the FNO-DenseNet based on the Fourier neural operator (FNO). By combining the advantages of the FNO and DenseNet, the FNO-DenseNet outperformed the FNO in our experiments with hundreds of times fewer parameters. The FNO-DenseNet also outperformed a pixel-based random forest by 18% in the mean absolute percentage error.

Nature-based carbon sequestration is currently the most viable solutions to extract CO2from the atmosphere and convert it into carbon. Oceans, soils and forests have the potential to capture and store large amount of carbon for decades. There is an ongoing debate about the permanence of the carbon sequestered by nature-based processes and the precise techniques required to monitor these carbon pools. Remote sensing plays a crucial role in the large scale observations of the Earth surface and provides a scalable method to monitor land use that can affect carbon sequestration. Optical spectral information and radar signals are the best candidates as proxy data to quantify and monitor the change in carbon sequestered. Here we outline the design of an AI enabled framework to monitor, verify, and quantify carbon sequestration in nature-based carbon sequestration processes.

Pancreatic neuroendocrine tumors are abnormal growths of hormone-producing cells in the pancreas. Unlike the brain which is protected by the skull, the pancreas can be significantly deformed by its surrounding organs. Consequently, the tumor shape differences observable from images at different time points arise from both tumor growth and pancreatic motion, and tumor growth model personalization may be compromised if such motion is ignored. Therefore, we incorporate pancreatic motion information derived from deformable image registration in model personalization. For more accurate mechanical interactions between tumor growth and pancreatic motion, elastic-growth decomposition is used with a hyperelastic constitutive law to model the mass effect, which allows growth modeling while conserving the mechanical properties. Furthermore, a way of coupling the finite difference method and the finite element method is proposed to greatly reduce the computation time. With both 2-[18F]-fluoro-2-deoxy-D-glucose positron emission tomographic and contrast-enhanced computed tomographic images, functional, structural, and motion data are combined for a patient-specific model. Experiments on synthetic and clinical data show the importance of image-derived motion on estimating pathophysiologically plausible mechanical properties and the promising performance of our framework. From seven patient data sets, the recall, precision, Dice coefficient, relative volume difference, and average surface distance between the personalized tumor growth simulations and the measurements were 83.2 ±8.8%, 86.9 ±8.3%, 84.4 ±4.0%, 13.9 ±9.8%, and 0.6 ±0.1 mm, respectively.

The problem of using surface data to reconstruct transmural electrophysiological (EP) signals is intrinsically ill-posed without a unique solution in its unconstrained form. Incorporating physiological spatiotemporal priors through probabilistic integration of dynamic EP models, we have previously developed a Bayesian approach to transmural electrophysiological imaging (TEPI) using body-surface electrocardiograms. In this study, we generalize TEPI to using electrical signals collected from heart surfaces, and we test its feasibility on two pre-clinical swine models provided through the STACOM 2011 EP simulation Challenge. Since this new application of TEPI does not require whole-body imaging, there may be more immediate potential in EP laboratories where it could utilize catheter mapping data and produce transmural information for therapy guidance. Another focus of this study is to investigate the consistency among three modalities in delineating scar after myocardial infarction: TEPI, electroanatomical voltage mapping (EAVM), and magnetic resonance imaging (MRI). Our preliminary data demonstrate that, compared to the low-voltage scar area in EAVM, the 3-D electrical scar volume detected by TEPI is more consistent with anatomical scar volume delineated in MRI. Furthermore, TEPI could complement anatomical imaging by providing EP functional features related to both scar and healthy tissue.

The recent advances in meaningful constraining models have resulted in increasingly useful quantitative information recovered from cardiac images. Nevertheless, as most frameworks utilize either functional or structural images, the analyses cannot benefit from the complementary information provided by the other image sources. To better characterize subject-specific cardiac physiology and pathology, data fusion of multiple image sources is essential. Traditional image fusion strategies are performed by fusing information of commensurate images through various mathematical operators. Nevertheless, when image data are dissimilar in physical nature and spatiotemporal quantity, such approaches may not provide meaningful connections between different data. In fact, as different image sources provide partial measurements of the same cardiac system dynamics, it is more natural and suitable to utilize cardiac physiological models for the fusions. Therefore, we propose to use the cardiac physiome model as the central link to fuse functional and structural images for more subject-specific cardiac deformation recovery through state-space filtering. Experiments were performed on synthetic and real data for the characteristics and potential clinical applicability of our framework, and the results show an increase of the overall subject specificity of the recovered deformations.

To systematically couple images and physiological models according to their respective merits, state-space filtering frameworks have been proposed for cardiac deformation recovery with promising results. Nevertheless, as thousands of forward simulations are required in every filtering step, the computational complexity is too high to be practical. To reduce the computational complexity without a significant loss of accuracy, we have adopted the mode superposition approach which transforms the cardiac system dynamics to a mathematically equivalent space of much lower dimensions. With the corresponding filtering procedures and components proposed, nonlinear cardiac deformation recovery can be performed in the transformed space with largely reduced computational complexity. Experiments were performed on synthetic data to evaluate the computational complexity and accuracy, and on human data for the clinical relevance.

Volumetric details of cardiac electrophysiology, such as transmembrane potential dynamics and tissue excitability of the myocardium, are of fundamental importance for understanding normal and pathological cardiac mechanisms, and for aiding the diagnosis and treatment of cardiac arrhythmia. Noninvasive observations, however, are made on body surface as an integration-projection of the volumetric phenomena inside patient's heart. We present a physiological-model-constrained statistical framework where prior knowledge of general myocardial electrical activity is used to guide the reconstruction of patient-specific volumetric cardiac electrophysiological details from body surface potential data. Sequential data assimilation with proper computational reduction is developed to estimate transmembrane potential and myocardial excitability inside the heart, which are then utilized to depict arrhythmogenic substrates. Effectiveness and validity of the framework is demonstrated through its application to evaluate the location and extent of myocardial infract using real patient data.

To noninvasively reconstruct transmembrane potential (TMP) dynamics throughout the 3D myocardium using body surface potential recordings, it is necessary to combine prior physiological models and patient's data with regard to their respective uncertainties. To fulfill model-data melding for this large-scale and high-dimensional system, data assimilation with proper computational reduction is needed for computational feasibility and efficiency. In this paper, we develop a reduced-rank square root TMP estimation algorithm, using dominant components of estimation uncertainties to guide a more efficient model-data coupling in the square root structure. The SVD-based reduced-rank error covariance is used to represent and track the dominant estimation errors, and unified into an integrated square root filtering framework. Phantom experiments demonstrate the ability of this framework to bring substantial computational reduction at slight expense of degraded estimation accuracy. It therefore improves the efficiency and applicability of the volumetric myocardial TMP imaging in practice.

To recover physiologically meaningful cardiac deformation from medical images, realistic physiological models are essential to constrain the recovery process, and a statistical filtering framework is required to couple the models and images according to their respective uncertainties. As realistic cardiac models are usually nonlinear, existing cardiac deformation recovery frameworks either ignore the statistical filtering part, or linearize the model and apply linear filtering techniques such as the extended Kalman filtering. This reduces the physiological plausibility and statistical optimality of the recovery results. In this paper, we propose a nonlinear cardiac deformation recovery framework with unscented Kalman filtering which preserves the intact system nonlinearity. Experiments were done on both synthetic data and magnetic resonance images to show the benefits and clinical relevance of our framework.

Body surface potential (BSP) has been used as the single data source in inverse electrocardiography (IECG). The lack of volumetric spatial resolutions in BSP, however, hinders the noninvasive mapping of volumetric cardiac transmembrane potentials (TMPs). Tomographic image sequence, which contains temporally sparse but spatially dense cardiac kinematic measures, becomes ideal dynamic complements to BSPs through cardiac electromechanical (EM) coupling. In this paper, we present a model-constrained Bayesian framework to integrate tomographic image sequence and BSP maps (BSPM) for volumetric cardiac TMP mapping. A priori physiological knowledge is incorporated via stochastic modeling of the cardiac electrophysiological system with unknown systematic errors. With this system as a platform for data integration, adaptive data assimilation is used to estimate patient specific TMPs from BSPMs under the guidance of tomographic images. In this way, complementary images are integrated in accord with their respective merits and limitations. Phantom and real data experiments exhibit notable improvements and practicability of the presented framework.

Proper spatial and temporal constraints are essential for image-based motion recovery of deforming objects. Since biological organs, such as the heart, are typically composed of fibrous tissues of anisotropic nature, one must adopt realistic spatial models, in addition to those important considerations for temporal modeling, in order to properly regularize the object behavior for kinematics recovery. We present a biomechanically constrained state space analysis framework for the multiframe estimation of the heart motion and deformation. While the anisotropic physical constraints enforce spatial regulations on the myocardial behavior and spatial filtering of the image data measurements, statistical filtering techniques impose temporal constraints to incorporate multiframe information. Implemented within a mesh-free particle representation and computation framework, excellent experimental results are achieved for both synthetic data with known ground truth and canine magnetic resonance image sequences with known clinical gold standard.