Atrial fibrillation, characterized by high prevalence and poor prognosis, presents a significant global health burden. Accurate segmentation and measurement of left ventricular and left atrial appendage morphology and function are essential for reliable risk assessment. However, these tasks are hindered by ambiguous boundaries, complex cardiac motion, and sparse annotations. To address these challenges, we propose a Keypoint-Guided Medical Video Segmentation Model with Spatiotemporal Feature Fusion (KG-STS). First, we propose a shape-constrained point encoder that explicitly encodes boundary points to improve the representation of ambiguous boundaries. Next, we introduce a motion-aware alignment module that models cardiac motion by forming coherent motion information across frames. Building on these two modules, we develop a keypoint-guided spatiotemporal feature fusion module that integrates spatial boundary representations with temporal motion cues to enhance decoding features under sparse annotations, enabling temporally consistent segmentation and supporting morphological measurement. We evaluate the segmentation and measurement performance of our method on a self-constructed multi-view transesophageal echocardiography dataset and two publicly available transthoracic echocardiography datasets. The results demonstrate that KG-STS achieves superior temporal consistency in segmentation and higher accuracy in morphological measurements compared to competing methods.

Single-template-based brain functional network analysis methods can provide limited functional connectivity information, which constrains the performance of brain disease diagnosis. Previous works have explored multi-template functional network analysis but failed to integrate the high-order correlation information within templates and the complementary information between templates into a unified relationship strength between nodes, and we extract the high-order correlation information within each template through hypergraph convolution. Secondly, for the analysis of functional connectivity between templates, we propose a cross-template Transformer to capture long-range dependencies between templates. A cross-template mask is applied to focus the model’s attention on important connections between templates, thereby enhancing model robustness. Finally, we progressively fuse the high-order information captured within templates with the global information across templates for downstream classification tasks. The proposed method has been validated on the public ABIDE dataset, and it outperforms existing methods in the ASD diagnosis task.

Elastography ultrasound imaging is increasingly important in the diagnosis of thyroid cancer and other diseases, but its reliance on specialized equipment and techniques limits widespread adoption. This paper proposes a novel multimodal ultrasound diagnostic pipeline that expands the application of elastography ultrasound by translating B-ultrasound (BUS) images into elastography images (EUS). Additionally, to address the limitations of existing image-to-image translation methods, which struggle to effectively model inter-sample variations and accurately capture regional-scale structural consistency, we propose a BUS-to-EUS translation method based on hierarchical structural consistency. By incorporating domain-level, sample-level, patch-level, and pixel-level constraints, our approach guides the model in learning a more precise mapping from BUS to EUS, thereby enhancing diagnostic accuracy. Experimental results demonstrate that the proposed method significantly improves the accuracy of BUS-to-EUS translation on the MTUSI dataset and that the generated elastography images enhance nodule diagnostic accuracy compared to solely using BUS images on the STUSI and the BUSI datasets. This advancement highlights the potential for broader application of elastography in clinical practice.

Directionally sensitive radiomic features including the histogram of oriented gradient (HOG) have been shown to provide objective and quantitative measures for predicting disease outcomes in multiple cancers. However, radiomic features are sensitive to imaging variabilities including acquisition differences, imaging artifacts and noise, making them impractical for using in the clinic to inform patient care. We treat the problem of extracting robust local directionality features by mapping via optimal transport a given local image patch to an iso-intense patch of its mean. We decompose the transport map into sub-work costs each transporting in different directions. To test our approach, we evaluated the ability of the proposed approach to quantify tumor heterogeneity from magnetic resonance imaging (MRI) scans of brain glioblastoma multiforme, computed tomography (CT) scans of head and neck squamous cell carcinoma as well as longitudinal CT scans in lung cancer patients treated with immunotherapy. By considering the entropy difference of the extracted local directionality within tumor regions, we found that patients with higher entropy in their images, had significantly worse overall survival for all three datasets, which indicates that tumors that have images exhibiting flows in many directions may be more malignant. This may seem to reflect high tumor histologic grade or disorganization. Furthermore, by comparing the changes in entropy longitudinally using two imaging time points, we found patients with reduction in entropy from baseline CT are associated with longer overall survival (hazard ratio = 1.95, 95% confidence interval of 1.4-2.8, p = 1.65e-5). The proposed method provides a robust, training free approach to quantify the local directionality contained in images.

Accurate and robust segmentation of lung cancers from CT, even those located close to mediastinum, is needed to more accurately plan and deliver radiotherapy and to measure treatment response. Therefore, we developed a new cross-modality educed distillation (CMEDL) approach, using unpaired CT and MRI scans, whereby an informative teacher MRI network guides a student CT network to extract features that signal the difference between foreground and background. Our contribution eliminates two requirements of distillation methods: (i) paired image sets by using an image to image (I2I) translation and (ii) pre-training of the teacher network with a large training set by using concurrent training of all networks. Our framework uses an end-to-end trained unpaired I2I translation, teacher, and student segmentation networks. Architectural flexibility of our framework is demonstrated using 3 segmentation and 2 I2I networks. Networks were trained with 377 CT and 82 T2w MRI from different sets of patients, with independent validation (N = 209 tumors) and testing (N = 609 tumors) datasets. Network design, methods to combine MRI with CT information, distillation learning under informative (MRI to CT), weak (CT to MRI) and equal teacher (MRI to MRI), and ablation tests were performed. Accuracy was measured using Dice similarity (DSC), surface Dice (sDSC), and Hausdorff distance at the$95^{\textit {th}}$percentile (HD95). The CMEDL approach was significantly (p < 0.001) more accurate (DSC of 0.77 vs. 0.73) than non-CMEDL methods with an informative teacher for CT lung tumor, with a weak teacher (DSC of 0.84 vs. 0.81) for MRI lung tumor, and with equal teacher (DSC of 0.90 vs. 0.88) for MRI multi-organ segmentation. CMEDL also reduced inter-rater lung tumor segmentation variabilities.

Image-guided adaptive lung radiotherapy requires accurate tumor and organs segmentation from during treatment cone-beam CT (CBCT) images. Thoracic CBCTs are hard to segment because of low soft-tissue contrast, imaging artifacts, respiratory motion, and large treatment induced intra-thoracic anatomic changes. Hence, we developed a novel Patient-specific Anatomic Context and Shape prior or PACS-aware 3D recurrent registration-segmentation network for longitudinal thoracic CBCT segmentation. Segmentation and registration networks were concurrently trained in an end-to-end framework and implemented with convolutional long-short term memory models. The registration network was trained in an unsupervised manner using pairs of planning CT (pCT) and CBCT images and produced a progressively deformed sequence of images. The segmentation network was optimized in a one-shot setting by combining progressively deformed pCT (anatomic context) and pCT delineations (shape context) with CBCT images. Our method, one-shot PACS was significantly more accurate (p <0.001) for tumor (DSC of 0.83 ± 0.08, surface DSC [sDSC] of 0.97 ± 0.06, and Hausdorff distance at 95th percentile [HD95] of 3.97±3.02mm) and the esophagus (DSC of 0.78 ± 0.13, sDSC of 0.90±0.14, HD95 of 3.22±2.02) segmentation than multiple methods. Ablation tests and comparative experiments were also done.

We developed a new joint probabilistic segmentation and image distribution matching generative adversarial network (PSIGAN) for unsupervised domain adaptation (UDA) and multi-organ segmentation from magnetic resonance (MRI) images. Our UDA approach models the co-dependency between images and their segmentation as a joint probability distribution using a new structure discriminator. The structure discriminator computes structure of interest focused adversarial loss by combining the generated pseudo MRI with probabilistic segmentations produced by a simultaneously trained segmentation sub-network. The segmentation sub-network is trained using the pseudo MRI produced by the generator sub-network. This leads to a cyclical optimization of both the generator and segmentation sub-networks that are jointly trained as part of an end-to-end network. Extensive experiments and comparisons against multiple state-of-the-art methods were done on four different MRI sequences totalling 257 scans for generating multi-organ and tumor segmentation. The experiments included, (a) 20 T1-weighted (T1w) in-phase mdixon and (b) 20 T2-weighted (T2w) abdominal MRI for segmenting liver, spleen, left and right kidneys, (c) 162 T2-weighted fat suppressed head and neck MRI (T2wFS) for parotid gland segmentation, and (d) 75 T2w MRI for lung tumor segmentation. Our method achieved an overall average DSC of 0.87 on T1w and 0.90 on T2w for the abdominal organs, 0.82 on T2wFS for the parotid glands, and 0.77 on T2w MRI for lung tumors.

Volumetric lung tumor segmentation and accurate longitudinal tracking of tumor volume changes from computed tomography images are essential for monitoring tumor response to therapy. Hence, we developed two multiple resolution residually connected network (MRRN) formulations called incremental-MRRN and dense-MRRN. Our networks simultaneously combine features across multiple image resolution and feature levels through residual connections to detect and segment the lung tumors. We evaluated our method on a total of 1210 non-small cell (NSCLC) lung tumors and nodules from three data sets consisting of 377 tumors from the open-source Cancer Imaging Archive (TCIA), 304 advanced stage NSCLC treated with anti- PD-1 checkpoint immunotherapy from internal institution MSKCC data set, and 529 lung nodules from the Lung Image Database Consortium (LIDC). The algorithm was trained using 377 tumors from the TCIA data set and validated on the MSKCC and tested on LIDC data sets. The segmentation accuracy compared to expert delineations was evaluated by computing the dice similarity coefficient, Hausdorff distances, sensitivity, and precision metrics. Our best performing incremental-MRRN method produced the highest DSC of 0.74 ± 0.13 for TCIA, 0.75±0.12 for MSKCC, and 0.68±0.23 for the LIDC data sets. There was no significant difference in the estimations of volumetric tumor changes computed using the incremental-MRRN method compared with the expert segmentation. In summary, we have developed a multi-scale CNN approach for volumetrically segmenting lung tumors which enables accurate, automated identification of and serial measurement of tumor volumes in the lung.

Crowd counting estimation is an extremely challenging task due to various crowded scenarios. In this paper, we present a deep learning framework for crowd counting from a single static image with different number of people and arbitrary perspective. In the design of convolutional neural network structure, we employ the VGG16 model but drop the fully connected layers. Meanwhile, high-level features are combined with low-level features through laterally connected feature pyramid network by element-wise addition to ensure higher resolution and more context information. Extensive experiments are conducted on ShanghaiTech and UCF_CC_50 datasets. The results show that our model achieves the lowest mean absolute error (MAE) and comparable mean square error (MSE), and outperforms the current state-of-the-art methods.

In this paper, we propose a novel flexible framework for hyperspectral image (HSI) classification using multi-view spectral-spatial feature extracted by nuclear norm based 2D PCA. We first use the multihyphonthesis (MH) prediction method based on ridge regression to generate the 3D spatial-feature array from the HSI. Then, we apply the nuclear norm based 2D PCA to multi-view slices (the image with the spatial width and spectral dimension or with the spatial height and spectral dimension) of the former feature array, which can provide a structured spatial-spectral characterization for the reconstruction error slice and further extract the spatial-spectral feature. Finally, the 3D spatial-spectral feature array is used to represent the HSI for classification by extreme learning machine (ELM) based on Radial Basis Function (RBF) kernal. Finally, majority voting procedure is used to further improve the classification accuracy. The efficiency of the proposed method is demonstrated by experimental results with real hyperspectral dataset.