As a critical statistical technique in oncology, survival prediction is used to estimate the probability of survival or time-to-event outcomes. Identifying survival-related factors from pathology and genomic data is a key approach for analyzing survival outcomes. However, current methods face several challenges, such as the suboptimal adaptation of pre-trained vision foundation models to specific tasks during feature extraction from whole slide images (WSIs), and the fact that many pathology-based models fail to integrate repetitive gene expression information during pre-training. In this study, we propose a plug-and-play multiple instance learning (MIL)-based foundation model tuning strategy to adapt vision foundation models for downstream tasks and incorporate knowledge from genomic data. Specifically, we introduce Task-specific Instance Selection, which utilizes zero-shot learning to efficiently select task-relevant WSI regions, improving tuning efficiency and reducing interference from irrelevant tissue areas. Additionally, we develop a multi-model prompt token for model fine-tuning, which integrates genetic information into the prompt-tuning process and transfers new modality information to pre-trained vision foundation models. To further enhance the model's ability to learn genetic information during fine-tuning, we introduce a Gene Distribution Aware Task as an auxiliary task to the traditional survival task. This auxiliary task helps the model better perceive multimodal information. Extensive experimental results on three public TCGA datasets demonstrate that our model outperforms all previous MIL-based methodologies and fine-tuning approaches in terms of performance.

Automated diagnostic systems (ADS) have shown significant potential in the early detection of polyps during endoscopic examinations, thereby reducing the incidence of colorectal cancer. However, due to high annotation costs and strict privacy concerns, acquiring high-quality endoscopic images poses a considerable challenge in the development of ADS. Despite recent advancements in generating synthetic images for dataset expansion, existing endoscopic image generation algorithms failed to accurately generate the details of polyp boundary regions and typically required medical priors to specify plausible locations and shapes of polyps, which limited the realism and diversity of the generated images. To address these limitations, we present Polyp-Gen, the first full-automatic diffusion-based endoscopic image generation framework. Specifically, we devise a spatial-aware diffusion training scheme with a lesion-guided loss to enhance the structural context of polyp boundary regions. Moreover, to capture medical priors for the localization of potential polyp areas, we introduce a hierarchical retrieval-based sampling strategy to match similar fine-grained spatial features. In this way, our Polyp-Gen can generate realistic and diverse endoscopic images for building reliable ADS. Extensive experiments demonstrate the state-of-the-art generation quality, and the synthetic images can improve the downstream polyp detection task. Additionally, our Polyp-Gen has shown remarkable zeroshot generalizability on other datasets. The source code is available at https://github.com/CUHK-AIM-Group/Polyp-Gen.

Prognostic assessment remains a critical challenge in medical research, often limited by the lack of well-labeled data. In this work, we introduce ContraSurv, a weakly-supervised learning framework based on contrastive learning, designed to enhance prognostic predictions in 3D medical images. ContraSurv utilizes both the self-supervised information inherent in unlabeled data and the weakly-supervised cues present in censored data, refining its capacity to extract prognostic representations. For this purpose, we establish a Vision Transformer architecture optimized for our medical image datasets and introduce novel methodologies for both self-supervised and supervised contrastive learning for prognostic assessment. Additionally, we propose a specialized supervised contrastive loss function and introduce SurvMix, a novel data augmentation technique for survival analysis. Evaluations were conducted across three cancer types and two imaging modalities on three real-world datasets. The results confirmed the enhanced performance of ContraSurv over competing methods, particularly in data with a high censoring rate.

Providing precise and comprehensive diagnostic information to clinicians is crucial for improving the treatment and prognosis of nasopharyngeal carcinoma. Multi-modal foundation models, which can integrate data from various sources, have the potential to significantly enhance clinical assistance. However, several challenges remain: (1) the lack of large-scale visual-language datasets for nasopharyngeal carcinoma; (2) the inability of existing pre-training and fine-tuning methods to capture the hierarchical features required for complex clinical tasks; (3) current foundation models having limited visual perception due to inadequate integration of multi-modal information. While curriculum learning can improve a model's ability to handle multiple tasks through systematic knowledge accumulation, it still lacks consideration for hierarchical features and their dependencies, affecting knowledge gains. To address these issues, we propose the Hierarchical Feature Fusion Curriculum Learning method, which consists of three stages: visual knowledge learning, coarse-grained alignment, and fine-grained fusion. First, we introduce the Hybrid Contrastive Masked Autoencoder to pre-train visual encoders on 755K multi-modal images of nasopharyngeal carcinoma CT, MRI, and endoscopy to fully extract deep visual information. Then, we construct a 65K visual instruction fine-tuning dataset based on open-source data and clinician diagnostic reports, achieving coarse-grained alignment with visual information in a large language model. Finally, we design a Mixture of Experts Cross Attention structure for deep fine-grained fusion of global multi-modal information. Our model outperforms previously developed specialized models in all key clinical tasks for nasopharyngeal carcinoma, including diagnosis, report generation, tumor segmentation, and prognosis.

A core challenge in survival analysis is to model the distribution of time-to-event data, where the event of interest may be a death, failure, or occurrence of a specific event. Previous studies have showed that ranking and maximum likelihood estimation loss functions are widely-used learning approaches for survival analysis. However, ranking loss only focus on the ranking of survival time and does not consider potential effect of samples’ exact survival time values. Furthermore, the maximum likelihood estimation is unbounded and easily subject to outliers (e.g., censored data), which may cause poor performance of modeling. To handle the complexities of learning process and exploit valuable survival time values, we propose a time-adaptive coordinate loss function, TripleSurv, to achieve adaptive adjustments by introducing the differences in the survival time between sample pairs into the ranking, which can encourage the model to quantitatively rank relative risk of pairs, ultimately enhancing the accuracy of predictions. Most importantly, the TripleSurv is proficient in quantifying the relative risk between samples by ranking ordering of pairs, and consider the time interval as a trade-off to calibrate the robustness of model over sample distribution. Our TripleSurv is evaluated on three real-world survival datasets and a public synthetic dataset. The results show that our method outperforms the state-of-the-art methods and exhibits good model performance and robustness on modeling various sophisticated data distributions with different censor rates.

Objective: Radiomics, an emerging tool for medical image analysis, is potential towards precisely characterizing gastric cancer (GC). Whether using one-slice 2D annotation or whole-volume 3D annotation remains a long-time debate, especially for heterogeneous GC. We comprehensively compared 2D and 3D radiomic features' representation and discrimination capacity regarding GC, via three tasks (TLNM, lymph node metastasis' prediction; TLVI, lymphovascular invasion's prediction; TpT, pT4 or other pT stages' classification). Methods: Four-center 539 GC patients were retrospectively enrolled and divided into the training and validation cohorts. From 2D or 3D regions of interest (ROIs) annotated by radiologists, radiomic features were extracted respectively. Feature selection and model construction procedures were customed for each combination of two modalities (2D or 3D) and three tasks. Subsequently, six machine learning models (ModelLNM2D, ModelLNM3D; ModelLVI2D, ModelLVI3Ds ModelpT2D,s ModelpT3D) were derived and evaluated to reflect modalities' performances in characterizing GC. Furthermore, we performed an auxiliary experiment to assess modalities' performances when resampling spacing different. Results: Regarding three tasks, the yielded areas under the curve (AUCs) were: ModelLNM2D's 0.712 (95% confidence interval, 0.613-0.811), ModelLNM3D's 0.680 (0.584-0.775); ModelLVI2D's 0.677 (0.595-0.761), ModelLVI3D's 0.615 (0.528-0.703); ModelpT2D's 0.840 (0.779-0.901), ModelpT3D's 0.813 (0.747-0.879). Moreover, the auxiliary experiment indicated that Models2Dare statistically advantageous than Models3Dwith different resampling spacings. Conclusion: Models constructed with 2D radiomic features revealed comparable performances with those constructed with 3D features in characterizing GC. Significance: Our work indicated that time-saving 2D annotation would be the better choice in GC, and provided a related reference to further radiomics-based researches.

OBJECTIVE: Coronavirus disease 2019 (COVID-19) has caused considerable morbidity and mortality, especially in patients with underlying health conditions. A precise prognostic tool to identify poor outcomes among such cases is desperately needed. METHODS: Total 400 COVID-19 patients with underlying health conditions were retrospectively recruited from 4 centers, including 54 dead cases (labeled as poor outcomes) and 346 patients discharged or hospitalized for at least 7 days since initial CT scan. Patients were allocated to a training set (n = 271), a test set (n = 68), and an external test set (n = 61). We proposed an initial CT-derived hybrid model by combining a 3D-ResNet10 based deep learning model and a quantitative 3D radiomics model to predict the probability of COVID-19 patients reaching poor outcome. The model performance was assessed by area under the receiver operating characteristic curve (AUC), survival analysis, and subgroup analysis. RESULTS: The hybrid model achieved AUCs of 0.876 (95% confidence interval: 0.752-0.999) and 0.864 (0.766-0.962) in test and external test sets, outperforming other models. The survival analysis verified the hybrid model as a significant risk factor for mortality (hazard ratio, 2.049 [1.462-2.871], P < 0.001) that could well stratify patients into high-risk and low-risk of reaching poor outcomes (P < 0.001). CONCLUSION: The hybrid model that combined deep learning and radiomics could accurately identify poor outcomes in COVID-19 patients with underlying health conditions from initial CT scans. The great risk stratification ability could help alert risk of death and allow for timely surveillance plans.

Gastric cancer (GC) is the third leading cause of cancer-associated deaths globally. Accurate risk prediction of the overall survival (OS) for GC patients shows significant prognostic value, which helps identify and classify patients into different risk groups to benefit from personalized treatment. Many methods based on machine learning algorithms have been widely explored to predict the risk of OS. However, the accuracy of risk prediction has been limited and remains a challenge with existing methods. Few studies have proposed a framework and pay attention to the low-level and high-level features separately for the risk prediction of OS based on computed tomography images of GC patients. To achieve high accuracy, we propose a multi-focus fusion convolutional neural network. The network focuses on low-level and high-level features, where a subnet to focus on lower-level features and the other enhanced subnet with lateral connection to focus on higher-level semantic features. Three independent datasets of 640 GC patients are used to assess our method. Our proposed network is evaluated by metrics of the concordance index and hazard ratio. Our network outperforms state-of-the-art methods with the highest concordance index and hazard ratio in independent validation and test sets. Our results prove that our architecture can unify the separate low-level and high-level features into a single framework, and can be a powerful method for accurate risk prediction of OS.

OBJECTIVE: The coronavirus disease 2019 (COVID-19) is rapidly spreading inside China and internationally. We aimed to construct a model integrating information from radiomics and deep learning (DL) features to discriminate critical cases from severe cases of COVID-19 using computed tomography (CT) images. METHODS: We retrospectively enrolled 217 patients from three centers in China, including 82 patients with severe disease and 135 with critical disease. Patients were randomly divided into a training cohort (n = 174) and a test cohort (n = 43). We extracted 102 3-dimensional radiomic features from automatically segmented lung volume and selected the significant features. We also developed a 3-dimensional DL network based on center-cropped slices. Using multivariable logistic regression, we then created a merged model based on significant radiomic features and DL scores. We employed the area under the receiver operating characteristic curve (AUC) to evaluate the model's performance. We then conducted cross validation, stratified analysis, survival analysis, and decision curve analysis to evaluate the robustness of our method. RESULTS: The merged model can distinguish critical patients with AUCs of 0.909 (95% confidence interval [CI]: 0.859-0.952) and 0.861 (95% CI: 0.753-0.968) in the training and test cohorts, respectively. Stratified analysis indicated that our model was not affected by sex, age, or chronic disease. Moreover, the results of the merged model showed a strong correlation with patient outcomes. SIGNIFICANCE: A model combining radiomic and DL features of the lung could help distinguish critical cases from severe cases of COVID-19.

Since its outbreak in December 2019, the persistent coronavirus disease (COVID-19) became a global health emergency. It is imperative to develop a prognostic tool to identify high-risk patients and assist in the formulation of treatment plans. We retrospectively collected 366 severe or critical COVID-19 patients from four centers, including 70 patients who died within 14 days (labeled as high-risk patients) since their initial CT scan and 296 who survived more than 14 days or were cured (labeled as low-risk patients). We developed a 3D densely connected convolutional neural network (termed De-COVID19-Net) to predict the probability of COVID-19 patients belonging to the high-risk or low-risk group, combining CT and clinical information. The area under the curve (AUC) and other evaluation techniques were used to assess our model. The De-COVID19-Net yielded an AUC of 0.952 (95% confidence interval, 0.928-0.977) on the training set and 0.943 (0.904-0.981) on the test set. The stratified analyses indicated that our model's performance is independent of age, sex, and with/without chronic diseases. The Kaplan-Meier analysis revealed that our model could significantly categorize patients into high-risk and low-risk groups (p < 0.001). In conclusion, De-COVID19-Net can non-invasively predict whether a patient will die shortly based on the patient's initial CT scan with an impressive performance, which indicated that it could be used as a potential prognosis tool to alert high-risk patients and intervene in advance.

Many complex systems are composed of coupled networks through different layers, where each layer represents one of many possible types of interactions. A fundamental question is how to extract communities in multi-layer networks. The current algorithms either collapses multi-layer networks into a single-layer network or extends the algorithms for single-layer networks by using consensus clustering. However, these approaches have been criticized for ignoring the connection among various layers, thereby resulting in low accuracy. To attack this problem, a quantitative function (multi-layer modularity density) is proposed for community detection in multi-layer networks. Afterward, we prove that the trace optimization of multi-layer modularity density is equivalent to the objective functions of algorithms, such as kernel$K$-means, nonnegative matrix factorization (NMF), spectral clustering and multi-view clustering, for multi-layer networks, which serves as the theoretical foundation for designing algorithms for community detection. Furthermore, aSemi-SupervisedjointNonnegativeMatrixFactorization algorithm (S2-jNMF) is developed by simultaneously factorizing matrices that are associated with multi-layer networks. Unlike the traditional semi-supervised algorithms, the partial supervision is integrated into the objective of the S2-jNMF algorithm. Finally, through extensive experiments on both artificial and real world networks, we demonstrate that the proposed method outperforms the state-of-the-art approaches for community detection in multi-layer networks.

Accurate lung nodule segmentation from computed tomography (CT) images is of great importance for image-driven lung cancer analysis. However, the heterogeneity of lung nodules and the presence of similar visual characteristics between nodules and their surroundings make it difficult for robust nodule segmentation. In this study, we propose a data-driven model, termed the Central Focused Convolutional Neural Networks (CF-CNN), to segment lung nodules from heterogeneous CT images. Our approach combines two key insights: 1) the proposed model captures a diverse set of nodule-sensitive features from both 3-D and 2-D CT images simultaneously; 2) when classifying an image voxel, the effects of its neighbor voxels can vary according to their spatial locations. We describe this phenomenon by proposing a novel central pooling layer retaining much information on voxel patch center, followed by a multi-scale patch learning strategy. Moreover, we design a weighted sampling to facilitate the model training, where training samples are selected according to their degree of segmentation difficulty. The proposed method has been extensively evaluated on the public LIDC dataset including 893 nodules and an independent dataset with 74 nodules from Guangdong General Hospital (GDGH). We showed that CF-CNN achieved superior segmentation performance with average dice scores of 82.15% and 80.02% for the two datasets respectively. Moreover, we compared our results with the inter-radiologists consistency on LIDC dataset, showing a difference in average dice score of only 1.98%.

Discovering evolving communities in dynamic networks is essential to important applications such as analysis for dynamic web content and disease progression. Evolutionary clustering uses the temporal smoothness framework that simultaneously maximizes the clustering accuracy at the current time step and minimizes the clustering drift between two successive time steps. In this paper, we propose two evolutionary nonnegative matrix factorization (ENMF) frameworks for detecting dynamic communities. To address the theoretical relationship among evolutionary clustering algorithms, we first prove the equivalence relationship between ENMF and optimization of evolutionary modularity density. Then, we extend the theory by proving the equivalence between evolutionary spectral clustering and ENMF, which serves as the theoretical foundation for hybrid algorithms. Based on the equivalence, we propose a semi-supervised ENMF (sE-NMF) by incorporating a priori information into ENMF. Unlike the traditional semi-supervised algorithms, a priori information is integrated into the objective function of the algorithm. The main advantage of the proposed algorithm is to escape the local optimal solution without increasing time complexity. The experimental results over a number of artificial and real world dynamic networks illustrate that the proposed method is not only more accurate but also more robust than the state-of-the-art approaches.

This paper introduces a new eddy detection method based on object segmentation strategies. It incorporates the advantages of the Okubo-Weiss (OW) method, and improves the algorithm robustness by using three different segmentation methods. First, by intersecting the OW mask with the positive and negative SLA masks, two separate initial eddy candidate segment masks for anticyclonic and cyclonic eddies are produced. Then the number of extrema inside the individual segments is calculated. If the number is two/greater than two, a histogram threshold/watershed method is applied to further divide the segments into subsegments. If the number is one, the segment is tested by predefined eddy criteria. If the criteria are not fulfilled, the eddy boundary is repeatedly shrunk by one pixel inward until the segment meets all the criteria. In this case it is saved as an eddy. A segment is discarded if the number of extrema is zero. The eddy detection results of this method, of the OW method and of a geometric eddy detection method are displayed and analyzed statistically. The seasonal cycle of eddy number from the three eddy detection methods is discussed in comparison to the EKE.

Two cases of mode-2 internal solitary waves (ISWs) induced by an anticyclonic eddy (AE) were clearly present in two synthetic aperture radar images acquired in the South China Sea (SCS) in April 2001. Similar ISWs were repeatedly observed in the same area one month later in May, but in that case, the ISW patterns indicated that they were regular mode-1 ISWs. To confirm that those ISWs in April are of mode-2 type, we analyze the in situ and other remote sensing data and propose two possible mode-2 ISW generation processes: 1) an eddy-induced change of the water stratification, which results in favorable hydrographic conditions for internal wave generation, and 2) the resonance between mode-1 internal tides and AE excites mode-2 internal tides, and the mode-2 internal tides disintegrate into mode-2 ISWs. The sea level anomaly and sea surface temperature data in April 2001 show that an AE existed in this area in April. Argo profile data within AEs from 1990 to 2014 in the SCS are used to show how an AE affects the vertical water properties and creates favorable conditions for mode-2 ISWs, and these conditions did exist in the study area in April 2001. The observed mode-2 ISW cases provide the first evidence of the existence of eddy-induced mode-2 ISWs in the SCS.

Mode 2 internal wave (IW) signatures are observed by the RADARSAT-1 synthetic aperture radar (SAR) in the Northeast South China Sea on April 07, 2001. In this study, we prove that the conditions of the study area are favorable for the generation and propagation of mode-2 waves to appear in SAR imagery. The generation mechanism of this mode 2 IW is mode-1 IW evolves into mode-2 convex IW packets when shoaling.

Finding the center of rotation is an essential step for accurate three-dimensional reconstruction in optical projection tomography (OPT). Unfortunately current methods are not convenient since they require either prior scanning of a reference phantom, small structures of high intensity existing in the specimen, or active participation during the centering procedure. To solve these problems this paper proposes a fast and automatic center of rotation search method making use of parallel programming in graphics processing units (GPUs). Our method is based on a two step search approach making use only of those sections of the image with high signal to noise ratio. We have tested this method both in non-scattering ex vivo samples and in in vivo specimens with a considerable contribution of scattering such as Drosophila melanogaster pupae, recovering in all cases the center of rotation with a precision 1/4 pixel or less.

In in vivo optical projection tomography (OPT), object motion will significantly reduce the quality and resolution of the reconstructed image. Based on the well-known Helgason-Ludwig consistency condition (HLCC), we propose a novel method for motion correction in OPT under parallel beam illumination. The method estimates object motion from projection data directly and does not require any other additional information, which results in a straightforward implementation. We decompose object movement into translation and rotation, and discuss how to correct for both translation and general motion simultaneously. Since finding the center of rotation accurately is critical in OPT, we also point out that the system's geometrical offset can be considered as object translation and therefore also calibrated through the translation estimation method. In order to verify the algorithm effectiveness, both simulated and in vivo OPT experiments are performed. Our results demonstrate that the proposed approach is capable of decreasing movement artifacts significantly thus providing high quality reconstructed images in the presence of object motion.

Visualized freehand 3-D ultrasound reconstruction offers to image incremental reconstruction during acquisition and guide users to scan interactively for high-quality volumes. We originally used the graphics processing unit (GPU) to develop a visualized reconstruction algorithm that achieves real-time level. Each newly acquired image was transferred to the memory of the GPU and inserted into the reconstruction volume on the GPU. The partially reconstructed volume was then rendered using GPU-based incremental ray casting. After visualized reconstruction, hole-filling was performed on the GPU to fill remaining empty voxels in the reconstruction volume. We examine the real-time nature of the algorithm using in vitro and in vivo datasets. The algorithm can image incremental reconstruction at speed of 26-58 frames/s and complete 3-D imaging in the acquisition time for the conventional freehand 3-D ultrasound.

Katsevich reconstruction algorithm represents a breakthrough for helical cone-beam computed tomography (CT) reconstruction, because it is the first exact cone-beam reconstruction algorithm of filtered backprojection (FBP) type with 1-D shift-invariant filtering. Although FBP-type reconstruction algorithm is effective, 3-D CT reconstruction is time-consuming, and the accelerations of Katsevich algorithm on CPU or cluster have been widely studied. In this paper, Katsevich algorithm is accelerated by using graphics processing unit, including flat-detector and curved-detector geometry in the case of helical orbit. An overscan formula is derived, which helps to avoid unnecessary overscan in practical CT scanning. Based on the overscan formula, a volume-blocking method in device memory is proposed. One advantage of the blocking method is that it can reconstruct large volume with high speed.

In this paper we present single antenna relay channel measurements conducted in an urban environment at 2.35 GHz. Three types of links, i.e. base station to mobile station (BS-MS), relay station to mobile station (RS-MS) and base station to relay station (BS-RS), were measured at two sites. Our investigation focuses on the characteristics of large scale parameters (LSP) of the RS-MS link, which is characterized by the low antenna height at RS and short RS-MS distance. Measurement results show that the current BS-MS path loss model cannot perfectly predict the propagation loss of RS-MS link. The distance dependent property and the distribution of Ricean K-factor are analyzed. The RS-MS link is found to exhibit lower Ricean K-factor compared to the BS-MS link. We also investigate the capacity gain provided by the relay link when the MS is located in the shadowing area of BS. Furthermore, it is observed that the capacity gap between decode-and-forward (DF) and the fixed gain amplify-and-forward (AF) relay schemes vanishes, provided the large K-factor and high SNR of BS-RS link. This gap becomes larger as the K-factor of BS-RS link decreases.

In this paper, we propose a novel spatial autocorrelation model of the shadow fading process in urban macro environments. The proposed model is based on the empirical results obtained from extensive wideband radio channel measurement campaigns at 2.35 GHz in an urban area of a typical medium-sized Chinese city. The shadow fading component was extracted assuming a single-slope log-distance path loss model. The consistency with the level crossing theory of Gaussian processes is achieved by an implicit constraint on the parameters of the model. The proposed model gives a better fit to the empirical results in individual measurement routes than the widely reported exponential and double exponential models. An heuristic explanation of the proposed autocorrelation property is also presented.

The wideband channel measurements at 580 MHz, 2.35 GHz and 4.90 GHz have been performed in the urban micro- and macrocell scenarios in the cities of China with multiple-input multiple-output (MIMO) channel sounder. The measured cases include line-of-sight (LOS) and non-line-of-sight (NLOS) propagation. Statistical results and comparative analysis for both scenarios are presented in this paper, including path loss (PL), root mean square (rms) delay spread (DS) and maximum excess delay (maxED), and angular spread (AS). In NLOS case, the frequency dependent factor (FDF) is observed as 32.1 by fitting the PL model of 2.35 GHz and 4.90 GHz. Moreover, a larger rms DS in urban microcell and AS at both base station (BS) and mobile subscriber (MS) are found for the denser and higher buildings in the cities of China. As the frequency varying from 580 MHz to 4.90 GHz, the median rms DS is decreased from 330 ns to 130 ns for NLOS case.