Neoadjuvant chemotherapy (NAC) is the primary method to reduce the burden of tumor and metastasis; in the treatment of breast cancer, it may provide additional opportunities for breast-conserving surgery. Preoperative assessment of pathological complete response (PCR) to NAC is important for developing individualized treatment approaches and predicting patient prognosis. Compared to magnetic resonance imaging (MRI) and mammography, ultrasonography (US) has the advantages of simplicity, flexibility, and real-time imaging. Moreover, it does not require radiation and can provide multi-time acquisition of the tumor during NAC treatment. Recently, deep learning radiomics models based on multi-time-point US images for the prediction of NAC effectiveness have been proposed. To further improve the prediction performance, we carefully designed four supporting modules for our proposed dual-input transformer (DiT): isolated tokens-to-token patch embedding module, shared position embedding, time embedding, and weighted average pooling feature representation modules. The design of each module considers the characteristics of the US images at multiple time points. We validated our model on our retrospective US dataset composed of 484 cases from two centers whose consistency is not sufficiently high. Patients were allocated to training (n = 297), validation (n = 99), and external test (n = 88) sets. The results show that our model can achieve better performance than the Siamese CNN and the standard tokens-to-token vision transformer without using multi-time-point images. The ablation study also proved the effectiveness of each module designed for DiT.

Identifying squamous cell carcinoma and adenocarcinoma subtypes of metastatic cervical lymphadenopathy (CLA) is critical for localizing the primary lesion and initiating timely therapy. B-mode ultrasound (BUS), color Doppler flow imaging (CDFI), ultrasound elastography (UE) and dynamic contrast-enhanced ultrasound provide effective tools for identification but synthesis of modality information is a challenge for clinicians. Therefore, based on deep learning, rationally fusing these modalities with clinical information to personalize the classification of metastatic CLA requires new explorations. In this paper, we propose Multi-step Modality Fusion Network (MSMFN) for multi-modal ultrasound fusion to identify histological subtypes of metastatic CLA. MSMFN can mine the unique features of each modality and fuse them in a hierarchical three-step process. Specifically, first, under the guidance of high-level BUS semantic feature maps, information in CDFI and UE is extracted by modality interaction, and the static imaging feature vector is obtained. Then, a self-supervised feature orthogonalization loss is introduced to help learn modality heterogeneity features while maintaining maximal task-consistent category distinguishability of modalities. Finally, six encoded clinical information are utilized to avoid prediction bias and improve prediction ability further. Our three-fold cross-validation experiments demonstrate that our method surpasses clinicians and other multi-modal fusion methods with an accuracy of 80.06%, a true-positive rate of 81.81%, and a true-negative rate of 80.00%. Our network provides a multi-modal ultrasound fusion framework that considers prior clinical knowledge and modality-specific characteristics. Our code will be available at: https://github.com/RichardSunnyMeng/MSMFN.

Automatic semantic segmentation in 2D echocardiography is vital in clinical practice for assessing various cardiac functions and improving the diagnosis of cardiac diseases. However, two distinct problems have persisted in automatic segmentation in 2D echocardiography, namely the lack of an effective feature enhancement approach for contextual feature capture and lack of label coherence in category prediction for individual pixels. Therefore, in this study, we propose a deep learning model, called deep pyramid local attention neural network (PLANet), to improve the segmentation performance of automatic methods in 2D echocardiography. Specifically, we propose a pyramid local attention module to enhance features by capturing supporting information within compact and sparse neighboring contexts. We also propose a label coherence learning mechanism to promote prediction consistency for pixels and their neighbors by guiding the learning with explicit supervision signals. The proposed PLANet was extensively evaluated on the dataset of cardiac acquisitions for multi-structure ultrasound segmentation (CAMUS) and sub-EchoNet-Dynamic, which are two large-scale and public 2D echocardiography datasets. The experimental results show that PLANet performs better than traditional and deep learning-based segmentation methods on geometrical and clinical metrics. Moreover, PLANet can complete the segmentation of heart structures in 2D echocardiography in real time, indicating a potential to assist cardiologists accurately and efficiently.

Improving the quality of image-guided radiation therapy requires the tracking of respiratory motion in ultrasound sequences. However, the low signal-to-noise ratio and the artifacts in ultrasound images make it difficult to track targets accurately and robustly. In this study, we propose a novel deep learning model, called a Cascaded One-shot Deformable Convolutional Neural Network (COSD-CNN), to track landmarks in real time in long ultrasound sequences. Specifically, we design a cascaded Siamese network structure to improve the tracking performance of CNN-based methods. We propose a one-shot deformable convolution module to enhance the robustness of the COSD-CNN to appearance variation in a meta-learning manner. Moreover, we design a simple and efficient unsupervised strategy to facilitate the network's training with a limited number of medical images, in which many corner points are selected from raw ultrasound images to learn network features with high generalizability. The proposed COSD-CNN has been extensively evaluated on the public Challenge on Liver UltraSound Tracking (CLUST) 2D dataset and on our own ultrasound image dataset from the First Affiliated Hospital of Sun Yat-sen University (FSYSU). Experiment results show that the proposed model can track a target through an ultrasound sequence with high accuracy and robustness. Our method achieves new state-of-the-art performance on the CLUST 2D benchmark set, indicating its strong potential for application in clinical practice.

Fluorescence molecular tomography (FMT) is a highly sensitive and noninvasive imaging modality for three-dimensional visualization of fluorescence probe distribution in small animals. However, the simplified photon propagation model and ill-posed inverse problem limit the improvement of FMT reconstruction. In this work, we proposed a novel K-nearest neighbor based locally connected (KNN-LC) network to improve the performance of morphological reconstruction in FMT. It directly builds the inverse process of photon transmission by learning the mapping relation between the surface photon intensity and the distribution of fluorescent source. KNN-LC network cascades a fully connected (FC) sub-network with a locally connected (LC) sub-network, where the FC part provides a coarse reconstruction result and LC part fine-tunes the morphological quality of reconstructed result. To assess the performance of our proposed network, we implemented both numerical simulation and in vivo studies. Furthermore, split Bregman-resolved total variation (SBRTV) regularization method and inverse problem simulation (IPS) method were utilized as baselines in all comparisons. The results demonstrated that KNN-LC network achieved accurate reconstruction in both source localization and morphology recovery in a short time. This promoted the in vivo application of FMT for visualizing the distribution of biomarkers inside biological tissue.

Fluorescence molecular tomography (FMT), as a powerful imaging technique in preclinical research, can offer the three-dimensional distribution of biomarkers by detecting the fluorescently labelled probe noninvasively. However, because of the light scattering effect and the ill-pose of inverse problem, it is challenging to develop an efficient reconstruction method, which can provide accurate location and morphology of the fluorescence distribution. In this research, we proposed a novel adaptive Gaussian weighted Laplace prior (AGWLP) regularization method, which assumed the variance of fluorescence intensity between any two voxels had a non-linear correlation with their Gaussian distance. It utilized an adaptive Gaussian kernel parameter strategy to achieve accurate morphological reconstructions in FMT. To evaluate the performance of the AGWLP method, we conducted numerical simulation and in vivo experiments. The results were compared with fast iterative shrinkage (FIS) thresholding method, split Bregman-resolved TV (SBRTV) regularization method, and Gaussian weighted Laplace prior (GWLP) regularization method. We validated in vivo imaging results against planar fluorescence images of frozen sections. The results demonstrated that the AGWLP method achieved superior performance in both location and shape recovery of fluorescence distribution. This enabled FMT more suitable and practical for in vivo visualization of biomarkers.

In [1], the affiliation for Y. Gao, K. Wang and J. Tian should have appeared as follows:.

In[1], the affiliation for Y. Gao, K. Wang and J. Tian should have appeared as follows:.

Bioluminescence tomography (BLT) is a powerful non-invasive molecular imaging tool for in vivo studies of glioma in mice. However, because of the light scattering and resulted ill-posed problems, it is challenging to develop a sufficient reconstruction method, which can accurately locate the tumor and define the tumor morphology in three-dimension. In this paper, we proposed a novel Gaussian weighted Laplace prior (GWLP) regularization method. It considered the variance of the bioluminescence energy between any two voxels inside an organ had a non-linear inverse relationship with their Gaussian distance to solve the over-smoothed tumor morphology in BLT reconstruction. We compared the GWLP with conventional Tikhonov and Laplace regularization methods through various numerical simulations and in vivo orthotopic glioma mouse model experiments. The in vivo magnetic resonance imaging and ex vivo green fluorescent protein images and hematoxylin-eosin stained images of whole head cryoslicing specimens were utilized as gold standards. The results demonstrated that GWLP achieved the highest accuracy in tumor localization and tumor morphology preservation. To the best of our knowledge, this is the first study that achieved such accurate BLT morphological reconstruction of orthotopic glioma without using any segmented tumor structure from any other structural imaging modalities as the prior for reconstruction guidance. This enabled BLT more suitable and practical for in vivo imaging of orthotopic glioma mouse models.

The computation of model matrix in the iterative imaging reconstruction process is crucial for the quantitative photoacoustic tomography (PAT). However, it is challenging to establish an outstanding model matrix to improve the overall imaging quality in PAT due to the noisy signal acquisition and inevitable artifacts. In this work, we present a novel method, named as the curve-driven-based model-matrix inversion (CDMMI), to calculate the model matrix for tomographic reconstruction in photoacoustic imaging. It eliminated the use of interpolation techniques, and thus avoided all interpolation related errors. The conventional interpolated-matrix-model inversion (IMMI) method was applied to evaluate its performance in numerical simulation, tissue-mimicking phantom and in vivo small animal studies. Results demonstrated that CDMMI achieved better reconstruction accuracy until IMMI kept increasing discrete points to 10000. Furthermore, the proposed method can suppress the negative influence of noise and artifacts effectively, which benefited the overall imaging quality of photoacoustic tomography.

The accurate segmentation of lung lesions from computed tomography (CT) scans is important for lung cancer research and can offer valuable information for clinical diagnosis and treatment. However, it is challenging to achieve a fully automatic lesion detection and segmentation with acceptable accuracy due to the heterogeneity of lung lesions. Here, we propose a novel toboggan based growing automatic segmentation approach (TBGA) with a three-step framework, which are automatic initial seed point selection, multi-constraints 3D lesion extraction and the final lesion refinement. The new approach does not require any human interaction or training dataset for lesion detection, yet it can provide a high lesion detection sensitivity (96.35%) and a comparable segmentation accuracy with manual segmentation (P > 0.05), which was proved by a series assessments using the LIDC-IDRI dataset (850 lesions) and in-house clinical dataset (121 lesions). We also compared TBGA with commonly used level set and skeleton graph cut methods, respectively. The results indicated a significant improvement of segmentation accuracy . Furthermore, the average time consumption for one lesion segmentation was under 8 s using our new method. In conclusion, we believe that the novel TBGA can achieve robust, efficient and accurate lung lesion segmentation in CT images automatically.