Glioblastoma (GBM) is a malignant brain tumor with poor prognosis even after extensive treatment. This poor prognosis is essentially due to the high rate of post-treatment pseudoprogression (PsP), which can be easily confused with true tumor progression (TTP). Thus, there has been a pressing need for developing reliable non-invasive methods to distinguish between these two types of tumor progression in clinical practice. Diffusion tensor imaging has demonstrated a high potential for automated GBM diagnosis. However, performance on this diagnosis task is typically restricted by data insufficiency and the difficulty of fine-grained classification. Hence, we developed a dual-graph neural network with a randomness-constrained strategy, which jointly achieves discriminative feature extraction and stable label inference. Specifically, we designed a dual-graph architecture comprising two structures. One is a multi-instance-learning-based graph embedding, which extracts features of local target regions. The other is a metric graph that accounts for interactions between patients and consequently enables patient-level label inference. Subsequently, we introduced a randomness-constraint strategy to maximize the utilization of historical data and employed data relationships to obtain highly discriminative graph structures. These aspects boosted performance stability on small-sample data. Extensive experiments have demonstrated promising performance of the proposed model for distinguishing between PsP and TTP. Furthermore, its scalability was verified on two public datasets addressing fine-grained classification challenges in different tasks. Overall, the proposed model provides an effective solution for GBM progression assessment and stable fine-grained classification under small-sample conditions. The code will be released at https://github.com/SJTUBME-QianLab/GBM-Dual_GNN .

Postural instability, one of the primary symptoms of Parkinson’s disease, reflects the severity of muscle rigidity and bradykinesia and is typically assessed via the Movement Disorder Society-Unified Parkinson’s Disease Rating Scale (MDS-UPDRS). However, the scarcity of qualified physicians and the subjective variability in clinical scoring underscore the necessity for developing automated assessment systems that provide reliable and objective diagnosis of postural instability. Therefore, we propose a causal graph-based stable scoring model with a clinical embedding architecture for multi-class postural stability assessment. This architecture systematically employs counterfactual reasoning, causal invariance constraints, and clinical embedding such that clinically causal features are extracted from video-based skeleton data, and thereby a more stable automated assessment is achieved. Specifically, we initially developed a causal counterfactual suppression strategy to deemphasize non-causal features and suppress their confounding influence. Subsequently, we designed a causal invariance reinforcement strategy that combines causal and non-causal features to reduce pseudo-correlation between postural stability and non-causal features. Finally, we proposed a clinical embedding guidance strategy that directs attention toward clinically important regions to extract salient features. The proposed method for postural stability four-class assessment achieved clinically interpretable results with an accuracy of 75.09% and an acceptable accuracy of 97.50% on the largest clinical dataset of more than 1,000 videos. The effectiveness and robustness of our method were further confirmed on an independent test set, and the wide applicability of this method was further demonstrated on two additional fine-grained recognition tasks of skating and medical gait. The proposed method offers an effective, robust, and objective solution for postural stability assessment, with a significant potential for broader applications in other fine-grained action recognition tasks.

Postural tremor of hands, a typical symptom of Parkinson's disease (PD), is assessed using the Movement Disorder Society-sponsored revision of the Unified PD Rating Scale (MDS-UPDRS). However, the manual assessment process is both subjective and time-consuming, underscoring the necessity for automated rating models in clinical practice. To accommodate large-scale applications, such models must effectively identify and mitigate the impact of confounding factors in complex clinical settings, ensuring stable and robust assessment performance. Therefore, we propose a causality-representation graph convolutional network (GCN) scheme for automated video-based assessment of postural tremor of hands. This scheme systematically eliminates the effects of confounding factors by extracting discriminative structure and representation from hand skeleton graphs, which are clinically significant and causally related to tremor assessment, ultimately achieving stable multiclass tremor scoring. Specifically, a causality-informed graph structure mining (CI-GSM) module was first proposed to automatically extract nodes with discriminative tremor features from skeletal graphs, construct hand skeleton graphs causally related to tremor assessment, and thereby suppress interference from irrelevant nodes. Afterward, a causality-representation enhancement (CRE) module was designed to improve the construction of graph representations that correlate strongly with tremor assessment, thereby further reducing noise during evaluation. Our method achieves an accuracy of 64.01% and an acceptable accuracy of 98.62% on a large clinical dataset, demonstrating satisfactory performance on independent test sets collected from multiple centers. In conclusion, our proposed approach offers a convenient and stable solution for objective assessment of PD-associated tremors and holds significant potential for large-scale applications in remote PD assessment.

Pancreatic cancer, characterized by its notable prevalence and mortality rates, demands accurate lesion delineation for effective diagnosis and therapeutic interventions. The generalizability of extant methods is frequently compromised due to the pronounced variability in imaging and the heterogeneous characteristics of pancreatic lesions, which may mimic normal tissues and exhibit significant inter-patient variability. Thus, we propose a generalization framework that synergizes pixel-level classification and regression tasks, to accurately delineate lesions and improve model stability. This framework not only seeks to align segmentation contours with actual lesions but also uses regression to elucidate spatial relationships between diseased and normal tissues, thereby improving tumor localization and morphological characterization. Enhanced by the reciprocal transformation of task outputs, our approach integrates additional regression supervision within the segmentation context, bolstering the model's generalization ability from a dual-task perspective. Besides, dual self-supervised learning in feature spaces and output spaces augments the model's representational capability and stability across different imaging views. Experiments on 594 samples composed of three datasets with significant imaging differences demonstrate that our generalized pancreas segmentation results comparable to mainstream in-domain validation performance (Dice: 84.07%). More importantly, it successfully improves the results of the highly challenging cross-lesion generalized pancreatic cancer segmentation task by 9.51%. Thus, our model constitutes a resilient and efficient foundational technological support for pancreatic disease management and wider medical applications. The codes will be released at https://github.com/SJTUBME-QianLab/Dual-Task-Seg.

Leg agility is a key indicator of bradykinesia, which in turn is a cardinal manifestation of Parkinson’s disease (PD). In fact, automated video assessment of the leg-agility task is critically required for improving the efficiency and objectivity of PD diagnosis. Therefore, we propose a causality-informed graph convolutional network to extract discriminative clinically-meaningful motion features from human skeletons in videos, finally achieving stable leg-agility 5-point scoring. The proposed scheme systematically mines causal features of each skeleton graph from graph node, structure, and representation levels. Specifically, we firstly developed a causality-informed node selection mechanism to mine the graph nodes representing the discriminative features, and thus identify nodes causally correlated to the clinical assessment aspects and suppress the interference from other nodes. Afterwards, a causality-informed structure generation mechanism was designed to generate a graph structure encoding the connections between the discriminative nodes, hence maintaining the discriminability of features associated with these causality-informed nodes. Finally, we employed a clinically-driven self-supervised learning scheme to embed clinical prior knowledge into the proposed model and hence boost the clinical significance of the causality-informed graph nodes, structures, and representations. The proposed method achieved a 71.11% accuracy and a 98.93% acceptable accuracy on a large clinical video dataset. Its effectiveness was also confirmed on an independent test set, and the obtained results exhibited interpretability from modeling and clinical perspectives. In conclusion, our method provides a highly stable scheme for objective video quantification of bradykinesia. Our source code will be released athttps://github.com/SJTUBME-QianLab/PD-CIGCN.

In the clinically widely used rating scale (MDS-UPDRS), the pronation-supination movement task of hands is required for assessment of bradykinesia, which is a typical clinical symptom of Parkinson’s disease (PD). Due to inter-rater variability in the task rating process, objective automated rating models are critically needed. Still, the performance of such models would be limited if prior knowledge of the clinical rating principles is not adequately accounted for. Therefore, we propose a clinically guided method which fully exploits the MDS-UPDRS rating principles to achieve consistent and accurate automated rating. First, a multi-scale framework which employs two graph convolutional networks (GCNs) as two streams is developed to extract transient and persistent features related to these rating principles. In particular, abnormal transient features are detected through a specialized multiple-instance-learning GCN. Moreover, the multiple-instance-learning GCN is equipped with an accumulation-aware ordered multiple-instance pooling module, which estimates sample-level severity by accounting for both the intra-instance intrinsic severity order and the inter-instance accumulation effects. Besides, an instance context encoding module is designed to combine the phase information in the pronation-supination movement cycles with instances’ motion features. This facilitates the differentiation between PD-induced halts and natural periodic halts. Our method demonstrated excellent performance on both a large clinical dataset and an additional independent test dataset. Our proposed scheme only requires consumer-level cameras, and therefore exhibits high potential for large-scale applications in PD telemedicine.

Freezing of gait (FoG) is a common disabling symptom of Parkinson's disease (PD). It is clinically characterized by sudden and transient walking interruptions for specific human body parts, and it presents the localization in time and space. Due to the difficulty in extracting global fine-grained features from lengthy videos, developing an automated five-point FoG scoring system is quite challenging. Therefore, we propose a novel video-based automated five-classification FoG assessment method with a causality-enhanced multiple-instance-learning graph convolutional network (GCN). This method involves developing a temporal segmentation GCN to segment each video into three motion stages for stage-level feature modeling, followed by a multiple-instance-learning framework to divide each stage into short clips for instance-level feature extraction. Subsequently, an uncertainty-driven multiple-instance-learning GCN is developed to capture spatial and temporal fine-grained features through GCN scheme and uncertainty learning, respectively, for acquiring global representations. Finally, a causality-enhanced graph generation strategy is proposed to exploit causal inference for mining and enhancing human structures causally related to clinical assessment, thereby extracting spatial causal features. Extensive experimental results demonstrate the excellent performance of the proposed method on five-classification FoG assessment with an accuracy of 62.72% and an acceptable accuracy of 91.32%, which is confirmed by independent testing. Additionally, it enables temporal and spatial localization of FoG events to a certain extent, facilitating reasonable clinical interpretations. In conclusion, our method provides a valuable tool for automated FoG assessment in PD, and the proposed causality-related component exhibits promising potential for extension to other general and medical fine-grained action recognition tasks.

Segmenting peripancreatic vessels in CT, including the superior mesenteric artery (SMA), the coeliac artery (CA), and the partial portal venous system (PPVS), is crucial for preoperative resectability analysis in pancreatic cancer. However, the clinical applicability of vessel segmentation methods is impeded by the low generalizability on multi-center data, mainly attributed to the wide variations in image appearance, namely the spurious correlation factor. Therefore, we propose a causal-invariance-driven generalizable segmentation model for peripancreatic vessels. It incorporates interventions at both image and feature levels to guide the model to capture causal information by enforcing consistency across datasets, thus enhancing the generalization performance. Specifically, firstly, a contrast-driven image intervention strategy is proposed to construct image-level interventions by generating images with various contrast-related appearances and seeking invariant causal features. Secondly, the feature intervention strategy is designed, where various patterns of feature bias across different centers are simulated to pursue invariant prediction. The proposed model achieved high DSC scores (79.69%, 82.62%, and 83.10%) for the three vessels on a cross-validation set containing 134 cases. Its generalizability was further confirmed on three independent test sets of 233 cases. Overall, the proposed method provides an accurate and generalizable segmentation model for peripancreatic vessels and offers a promising paradigm for increasing the generalizability of segmentation models from a causality perspective. Our source codes will be released at https://github.com/ SJTUBME-QianLab/PC_VesselSeg.

Rigidity is one of the common motor disorders in Parkinson's disease (PD), which lead to life quality deterioration. The widely-used rating-scale-based approach for rigidity assessment still depends on the availability of experienced neurologists and is limited by rating subjectivity. Given the recent successful applications of quantitative susceptibility mapping (QSM) in auxiliary PD diagnosis, automated assessment of PD rigidity can be essentially achieved through QSM analysis. However, a major challenge is the performance instability due to the confounding factors (e.g., noise and distribution shift) which conceal the truly-causal features. Therefore, we propose a causality-aware graph convolutional network (GCN) framework, where causal feature selection is combined with causal invariance to ensure that causality-informed model decisions are reached. Firstly, a GCN model that integrates causal feature selection is systematically constructed at three graph levels: node, structure, and representation. In this model, a causal diagram is learned to extract a subgraph with truly-causal information. Secondly, a non-causal perturbation strategy is developed along with an invariance constraint to ensure the stability of the assessment results under different distributions, and thus avoid spurious correlations caused by distribution shifts. The superiority of the proposed method is shown by extensive experiments and the clinical value is revealed by the direct relevance of selected brain regions to rigidity in PD. Besides, its extensibility is verified on other two tasks: PD bradykinesia and mental state for Alzheimer's disease. Overall, we provide a clinically-potential tool for automated and stable assessment of PD rigidity. Our source code will be available at https://github.com/SJTUBME-QianLab/Causality-Aware-Rigidity.

Accurate pancreas segmentation is highly crucial for diagnosing and treating pancreatic diseases. Although CNN has demonstrated promising outcomes, the performance on unseen data can be significantly compromised by the wide appearance-style variations induced by different imaging factors. Thus, we propose a generalizable pancreas segmentation model based on a meta-learning strategy and latent-space feature flow generation method. Our approach enhances the generalizability by systematically reducing the interference from the cluttered background and appearance-style discrepancies through a coarse-to-fine workflow. Specifically, the integrity-preserving coarse segmentation module is designed to adaptively balance the pancreas coverage and segmentation accuracy with the meta-learning strategy for filtering out background clutter. It also enhances the generalization of the coarse model to reasonably-accurate ROIs thereby promoting the stability of fine segmentation. Subsequently, the appearance-style feature flow generation method is developed to generate a series of progressively-varying style-related intermediate representations between two latent spaces. This feature flow effectively models the distribution variations caused by appearance-style discrepancies, and thus enhances the adaptability of the fine model. Our method achieves superior performance on three pancreas datasets and outperforms state-of-the-art generalization methods. Besides, it can be easily integrated into other workflows, leading to a potential paradigm for enhancing generalization performance.

Recently, numerous pancreas segmentation methods have achieved promising performance on local single-source datasets. However, these methods don't adequately account for generalizability issues, and hence typically show limited performance and low stability on test data from other sources. Considering the limited availability of distinct data sources, we seek to improve the generalization performance of a pancreas segmentation model trained with a single-source dataset, i.e., the single-source generalization task. In particular, we propose a dual self-supervised learning model that incorporates both global and local anatomical contexts. Our model aims to fully exploit the anatomical features of the intra-pancreatic and extra-pancreatic regions, and hence enhance the characterization of the high-uncertainty regions for more robust generalization. Specifically, we first construct a global-feature contrastive self-supervised learning module that is guided by the pancreatic spatial structure. This module obtains complete and consistent pancreatic features through promoting intra-class cohesion, and also extracts more discriminative features for differentiating between pancreatic and non-pancreatic tissues through maximizing inter-class separation. It mitigates the influence of surrounding tissue on the segmentation outcomes in high-uncertainty regions. Subsequently, a local-image-restoration self-supervised learning module is introduced to further enhance the characterization of the high-uncertainty regions. In this module, informative anatomical contexts are actually learned to recover randomly-corrupted appearance patterns in those regions. The effectiveness of our method is demonstrated with state-of-the-art performance and comprehensive ablation analysis on three pancreas datasets (467 cases). The results demonstrate a great potential in providing a stable support for the diagnosis and treatment of pancreatic diseases.

Pancreatic cancer is the emperor of all cancer maladies, mainly because there are no characteristic symptoms in the early stages, resulting in the absence of effective screening and early diagnosis methods in clinical practice. Non-contrast computerized tomography (CT) is widely used in routine check-ups and clinical examinations. Therefore, based on the accessibility of non-contrast CT, an automated early diagnosismethod for pancreatic cancer is proposed. Among this, we develop a novel causalitydriven graph neural network to solve the challenges of stability and generalization of early diagnosis, that is, the proposed method achieves stable performance for datasets from different hospitals, which highlights its clinical significance. Specifically, a multiple-instance-learning framework is designed to extract fine-grained pancreatic tumor features. Afterwards, to ensure the integrity and stability of the tumor features, we construct an adaptivemetric graph neural network that effectively encodes prior relationships of spatial proximity and feature similarity for multiple instances, and hence adaptively fuses the tumor features. Besides, a causal contrastivemechanism is developed to decouple the causality-driven and non-causal components of the discriminative features, suppress the non-causal ones, and hence improve the model stability and generalization. Extensive experiments demonstrated that the proposed method achieved the promising early diagnosis performance, and its stability and generalizability were independently verified on amulti-center dataset. Thus, the proposed method provides a valuable clinical tool for the early diagnosis of pancreatic cancer. Our source codes will be released at https://github.com/SJTUBME-QianLab/ CGNN-PC-Early-Diagnosis.

Glioblastoma multiforme (GBM) is the most common type of brain tumors with high recurrence and mortality rates. After chemotherapy treatment, GBM patients still show a high rate of differentiating pseudoprogression (PsP), which is often confused as true tumor progression (TTP) due to high phenotypical similarities. Thus, it is crucial to construct an automated diagnosis model for differentiating between these two types of glioma progression. However, attaining this goal is impeded by the limited data availability and the high demand for interpretability in clinical settings. In this work, we propose an interpretable structure-constrained graph neural network (ISGNN) with enhanced features to automatically discriminate between PsP and TTP. This network employs a metric-based meta-learning strategy to aggregate class-specific graph nodes, focus on meta-tasks associated with various small graphs, thus improving the classification performance on small-scale datasets. Specifically, a node feature enhancement module is proposed to account for the relative importance of node features and enhance their distinguishability through inductive learning. A graph generation constraint module enables learning reasonable graph structures to improve the efficiency of information diffusion while avoiding propagation errors. Furthermore, model interpretability can be naturally enhanced based on the learned node features and graph structures that are closely related to the classification results. Comprehensive experimental evaluation of our method demonstrated excellent interpretable results in the diagnosis of glioma progression. In general, our work provides a novel systematic GNN approach for dealing with data scarcity and enhancing decision interpretability. Our source codes will be released at https://github.com/SJTUBME-QianLab/GBM-GNN.

Abnormal posture is a common movement disorder in the progress of Parkinson's disease (PD), and this abnormality can increase the risk of falls or even disabilities. The conventional assessment approach depends on the judgment of well-trained experts via canonical scales. However, this approach requires extensive clinical expertise and is highly subjective. Considering the potential of quantitative susceptibility mapping (QSM) in PD diagnosis, this study explored the QSM-based method for the automated classification between PD patients with and without postural abnormalities. Nevertheless, a major challenge is that unstable non-causal features typically lead to less reliable performance. Therefore, we propose a causality-driven graph-convolutional-network framework based on multi-instance learning, where performance stability is enhanced through the invariant prediction principle and causal interventions. Specifically, we adopt an intervention strategy that combines a non-causal intervenor with causal prediction. A stability constraint is proposed to ensure robust integrated prediction under different interventions. Moreover, an intra-class homogeneity constraint is enforced for each individually-learned causality scoring module to promote the extraction of group-level general features, and hence achieve a balance between subject-specific and group-level features. The proposed method demonstrated promising performance through extensive experiments on a real clinical dataset. Also, the features extracted by our method coincide with those reported in previous medical studies on PD posture abnormalities. In general, our work provides a clinically-valuable approach for automated, objective, and reliable diagnosis of postural abnormalities in Parkinsonians. Our source code is publicly available at https://github.com/SJTUBME-QianLab/CausalGCN-PDPA.

The onset and progression of Parkinson’s disease (PD) gradually affect the patient’s motor functions and quality of life. The PD motor symptoms are usually assessed using the Movement Disorder Society-sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS). Automated MDS-UPDRS assessment has been recently required as an invaluable tool for PD diagnosis and telemedicine, especially with the recent novel coronavirus pandemic outbreak. This paper proposes a novel vision-based method for automated assessment of the arising-from-chair task, which is one of the key MDS-UPDRS components. The proposed method is based on a self-supervised metric learning scheme with a graph convolutional network (SSM-GCN). Specifically, for human skeleton sequences extracted from videos, a self-supervised intra-video quadruplet learning strategy is proposed to construct a metric learning formulation with prior knowledge, for improving the spatial-temporal representations. Afterwards, a vertex-specific convolution operation is designed to achieve effective aggregation of all skeletal joint features, where each joint or feature is weighted differently based on its relative factor of importance. Finally, a graph representation supervised mechanism is developed to maximize the potential consistency between the joint and bone information streams. Experimental results on a clinical dataset demonstrate the superiority of the proposed method over the existing sensor-based methods, with an accuracy of 70.60% and an acceptable accuracy of 98.65%. The analysis of discriminative spatial connections makes our predictions more clinically interpretable. This method can achieve reliable automated PD assessment using only easily-obtainable videos, thus providing an effective tool for real-time PD diagnosis or remote continuous monitoring.

One of the common motor symptoms of Parkinson’s disease (PD) is bradykinesia. Automated bradykinesia assessment is critically needed for helping neurologists achieve objective clinical diagnosis and hence provide timely and appropriate medical services. This need has become especially urgent after the outbreak of the coronavirus pandemic in late 2019. Currently, the main factor limiting the accurate assessment is the difficulty of mining the fine-grained discriminative motion features. Therefore, we propose a novel contrastive graph convolutional network for automated and objective toe-tapping assessment, which is one of the most important tests of lower-extremity bradykinesia. Specifically, based on joint sequences extracted from videos, a supervised contrastive learning strategy was followed to cluster together the features of each class, thereby enhancing the specificity of the learnt class-specific features. Subsequently, a multi-stream joint sparse learning mechanism was designed to eliminate potentially similar redundant features of joint position and motion, hence strengthening the discriminability of features extracted from different streams. Finally, a spatial-temporal interaction graph convolutional module was developed to explicitly model remote dependencies across time and space, and hence boost the mining of fine-grained motion features. Comprehensive experimental results demonstrate that this method achieved remarkable classification performance on a clinical video dataset, with an accuracy of 70.04% and an acceptable accuracy of 98.70%. These results obviously outperformed other existing sensor- and video-based methods. The proposed video-based scheme provides a reliable and objective tool for automated quantitative toe-tapping assessment, and is expected to be a viable method for remote medical assessment and diagnosis.

Automated pancreatic cancer segmentation is highly crucial for computer-assisted diagnosis. The general practice is to label images from selected modalities since it is expensive to label all modalities. This practice brought about a significant interest in learning the knowledge transfer from the labeled modalities to unlabeled ones. However, the imaging parameter inconsistency between modalities leads to a domain shift, limiting the transfer learning performance. Therefore, we propose an unsupervised domain adaptation segmentation framework for pancreatic cancer based on GCN and meta-learning strategy. Our model first transforms the source image into a target-like visual appearance through the synergistic collaboration between image and feature adaptation. Specifically, we employ encoders incorporating adversarial learning to separate domain-invariant features from domain-specific ones to achieve visual appearance translation. Then, the meta-learning strategy with good generalization capabilities is exploited to strike a reasonable balance in the training of the source and transformed images. Thus, the model acquires more correlated features and improve the adaptability to the target images. Moreover, a GCN is introduced to supervise the high-dimensional abstract features directly related to the segmentation outcomes, and hence ensure the integrity of key structural features. Extensive experiments on four multi-parameter pancreatic-cancer magnetic resonance imaging datasets demonstrate improved performance in all adaptation directions, confirming our model's effectiveness for unlabeled pancreatic cancer images. The results are promising for reducing the burden of annotation and improving the performance of computer-aided diagnosis of pancreatic cancer. Our source codes will be released at https://github.com/SJTUBME-QianLab/UDAseg, once this manuscript is accepted for publication.

Pancreatic cancer is a lethal malignant tumor with one of the worst prognoses. Accurate segmentation of pancreatic cancer is vital in clinical diagnosis and treatment. Due to the unclear boundary and small size of cancers, it is challenging to both manually annotate and automatically segment cancers. Considering 3D information utilization and small sample sizes, we propose a model-driven deep learning method for pancreatic cancer segmentation based on spiral transformation. Specifically, a spiral-transformation algorithm with uniform sampling was developed to map 3D images onto 2D planes while preserving the spatial relationship between textures, thus addressing the challenge in effectively applying 3D contextual information in a 2D model. This study is the first to introduce spiral transformation in a segmentation task to provide effective data augmentation, alleviating the issue of small sample size. Moreover, a transformation-weight-corrected module was embedded into the deep learning model to unify the entire framework. It can achieve 2D segmentation and corresponding 3D rebuilding constraint to overcome non-unique 3D rebuilding results due to the uniform and dense sampling. A smooth regularization based on rebuilding prior knowledge was also designed to optimize segmentation results. The extensive experiments showed that the proposed method achieved a promising segmentation performance on multi-parametric MRIs, where T2, T1, ADC, DWI images obtained the DSC of 65.6%, 64.0%, 64.5%, 65.3%, respectively. This method can provide a novel paradigm to efficiently apply 3D information and augment sample sizes in the development of artificial intelligence for cancer segmentation. Our source codes will be released at https://github.com/SJTUBME-QianLab/ Spiral-Segmentation.

Automated assessment of patients with Parkinson's disease (PD) is urgently required in clinical practice to improve the diagnostic efficiency and objectivity and to remotely monitor the motor disorder symptoms and general health of these patients, especially in view of the travel restrictions due to the recent coronavirus epidemic. Gait motor disorder is one of the critical manifestations of PD, and automated assessment of gait is vital to realize automated assessment of PD patients. To this end, we propose a novel two-stream spatial-temporal attention graph convolutional network (2s-ST-AGCN) for video assessment of PD gait motor disorder. Specifically, the skeleton sequence of human body is extracted from videos to construct spatial-temporal graphs of joints and bones, and a two-stream spatial-temporal graph convolutional network is then built to simultaneously model the static spatial information and dynamic temporal variations. The multi-scale spatial-temporal attention-aware mechanism is also designed to effectively extract the discriminative spatial-temporal features. The deep supervision strategy is then embedded to minimize classification errors, thereby guiding the weight update process of the hidden layer to promote significant discriminative features. Besides, two model-driven terms are integrated into this deep learning framework to strengthen multi-scale similarity in the deep supervision and realize sparsification of discriminative features. Extensive experiments on the clinical video dataset show that the proposed model exhibits good performance with an accuracy of 65.66% and an acceptable accuracy of 98.90%, which is much better than that of the existing sensor- and vision-based methods for Parkinsonian gait assessment. Thus, the proposed method is potentially useful for assessing PD gait motor disorder in clinical practice.

Alzheimer's disease (AD) is the most common cognitive disorder. In recent years, many computer-aided diagnosis techniques have been proposed for AD diagnosis and progression predictions. Among them, graph neural networks (GNNs) have received extensive attention owing to their ability to effectively fuse multimodal features and model the correlation between samples. However, many GNNs for node classification use an entire dataset to construct a large fixed-graph structure, which cannot be used for independent testing. To overcome this limitation while maintaining the advantages of the GNN, we propose an auto-metric GNN (AMGNN) model for AD diagnosis. First, a metric-based meta-learning strategy is introduced to realize inductive learning for independent testing through multiple node classification tasks. In the meta-tasks, the small graphs help make the model insensitive to the sample size, thus improving the performance under small sample size conditions. Furthermore, an AMGNN layer with a probability constraint is designed to realize node similarity metric learning and effectively fuse multimodal data. We verified the model on two tasks based on the TADPOLE dataset: early AD diagnosis and mild cognitive impairment (MCI) conversion prediction. Our model provides excellent performance on both tasks with accuracies of 94.44% and 87.50% and median accuracies of 94.19% and 86.25%, respectively. These results show that our model improves flexibility while ensuring a good classification performance, thus promoting the development of graph-based deep learning algorithms for disease diagnosis.

Pancreatic cancer is a malignant form of cancer with one of the worst prognoses. The poor prognosis and resistance to therapeutic modalities have been linked to TP53 mutation. Pathological examinations, such as biopsies, cannot be frequently performed in clinical practice; therefore, noninvasive and reproducible methods are desired. However, automatic prediction methods based on imaging have drawbacks such as poor 3D information utilization, small sample size, and ineffectiveness multi-modal fusion. In this study, we proposed a model-driven multi-modal deep learning scheme to overcome these challenges. A spiral transformation algorithm was developed to obtain 2D images from 3D data, with the transformed image inheriting and retaining the spatial correlation of the original texture and edge information. The spiral transformation could be used to effectively apply the 3D information with less computational resources and conveniently augment the data size with high quality. Moreover, model-driven items were designed to introduce prior knowledge in the deep learning framework for multi-modal fusion. The model-driven strategy and spiral transformation-based data augmentation can improve the performance of the small sample size. A bilinear pooling module was introduced to improve the performance of fine-grained prediction. The experimental results show that the proposed model gives the desired performance in predicting TP53 mutation in pancreatic cancer, providing a new approach for noninvasive gene prediction. The proposed methodologies of spiral transformation and model-driven deep learning can also be used for the artificial intelligence community dealing with oncological applications. Our source codes with a demon will be released at https://github.com/SJTUBME-QianLab/SpiralTransform.

With the advances in commercial light field cameras, light field image processing has attracted considerable attention from researchers. In this paper, we propose a novel method for generating stereoscopic images from a light field image pair with flexible control over the convergence of the virtual stereo cameras. We have developed a light field image-capturing prototype that consists of two horizontally arranged light field cameras (i.e., with their optical axes being parallel to each other). When using our proposed device for image/video capture, stereo photographers can concentrate on how to capture the desired visual experience without being frequently disturbed by having to manipulate the stereo camera parameters, i.e., the convergence angle of a stereo camera. During postprocessing, our method estimates accurate disparity maps for the light field image pair and then generates the target stereoscopic images that satisfy the desired stereo camera convergence requirements by adopting a novel view synthesis method for light field images. We have conducted extensive experiments to demonstrate the effectiveness of our proposed method.

The success of fuzzy clustering heavily relies on the features of the input data. Based on the fact that deep architectures are able to more accurately characterize the data representations in a layer-by-layer manner, this paper proposes a novel feature mapping technique called cascaded hidden-space (CHS) feature mapping and investigates its combination with classical fuzzy c-means (FCM) and fuzzy c-regressions (FCR). Since the parameters between the layers of CHS feature mapping are randomly generated and need not be tuned layer-by-layer, CHS is easily implemented with less training data. By performing classical FCM in CHS, a novel fuzzy clustering framework called CHS-FCM is developed; several of its variants are presented using different dimension-reduction methods in a CHS-FCM clustering framework. The combination of CHS-FCM with nonlinear switch regressions is called CHS-FCR, and it performs FCR in CHS. The proposed CHS-FCR provides better results than FCR for nonlinear process modeling. Both CHS-FCM and CHS-FCR exhibit low memory consumption and require less training data. The experimental results verify the superiority of the proposed methods over classical fuzzy clustering methods.