The diagnosis of depression in individuals with Parkinson's Disease (PD) through the utilization of multimodal fusion techniques represents a significant domain. The primary challenge involves the creation of a robust fusion framework to address the heterogeneity among different modalities effectively. However, previous studies primarily focused on interactions between heterogeneous data, neglecting the structural similarities among isomorphic data, resulting in a substantial loss of feature information when merging heterogeneous data. In this study, we introduced a multi-modal data image encoding and fusion approach for diagnosing depression in PD patients. Additionally, we proposed a multi-modal dataset encompassing motion, facial expression, and audio data. First, we designed an RGB and sparse coding method to encode the multi-modal data, achieving the isomorphic transformation of multi-modal information and extracting feature information from lower-dimensional spaces. Furthermore, we introduced a Spatial-Temporal Network (STN) to fuse the three types of encoded images. We incorporated the Relation Global Attention (RGA) to enhance feature extraction and leverage all encoded image location feature nodes for balanced decision attention. Finally, recognizing the limitations of traditional machine learning algorithms in handling multi-tasks in medical diagnosis, we established a multi-task weighted loss function to achieve depression identification and severity prediction through Multi-Task learning (MTL).

Elderly individuals often suffer from underlying medical conditions, resulting in a significant decline in quality of life and a heightened susceptibility to depression. Presently, AI screening tools based on behavioral indicators offer an objective and effective approach to diagnosing depression. However, current AI depression screening tools are primarily tailored to adolescents and adults, exhibiting shortcomings in their applicability and accuracy for elderly individuals with underlying medical conditions. To address the above issues, first, this paper constructs a depression dataset for elderly people with underlying diseases by using semi-structured interviews. Second, based on cognitive science insights, it is recognized that personality factors significantly influence behavioral expressions and also determine the attitudes of elderly individuals toward current life circumstances/health issues. Therefore, besides annotating depression severity, the Big Five-10 personality scale was utilized to annotate participant personalities. Finally, a late fusion-based multi-task learning framework was proposed, and the effects of introducing gait information and personality annotation on the performance of depression assessment were investigated. The experimental findings affirm the importance of integrating gait information and personality assessment in improving depression detection effectiveness. This study provides valuable foundational resources, as well as beneficial references and insights, for the research on depression in the elderly.