Ultrasound imaging and biochemical examinations are the primary methods for diagnosing Hashimoto's thyroiditis (HT). However, neither of them is sufficient to accurately diagnose HT alone. Most existing multimodal models for HT diagnosis focus primarily on extracting and concatenating features from different modalities, which are ineffective due to the dimensional imbalance of the features between the textual and image data. To address this issue, we propose a novel Multimodal Collaborative Fusion Learning (MCFL) approach, which can enhance and recalibrate the biochemical indicators using ultrasound images, effectively improving the significance and specificity of biochemical indicators for the diagnosis of HT. Specifically, MCFL first constructs a novel INNet to convert the image-level characteristics of the HT ultrasound image into two numerical indicators, i.e., the Local prominent inflammatory (Lpi) and the Global diffuse lesion (Gdl), unifying image data and textual data into a single representation space. Then, a decision tree-based optimization strategy is employed to supervise the training of INNet, interactively recalibrating biochemical indicators with the guidance of the two numerical indicators mentioned above and obtaining a more accurate feature representation of HT. Finally, based on the deep Q-learning framework, a reward mechanism is established to guide the HT diagnostic process, in which the experience replay mechanism and the $\epsilon $ -greedy strategy are utilized collaboratively to improve the accuracy and robustness of the model. Extensive experiments are conducted on a multimodal dataset from multiple medical centers, and the results demonstrate that MCFL achieves state-of-the-art performance, setting a new benchmark.

Ultrasound imaging can reveal the typical changes in thyroid tissue caused by Hashimoto's Thyroiditis (HT), which plays a crucial role in HT diagnosis. Clinicians should perform ultrasound imaging from multiple anatomical planes to obtain a series of thyroid images, thereby collecting comprehensive information about HT lesions from different perspectives. However, ultrasound images lacking typical HT characteristics or containing extensive non-HT regions may hinder the identification of localized lesions, thereby increasing the complexity of diagnosis. In addition, the ability to capture synergistic interactions among multiple lesions in critical ultrasound images is crucial for improving the diagnostic accuracy of HT. To address these challenges, a novel weakly supervised multi-instance learning model, HTMIL, is proposed for HT diagnosis, requiring only patient-level data annotation. HTMIL consists of a Global-Neighbor extraction Layer (GNL) and a Cross-Aggregation Layer (CAL). Specifically, a Global-Neighbor Awareness (GNA) module is proposed in GNL to allow HTMIL to focus on key localized lesions and ignore noise from unrelated regions in HT ultrasound images, thus improving the effectiveness of extracting focused features from localized lesions. The Adaptive Bilinear Pooling (ABP) module is introduced in CAL to capture the interaction features between key localized lesions and other characteristics of HT in critical ultrasound images, thus achieving synergistic diagnosis and further increasing diagnostic accuracy. HTMIL achieves state-of-the-art (SOTA) performance with 87.67% accuracy and 91.73% AUC on a multicenter HT dataset, and its robustness is further validated on a public dataset.