Complete multimodal Magnetic Resonance Imaging (MRI) plays an indispensable role in the task of brain tumor segmentation. However, the issue of missing-modality often arises in clinical practice, leading to a significant decline in the accuracy of segmentation. Current methods exhibit suboptimal performance under such scenarios of severe missing modalities due to either limited synthesis capacity for missing modalities or modality-specific information loss caused by strict alignment in latent space. To address this challenge, we propose a Pre-Post Interaction Learning (PPIL) approach that enhances the model’s robustness under severe missing-modality scenarios while maintaining competitive performance when most of the modalities are available. Specifically, separate branches are introduced for each modality to preserve modality-specific information. Meanwhile, a Pre-Interaction component that takes the concatenation of available modalities as input is introduced to capture more inter-modal correlations. Furthermore, a Post-Interaction component is proposed to perceive the importance of all branches and dynamically combine their information, thus mitigating the information loss from the strict latent feature alignment. We validate the effectiveness of PPIL on two benchmark datasets, BraTS2020 and BraTS2018, demonstrating a significant improvement in performance under severe missing-modality scenarios while preserving competitive performance when most modalities are available.

Video try-on is challenging and has not been well tackled in previous works. The main obstacle lies in preserving the clothing details and modeling the coherent motions simultaneously. Faced with those difficulties, we address video try-on by proposing a diffusion-based framework named ''Tunnel Try-on.'' The core idea is excavating a ''focus tunnel'' in the input video that gives close-up shots around the clothing regions. We zoom in on the region in the tunnel to better preserve the fine details of the clothing. To generate coherent motions, we leverage the Kalman filter to smooth the tunnel and inject its position embedding into attention layers to improve the continuity of the generated videos. In addition, we develop an environment encoder to extract the context information outside the tunnels. Equipped with these techniques, Tunnel Try-on keeps fine clothing details and synthesizes stable and smooth videos. Demonstrating significant advancements, Tunnel Try-on could be regarded as the first attempt toward the commercial-level application of virtual try-on in videos. The project page is https://mengtingchen.github.io/tunnel-try-on-page/.

This paper explores the tasks of leveraging auxiliary modalities which are only available at training to enhance multimodal representation learning through cross-modal Knowledge Distillation (KD). The widely adopted mutual information maximization-based objective leads to a short-cut solution of the weak teacher, i.e., achieving the maximum mutual information by simply making the teacher model as weak as the student model. To prevent such a weak solution, we introduce an additional objective term, i.e., the mutual information between the teacher and the auxiliary modality model. Besides, to narrow down the information gap between the student and teacher, we further propose to minimize the conditional entropy of the teacher given the student. Novel training schemes based on contrastive learning and adversarial learning are designed to optimize the mutual information and the conditional entropy, respectively. Experimental results on three popular multimodal benchmark datasets have shown that the proposed method outperforms a range of state-of-the-art approaches for video recognition, video retrieval and emotion classification.