Sign Language Translation (SLT) is a challenging task, with existing approaches often constrained by the necessity of gloss annotations (sign language lexemes). Since gloss annotations are expensive and difficult to obtain, they severely limit the scalability of SLT in practical applications. To address this, we propose leveraging inter-frame optical flow as crucial prior information to enhance visual feature extraction, which naturally corresponds to signers’ semantic movements. We design a novel gloss-free optical-Flow Guided Two-Stream Network architecture (FGTSN): The Image Encoder Stream employs optical flow as a static prior to guide spatial attention while the Skeleton Encoder Stream integrates it to enrich the keypoint features and enhance the representation of local temporal dynamics and structural changes. To tackle the alignment challenge inherent in gloss-free SLT, we introduce a novel auxiliary block prediction loss that strengthens the local temporal relationships of the visual features. Experiments on three major datasets demonstrate that our FGTSN achieves state-of-the-art performance with significant improvements over existing methods, providing a novel and lightweight perspective for gloss-free sign language translation.

Large-scale multimodal models have shown excellent performance over a series of tasks powered by the large corpus of paired multimodal training data. Generally, they are always assumed to receive modality-complete inputs. However, this simple assumption may not always hold in the real world due to privacy constraints or collection difficulty, where models pretrained on modality-complete data easily demonstrate degraded performance on missing-modality cases. To handle this issue, we refer to prompt learning to adapt large pretrained multimodal models to handle missing-modality scenarios by regarding different missing cases as different types of input. Instead of only prepending independent prompts to the intermediate layers, we present to leverage the correlations between prompts and input features and excavate the relationships between different layers of prompts to carefully design the instructions. We also incorporate the complementary semantics of different modalities to guide the prompting design for each modality. Extensive experiments on three commonly-used datasets consistently demonstrate the superiority of our method compared to the previous approaches upon different missing scenarios. Plentiful ablations are further given to show the generalizability and reliability of our method upon different modality-missing ratios and types.