In this article, we present skeleton-based isolated sign language recognition (IsoSLR) with part mixing - SKIM. An IsoSLR model that solely takes the skeleton representation of the human body as input. Previous skeleton-based works either perform worse when compared to RGB-based counterparts or require fusion with other modalities to obtain competitive results. With SKIM, a single skeleton-based model without complex pre-training can obtain similar or even higher accuracy than current state-of-the-art methods. This margin can be further increased by simple late fusion within the same modality. To achieve this, we first develop a novel data augmentation technique called part mixing. It swaps the corresponding keypoints within one region (e.g. hand) between two randomly selected samples and combines their labels linearly as the new label. As regions like hand and face are key articulators for sign language, direct swapping of such parts creates a believable pseudo sign that promotes the model to recognize the true pairs. Secondly, following current advances in skeleton-based action recognition, we devise a channel-wise graph neural network with multi-scale awareness and per-keypoint temporal re-weighting. With this design, the backbone is capable of leveraging both manual and non-manual features. The combination of hand mixing and the channel-wise multi-scale GCN backbone allows us to achieve state-of-the-art accuracy on both WLASL and NMFs-CSL benchmarks.

In this paper, we tackle the problem of sign language translation (SLT) without gloss annotations.Although intermediate representation like gloss has been proven effective, gloss annotations are hard to acquire, especially in large quantities.This limits the domain coverage of translation datasets, thus handicapping real-world applications.To mitigate this problem, we design the Gloss-Free End-to-end sign language translation framework (GloFE).Our method improves the performance of SLT in the gloss-free setting by exploiting the shared underlying semantics of signs and the corresponding spoken translation.Common concepts are extracted from the text and used as a weak form of intermediate representation.The global embedding of these concepts is used as a query for cross-attention to find the corresponding information within the learned visual features.In a contrastive manner, we encourage the similarity of query results between samples containing such concepts and decrease those that do not.We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign. 1