Sign language is an essential communication tool, however, it can be highly challenging for non-deaf students to learn. We propose a Pepper robot based sign language teaching assistant called SignPepper, with the ability to communicate in both spoken and sign language. Using Whisper speech to text and Llama 3.3, SignPepper can engage in two way spoken lessons, with the ability to physically demonstrate signs to students. Furthermore, using 3D convolutional neural networks trained on sign language recognition, SignPepper can watch, analyze and give personalized feedback on students attempts at performing newly learned signs in real-time; including hand-based error localization.

Isolated sign language recognition is a challenging task involving the learning of complex relationships between spatial and temporal features. Due to the high complexity and relatively small datasets available, state-of-the-art methods often adopt language modeling and convolutional neural network based multimodal designs, achieving high accuracy at the cost of significant architectural complexity. Conceptually simpler, transformers have gained widespread adoption in related computer vision tasks, outperforming 3D convolutional network competitors. However, due to a lack of training data, video transformers struggle with sign language recognition and have not demonstrated competitive accuracy compared to 3D convolutional neural network designs. We introduce DeepSign, a family of vision transformer based sign language recognition models with superior performance to 3D convolutional neural network designs. Through careful model ablation we select the UniFormerV2 and VideoMAE V2 architectures and perform mixture of dataset pretraining. Our strongest model DeepSign UniFormerV2-L achieves state-of-the-art on the WLASL100 and MSASL100 benchmarks, producing 92.64% and 94% top-1 accuracies respectively. Armed with VideoMAE V2’s powerful pretrained backbone, DeepSign ViT base offers greater efficiency for a small accuracy tradeoff. We hope DeepSign will help advance future sign language research by providing strong foundational models to kickstart experiments.

Sign languages are widely used forms of communication by the deaf and hearing impaired communities. Due to a lack of qualified teachers, robot sign language teaching systems have been proposed, aiming to aid in sign language education. In this paper, we conduct a study utilizing the SignPepper system that we developed; a humanoid Pepper robot based system with capabilities in sign demonstration, verbal communication, and sign language recognition. Specifically, SignPepper adopts a 3D convolutional neural network trained on 100 American sign language signs, Whisper for speech recognition, ChatGPT 4o for context phrasing and Pepper’s built-in text-to-speech functionality. Our study consisted of 33 participants split into two groups, 18 participants were taught sign language by SignPepper whilst the other 15 were shown the same signs as videos. The same sign recognition neural network is used for evaluating the recall accuracy of students in both groups. Survey results showed the SignPepper group had higher sign recall accuracy, greater interest in learning more sign language and greater engagement. However, comfort during the lesson and comfort towards robotic platforms as teaching platforms was lower than the video group. The SignPepper group also rated instruction clarity as slightly lower. Our results indicate that the physical dexterity limits of the Pepper robot platform are a major limitation; as performance of signs may not match human experts exactly. Student comfort is also an area which requires future improvements. Nevertheless, SignPepper demonstrates strong viability for the adoption of robotic sign language teaching systems.

Multiple modalities can boost accuracy in the difficult task of Sign Language Recognition (SLR), however, each modality does not necessarily contribute the same quality of information. Current multi-modal approaches assign the same importance weightings to each modality, or set weightings based on unproven heuristics. This paper takes a systematic approach to find the optimal weights by performing grid search. Firstly, we create a multi-modal version of the RGB only WLASL100 data with additional hand crop and skeletal pose modalities. Secondly, we create a 3D CNN based weighted multi-modal sign language network (WMSLRnet). Finally, we run various grid searches to find the optimal weightings for each modality. We show that very minor adjustments in the weightings can have major effects on the final SLR accuracy. On WLASL100, we significantly outperform previous networks of similar design, and achieve high accuracy in SLR without highly complex pre-training schemes or extra data.