Current sign language translation (SLT) approaches often rely on gloss-based supervision with Connectionist Temporal Classification (CTC), limiting their ability to handle non-monotonic alignments between sign language video and spoken text. In this work, we propose a novel method combining joint CTC/Attention and transfer learning. The joint CTC/Attention introduces hierarchical encoding and integrates CTC with the attention mechanism during decoding, effectively managing both monotonic and non-monotonic alignments. Meanwhile, transfer learning helps bridge the modality gap between vision and language in SLT. Experimental results on two widely adopted benchmarks, RWTH-PHOENIX-Weather 2014 T and CSL-Daily, show that our method achieves results comparable to state-of-the-art and outperforms the pure-attention baseline. Additionally, this work opens a new door for future research into gloss-free SLT using text-based CTC alignment.

Existing arbitrary-shaped text (ArT) spotting models have used coordinate-based error (CBE) loss to optimize polygon region detection. Optimization using CBE loss requires a fine-grained ground truth (FineGT), which is a fixed-length list of polygon vertices. However, not all datasets have FineGT, and creating FineGT is costly. In this paper, we propose Polygon Pixel Intersection over Union (PPIoU), which shows the similarity of polygon regions and can be used for neural network training without FineGT. Unlike CBE loss, PPIoU loss represents the region-based error for polygons and is not affected by the number of polygon vertices. PPIoU loss allows the use of multiple datasets containing polygons with different numbers of vertices, making it easy to increase the amount of training data. This is a useful feature since more training data generally improves accuracy. We show that the PPIoU loss can easily improve the accuracy of existing text spotting models.

A transformer model is used in general tasks such as pre-trained language models and specific tasks including machine translation. Such a model mainly relies on positional encodings (PEs) to handle the sequential order of input vectors. There are variations of PEs, such as absolute and relative, and several studies have reported on the superiority of relative PEs. In this paper, we focus on analyzing in which part of a transformer model PEs work and the different characteristics between absolute and relative PEs through a series of experiments. Experimental results indicate that PEs work in both self- and cross-attention blocks in a transformer model, and PEs should be added only to the query and key of an attention mechanism, not to the value. We also found that applying two PEs in combination, a relative PE in the self-attention block and an absolute PE in the cross-attention block, can improve translation quality.

Interest in emotion recognition in conversations (ERC) has been increasing in various fields, because it can be used to analyze user behaviors and detect fake news.Many recent ERC methods use graph-based neural networks to take the relationships between the utterances of the speakers into account.In particular, the state-of-the-art method considers self-and inter-speaker dependencies in conversations by using relational graph attention networks (RGAT).However, graph-based neural networks do not take sequential information into account.In this paper, we propose relational position encodings that provide RGAT with sequential information reflecting the relational graph structure.Accordingly, our RGAT model can capture both the speaker dependency and the sequential information.Experiments on four ERC datasets show that our model is beneficial to recognizing emotions expressed in conversations.In addition, our approach empirically outperforms the state-ofthe-art on all of the benchmark datasets.

Taro Miyazaki, Kiminobu Makino, Yuka Takei, Hiroki Okamoto, Jun Goto. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

As information support to deaf and hard of hearing people who are viewing sports programs, we have developed a sign language support system. The system automatically generates Japanese Sign Language (JSL) computer graphics (CG) animation and subtitles from prepared templates of JSL phrases corresponding to fixed format game data. To verify the system's performance, we carried out demonstration experiments on the generation and displaying of contents using real-time match data from actual games. From the experiment results we concluded that the automatically generated JSL CG is practical enough for understanding the information. We also found that among several display methods, the one providing game video and JSL CG on a single tablet screen was most preferred in this small-scale experiment.

To expand the services that are based on Japanese Sign Language (JSL) for deaf and hard of hearing people, we developed a support system for viewing sports program. The system provides sign language computer graphics (CG) animations and other auxiliary information such as text, image, and notifications automatically generated from game metadata. Results obtained from gathered opinions showed that the system is effective for understanding the situation when a game is interrupted.