Scene text spotting plays a pivotal role in image understanding.However, building a robust model for such a task necessitates substantial annotated data.Various efforts have been made to reduce the burden of extensive data labeling.In this paper, we focus on the minimum labor cost approach that solely relies on text-only annotations.Under this weakly supervised paradigm, existing methods encounter intrinsic difficulties since the location information is not available for training.To compensate, these methods often employ attention maps generated from models pre-trained on tasks such as text recognition or classification to predict spatial information.This approach, however, impedes the possibility of comprehensive end-to-end training and does not ensure optimal performance outcomes.What's more, the attention map for a single word tends to focus on the distinguishing areas, which often yields location predictions with suboptimal boundaries.To overcome these limitations, we introduce an innovative methodology that integrates pseudo-label generation to enable end-to-end training of the spotting network that optimizes text recognition and location estimation at the same time called TWIST.During the training, to address the problem associated with incomplete attention maps and obtain pseudo-labels that can cover the whole word, TWIST treats characters as elemental units.So the pseudo-label for each given text instance is generated by aggregating the inferred locations of their constituent characters, through a masked character prediction task.Then the generated pseudo-labels with corresponding textual content are used to further optimize the parameters of the spotting network.This integrated approach facilitates end-to-end training and achieves new state-of-the-art results in several public detection and end-to-end recognition benchmarks under text-only supervision.

Given a text query, the task of scene text retrieval aims at searching and localizing all the text instances that are contained in an image gallery. The state-of-the-art method learns a cross-modal similarity between the query text and the detected text regions in natural images to facilitate retrieval. However, this cross-modal approach still cannot well bridge the heterogeneity gap between the text and image modalities. In this paper, we propose a new paradigm that converts the task into a single-modality retrieval problem. Unlike previous works that rely on character recognition or embedding, we directly leverage pictorial information by rendering query text into images to learn the glyph feature of each character, which can be utilized to capture the similarity between query and scene text images. With the extracted visual features, we devise a synthetic label image guided feature alignment mechanism that is robust to different scene text styles and layouts. The modules of glyph feature learning, text instance detection, and visual matching are jointly trained in an end-to-end framework. Experimental results show that our proposed paradigm achieves the best performance in multiple benchmark datasets. As a side product, our method can also be easily generalized to support text queries with unseen characters or languages in a zero-shot manner.