With the rapid advancement of image generation, visual text editing using natural language instructions has received increasing attention. The main challenge of this task is to fully understand the instruction and reference image, and thus generate visual text that is style-consistent with the image. Previous methods often involve complex steps of specifying the text content and attributes, such as font size, color, and layout, without considering the stylistic consistency with the reference image. To address this, we propose UM-Text, a unified multimodal model for context understanding and visual text editing by natural language instructions. Specifically, we introduce a Visual Language Model (VLM) to process the instruction and reference image, so that the text content and layout can be elaborately designed according to the context information. To generate an accurate and harmonious visual text image, we further propose the UM Encoder to combine the embeddings of various condition information, where the combination is automatically configured by VLM according to the input instruction. During training, we propose a regional consistency loss to offer more effective supervision for glyph generation on both latent and RGB space, and design a tailored three-stage training strategy to further enhance model performance. In addition, we contribute the UM-DATA-200K, a large-scale visual text image dataset on diverse scenes for model training. Extensive qualitative and quantitative results on multiple public benchmarks demonstrate that our method achieves state-of-the-art performance.

Abstract Extreme multi-label text classification (XMTC) annotates related labels for unknown text from large-scale label sets. Transformer-based methods have become the dominant approach for solving the XMTC task due to their effective text representation capabilities. However, the existing Transformer-based methods fail to effectively exploit the correlation between labels in the XMTC task. To address this shortcoming, we propose a novel model called TLC-XML, i.e., a Transformer with label correlation for extreme multi-label text classification. TLC-XML comprises three modules: Partition, Matcher and Ranker. In the Partition module, we exploit the semantic and co-occurrence information of labels to construct the label correlation graph, and further partition the strongly correlated labels into the same cluster. In the Matcher module, we propose cluster correlation learning, which uses the graph convolutional network (GCN) to extract the correlation between clusters. We then introduce these valuable correlations into the classifier to match related clusters. In the Ranker module, we propose label interaction learning, which aggregates the raw label prediction with the information of the neighboring labels. The experimental results on benchmark datasets show that TLC-XML significantly outperforms state-of-the-art XMTC methods.

Document-level Relation Extraction (DocRE)intends to extract relationships from documents.Some works introduce logic constraints into DocRE, addressing the issues of opacity and weak logic in original DocRE models.However, they only focus on forward logic constraints and the rules mined in these works often suffer from pseudo rules with high standardconfidence but low support.In this paper, we proposes Bidirectional Constraints of Beta Rules(BCBR), a novel logic constraint framework.BCBR first introduces a new rule miner which model rules by beta contribtion.Then forward and reverse logic constraints are constructed based on beta rules.Finally, BCBR reconstruct rule consistency loss by bidirectional constraints to regulate the output of the DocRE model.Experiments show that BCBR outperforms original DocRE models on relation extraction performance (∼2.7 F1) and logic consistency(∼3.1 Logic).Furthermore, BCBR consistently outperforms two other logic constraint frameworks.Our code is available at https://github.com/Louisliu1999/BCBR.