Text-to-image diffusion models have made remarkable progress in generating high-quality images, but achieving fine-grained control over visual text remains a challenge. Existing methods fall short in enabling precise manipulation of font and color in visual text. To address this, we propose FC-Render, a method that introduces adaptive Font and Color control into a pre-trained TextDiffuser-2 model. Our approach consists of two stages: first, we develop independent font and color experts for decoupled control over visual text attributes. Then, we introduce an adaptive router that dynamically adjusts the involvement of each expert in font, color, or combined tasks, enabling precise fusion of font and color features. Through extensive experiments, we demonstrate that FC-Render enables fine-grained control over font and color while maintaining text rendering quality, providing new insights into the interaction between font, color, and spelling accuracy.

Large language models (LLMs) have made significant advancements in natural language processing, yet they still face challenges in tasks requiring deep comprehension of structurally complex and visually rich documents. Human reading patterns, captured through eye-tracking, provide valuable insights into how meaning is extracted from text, particularly in Visually Rich Documents (VRDs). In this work, we propose a novel approach to integrating eye-tracking data into LLMs to enhance reading strategies that more closely reflect human cognitive processes. By mimicking human gaze paths when reading a VRD, we explore how these insights can improve LLMs’ comprehension abilities. Our experiments demonstrate that LLMs enhanced with human-like reading orders outperform baseline models in VRD understanding tasks. These findings suggest that incorporating human-like reading strategies can bridge the gap between machine and human understanding. Overall, our results indicate that integrating eye-tracking data significantly enhances LLM performance, paving the way for more human-like document comprehension in future AI systems.

While promptable segmentation (e.g., SAM) has shown promise for various segmentation tasks, it still requires manual visual prompts for each object to be segmented. In contrast, task-generic promptable segmentation aims to reduce the need for such detailed prompts by employing only a task-generic prompt to guide segmentation across all test samples. However, when applied to Camouflaged Object Segmentation (COS), current methods still face two critical issues: 1) semantic ambiguity in getting instance-specific text prompts, which arises from insufficient discriminative cues in holistic captions, leading to foreground-background confusion; 2) semantic discrepancy combined with spatial separation in getting instance-specific visual prompts, which results from global background sampling far from object boundaries with low feature correlation, causing SAM to segment irrelevant regions. To mitigate the issues above, we propose RDVP-MSD, a novel training-free test-time adaptation framework that synergizes Region-constrained Dual-stream Visual Prompting (RDVP) via Multimodal Stepwise Decomposition Chain of Thought (MSD-CoT). MSD-CoT progressively disentangles image captions to eliminate semantic ambiguity, while RDVP injects spatial constraints into visual prompting and independently samples visual prompts for foreground and background points, effectively mitigating semantic discrepancy and spatial separation. Without requiring any training or supervision, RDVP-MSD achieves a state-of-the-art segmentation result on multiple COS benchmarks. The codes will be available at https://github.com/ycyinchao/RDVP-MSD.

Key-value relations are prevalent in Visually-Rich Documents (VRDs), often depicted in distinct spatial regions accompanied by specific color and font styles. These non-textual cues serve as important indicators that greatly enhance human comprehension and acquisition of such relation triplets. However, current document AI approaches often fail to consider this valuable prior information related to visual and spatial features, resulting in suboptimal performance, particularly when dealing with limited examples. To address this limitation, our research focuses on few-shot relational learning, specifically targeting the extraction of key-value relation triplets in VRDs. Given the absence of a suitable dataset for this task, we introduce two new few-shot benchmarks built upon existing supervised benchmark datasets. Furthermore, we propose a variational approach that incorporates relational 2D-spatial priors and prototypical rectification techniques. This approach aims to generate relation representations that are more aware of the spatial context and unseen relation in a manner similar to human perception. Experimental results demonstrate the effectiveness of our proposed method by showcasing its ability to outperform existing methods. This study also opens up new possibilities for practical applications.

Weakly supervised semantic segmentation (WSSS) using image-level labels is a challenging task, with relying on Class Activation Map (CAM) to derive segmentation supervision. Although many efficient single-stage solutions have been proposed, their performance is hindered by the inherent ambiguity of CAM. This paper introduces a new approach, dubbed ECA, to Exploit the self-supervised Vision Transformer, DINO, inducing the Class-aware semantic Affinity to overcome this limitation. Specifically, we introduce a Semantic Affinity Exploitation module (SAE). It establishes the class-agnostic affinity graph through the self-attention of DINO. Using the highly activated patches on CAMs as 'seeds', we propagate them across the affinity graph and yield the Class-aware Affinity Region Map (CARM) as supplementary semantic guidance. Moreover, the selection of reliable 'seeds' is crucial to the CARM generation. Inspired by the observed CAM inconsistency between the global and local views, we develop a CAM Correspondence Enhancement module (CCE) to encourage dense local-to-global CAM correspondences, advancing high-fidelity CAM for seed selection in SAE. Our experimental results demonstrate that ECA effectively improves the model's object pattern understanding. Remarkably, it outperforms state-of-the-art alternatives on the PASCAL VOC 2012 and MS COCO 2014 datasets, achieving 90.1% upper bound performance compared to its fully supervised counterpart. Code is available at https://github.com/Wu0409/ECA.

Few-shot link prediction is an important recent research direction in the field of knowledge graph.To learn better entity and relation representations, previous works utilize the neighborhood information of entities but they ignore the associated relations.More than this, the link invovles few-shot entities cannot be well linked by these methods.Therefore, this paper proposes a novel method that dynamically aggregates local neighbourhood information from entities and relations and introduces the information bottleneck principle to filter irrelevant features to better capture the feature of few-shot entities.Experimental results on three benchmark datasets demonstrate that our model outperforms other models and that our approach can effectively improve the model performance on those entities in the tail of the long-tailed distribution.

Taxonomy is indispensable in understanding natural language. A variety of large scale, usage-based, data-driven lexical taxonomies have been constructed in recent years.Hypernym-hyponym relationship, which is considered as the backbone of lexical taxonomies can not only be used to categorize the data but also enables generalization. In particular, we focus on one of the most prominent properties of the hypernym-hyponym relationship, namely, transitivity, which has a significant implication for many applications. We show that, unlike human crafted ontologies and taxonomies, transitivity does not always hold in data-drivenlexical taxonomies. We introduce a supervised approach to detect whether transitivity holds for any given pair of hypernym-hyponym relationships. Besides solving the inferencing problem, we also use the transitivity to derive new hypernym-hyponym relationships for data-driven lexical taxonomies. We conduct extensive experiments to show the effectiveness of our approach.