Document layout analysis is a crucial step for intelligent document understanding. However, many existing methods primarily focus on the visual aspects and overlook the textual features of documents. Although document pre-trained models utilize multi-modal features during the pre-training phase, they tend to operate as a unimodal pipeline when it comes to layout analysis tasks. Furthermore, current multi-modal methods perform worse than unimodal detectors on complex layout analysis datasets. To address these limitations, we propose an effective and pluggable multi-modal fusion approach named M2Doc, which fuses visual and textual features for better layout detection. M2Doc contains two pluggable multi-modal fusion modules, early-fusion and late-fusion, which align and fuse visual and textual features at the pixel level and block level. Benefitting from the concision and effectiveness of M2Doc, it can be easily applied to various detectors for better layout detection, including two-stage and end-to-end object detectors. Our experimental results demonstrate significant performance improvements in detectors equipped with M2Doc on datasets such as DocLayNet (+11.3 mAP) and M6Doc (+1.9 mAP). Furthermore, through the integration of the DINO detector with M2Doc, we achieve state-of-the-art results on DocLayNet (89.0 mAP), M6Doc (69.9 mAP), and PubLayNet (95.5 mAP). The code will be publicly released at https://github.com/johnning2333/M2Doc.

Foundation model CLIP has garnered significant attention worldwide in recent years due to its tremendous capabilities in various domains of deep learning. However, the knowledge acquired from image-text pairs in CLIP cannot be sufficiently transferred to dense prediction tasks like referring image segmentation. In this paper, we propose an effective framework, termed RISC, to thoroughly exploit the potential of CLIP to boost high-quality referring image segmentation. Specifically, to transfer the remarkable knowledge from CLIP to the pixel-text level, we introduce a CLIP-driven Dense Decoder to integrate features at different scales and modalities from CLIP in a fine-grained manner. Furthermore, to maximize the vision-text matching capabilities from CLIP, a Lightweight Pixel Refiner is proposed to generate masks with distinct boundaries through point sampling and matching strategies. Extensive experiments demonstrate that our approach outperforms the previous state- of-the-art methods by a notable margin on three widely-used datasets (RefCOCO, RefCOCO+ and RefCOCOg).