The diagnosis of colon polyps is important for the prevention of colorectal cancer. Polyp segmentation, however, is still a challenging problem given that recent medical computer-aided equipment suffers from situations of polyp variations in terms of size, color, texture, and poor illuminations in endoscopy videos. These obstacles hinder the prediction of polyp boundaries. Inspired by the observation that the values of pixels on the border region change more sharply than others, we propose the oriented-derivative (OD) representation to capture the relationship between pixels and the boundary region given distance and orientation. To adaptively use the proposed representation in arbitrary frameworks, we design plug-in modules to learn the representation and aggregate features to improve the accuracy of boundary predictions in the polyp segmentation task, which can be implemented in frameworks including the encoder-decoder and top-down architectures. Extensive experimental results show the improvement from the proposed oriented-derivative representation for the polyp segmentation task and the extendibility of our proposed modules in different architectures. Our methods achieved an improvement ranging from 0.3% to 2.5% (mDice) compared with the baseline on five publicly available datasets, includingKvasir, CVC-ClinicDB, EndoScene, CVC-ColonDB, andETIS.

Visual appearance is considered to be the most important cue to understand images for cross-modal retrieval, while sometimes the scene text appearing in images can provide valuable information to understand the visual semantics. Most of existing cross-modal retrieval approaches ignore the usage of scene text information and directly adding this information may lead to performance degradation in scene text free scenarios. To address this issue, we propose a full transformer architecture to unify these cross-modal retrieval scenarios in a single Vision and Scene Text Aggregation framework (ViSTA). Specifically, ViSTA utilizes transformer blocks to directly encode image patches and fuse scene text embedding to learn an aggregated visual representation for cross-modal retrieval. To tackle the modality missing problem of scene text, we propose a novel fusion token based transformer aggregation approach to exchange the necessary scene text information only through the fusion token and concentrate on the most important features in each modality. To further strengthen the visual modality, we develop dual contrastive learning losses to embed both image-text pairs and fusion-text pairs into a common cross-modal space. Compared to existing methods, ViSTA enables to aggregate relevant scene text semantics with visual appearance, and hence improve results under both scene text free and scene text aware scenarios. Experimental results show that ViSTA outperforms other methods by at least 8.4% at Recall@ 1 for scene text aware retrieval task. Compared with state-of-the-art scene text free retrieval methods, ViSTA can achieve better accuracy on Flicker30K and MSCOCO while running at least three times faster during the inference stage, which validates the effectiveness of the proposed framework.