Referring expression comprehension (REC) aims to identify and locate a specific object in visual scenes referred to by a natural language expression. Existing studies of REC only focus on basic visual attributes and neglect scene text. Since scene text has the functions of object identification and disambiguation, it is naturally and frequently used to refer to objects. However, existing methods do not explicitly recognize text in images and fail to align scene text mentioned in expressions with the text shown in images, resulting in object localization errors. This article takes the first step toward addressing these limitations. First, we introduce a new task called scene-text oriented referring expression comprehension, which aims to align visual cues and textual semantics of scene text with referring expressions and visual contents. Second, we propose a scene text awareness network that can bridge the gap between texts from two modalities by grounding visual representations of expression-correlated scene texts. Specifically, we propose a correlated text extraction module to solve the problem of lacking semantic understanding, and a correlated region activation module to address the fixed alignment problem and absent alignment problem. These modules ensure that the proposed method focuses on local regions that are most relevant to scene text, thus mitigating the misalignment of scene text with irrelevant regions. Third, to conduct quantitative evaluations, we establish a new benchmark dataset called RefText. Experimental results demonstrate that the proposed method can effectively comprehend scene-text oriented referring expressions and achieves excellent performance.

Referring expression comprehension aims at grounding the object in an image referred to by the expression. Scene text that serves as an identifier has a natural advantage in referring to objects. However, existing methods only consider the text in the expression, but ignore the text in the image, leading to a mismatch. In this paper, we propose a novel model that can recognize the scene text. We assign the extracted scene text to its corresponding visual region and ground the target object guided by expression. Experimental results on two benchmarks demonstrate the effectiveness of our model.

Aspect detection aims to identify aspects of reviews and is an essential up-stream task of opinion mining and so on. However, existing weakly-supervised methods suffer from lacking the ability of identifying implicit aspects with infrequent aspect terms and "Misc" aspects. To tackle these problems, we propose to enhance the representation of segment with external knowledge by a weakly-supervised method. Experiments demonstrate the effectiveness of our model and the improvement by incorporating external knowledge.

In this paper, we propose a one-stage approach to improve referring expression comprehension (REC) which aims at grounding the referent according to a natural language expression. We observe that humans understand referring expressions through a fine-to-coarse bottom-up way, and bidirectionally obtain vision-language information between image and text. Inspired by this, we define the language granularity and the vision granularity. Otherwise, existing methods do not follow the mentioned way of human understanding in referring expression. Motivated by our observation and to address the limitations of existing methods, we propose a bottom-up and bidirectional alignment (BBA) framework. Our method constructs the cross-modal alignment starting from fine-grained representation to coarse-grained representation and bidirectionally obtains vision-language information between image and text. Based on the structure of BBA, we further propose a progressive visual attribute decomposing approach to decompose visual proposals into several independent spaces to enhance the bottom-up alignment framework. Experiments on five benchmark datasets of RefCOCO, RefCOCO+, ReferItGame, RefCOCOg and Flick30K show that our approach obtains +2.16%, +4.47%, +2.85%, +3.44%, and +2.91% improvements over the one-stage SOTA approaches, which validates the effectiveness of our approach.

In this paper, a text-based characterization combined with numerical features for Social Media Headline Prediction (SMHP) is proposed. Description of images, users' emotions and opinions are all described in text, our text-based characterization learns these important features by training a Doc2vec model. Numerical features of social cues contain general characteristics of social media headline, we build an effective method to extract numerical features. Experiments conducted on real-world SMHP dataset manifest the effectiveness of the proposed approach, which achieves the following performance: Spearmanr's Rho: 0.4559, MAE:1.9797.

A hybrid model for social media popularity prediction is proposed by combining Convolutional Neural Network (CNN) with XGBoost. The CNN model is exploited to learn high-level representations from the social cues of the data. These high-level representations are used in XGBoost to predict the popularity of the social posts. We evaluate our approach on a real-world Social Media Prediction (SMP) dataset, which consists of 432K Flickr images. The experimental results show that the proposed approach is effective, achieving the following performance: Spearman's Rho: 0.7406, MSE: 2.7293, MAE: 1.2475.