Not all images are interesting to people. People are drawn by interesting images and ignore tasteless ones. Image interestingness has the importance no less than other subjective image properties that have received significant research interest, but has not been systematically studied before. In this work we focus on visual and social aspects of image interestingness. We rely on crowdsourcing tools to survey human perceptions for these subjective properties and verify data by analyzing consistency and reliability. We show that people have an agreement when deciding if an image is interesting or not. We examine the correlation between the social, visual aspects of interestingness and aesthetics. By exploring the correlation, we find that: (1) Weak correlation between social interestingness and both of visual interestingness and image aesthetics indicates that the images frequently re-shared by people are not necessarily aesthetic or visually interesting. (2) High correlation between image aesthetics and visual interestingness implies aesthetic images are more likely to be visually interesting to people. Then we wonder what features of an image lead to social interestingness, e.g. receiving more likes and shares on social networking sites? We train classifiers to predict visual and social interestingness and investigate the contribution from different image features. We find that social and visual interestingness can be best predicted with color and texture, respectively, providing a way to manipulate social and visual liking of images with image features. Further, we investigate the correlation between social/visual image interestingness and image color. We find that colors with arousal effect show more frequently in images with higher social interestingness. That could be explained by previous studies for activation-related affect of colors and provides useful and important advice when advertising on social networking sites.

This work attempts to tackle the IBM grand challenge - seeing the daily life of New York City (NYC) in various perspectives by exploring rich and diverse social media content. Most existing works address this problem relying on single media source and covering limited life aspects. Because different social media are usually chosen for specific purposes, multiple social media mining and integration are essential to understand a city comprehensively. In this work, we first discover the similar and unique natures (e.g., attractions, topics) across social media in terms of visual and semantic perceptions. For example, Instagram users share more food and travel photos while Twitter users discuss more about sports and news. Based on these characteristics, we analyze a broad spectrum of life aspects - trends, events, food, wearing and transportation in NYC by mining a huge amount of diverse and freely available media (e.g., 1.6M Instagram photos, 5.3M Twitter posts). Because transportation logs are hardly available in social media, the NYC Open Data (e.g., 6.5B subway station transactions) is leveraged to visualize temporal traffic patterns. Furthermore, the experiments demonstrate that our approaches can effectively overview urban life with considerable technical improvement, e.g., having 16% relative gains in food recognition accuracy by a hierarchy cross-media learning strategy, reducing the feature dimensions of sentiment analysis by 10 times without sacrificing precision.

Image graph attracts attention from researchers due to the empirical success of graph based semi-supervised learning (SSL) methods and tasks such as image clustering, image navigation. Despite its simple structure, overwhelming scale of online images makes image graph construction a difficult problem. The challenge lies in time-consuming computation, and the difficulty of storing, processing the resulted graphs of huge size. We propose a novel method of image graph construction on MapReduce for large-scale data. The method consists of two stages: the first stage separates images into overlapping groups called image pools by using hash method, and the second computes pairwise similarities for pairs of images that are grouped into common pools. Both stages are performed on MapReduce. Our experiments on large-scale data show that the proposed method generates more sparse image graphs that reserve same or improved accuracy when comparing with previous method.

Microblogging as a new form of communication on Internet, has attracted the attention from researchers recently. Relying the real-time and conversational properties of microblogging, its users update their statuses and share experience within their the social network. Those characteristics also make microblogging an important tool for users to share or discuss real world events such as earth quake or sport game. In this paper, we propose a novel and flexible solution to detect and recognize real-time events from sport games based on analyzing the messages posted on microblogging services. We take Twitter as the experiment platform and collect a large-scale dataset of Twitter messages that are called tweets for 18 prominent sport games covering four types of sports in 2011. We also collect corresponding sport videos for those games. The proposed solution applies moving-threshold burst detection on the volume of tweets to detect highlights in sport games. A tf-idf-based weighting method is applied on the tweets within detected highlights for semantic extraction. According to the experiments we perform on the tweet and video datasets, we find that the proposed methods can achieve competent performance in sport event detection and recognition. Besides, our method can find non pre-defined tidbits that are difficult to detect in previous works.

In this work, we try to estimate the viewing directions of photos to browse landmarks with different views. To detect the viewing directions of the photos, we propose to leverage tags and geo-information associated with user-contributed photos to predict the locations of the landmarks. By utilizing the tags along with these photos, we demonstrate that the deviation between the estimated locations and the landmarks is about 30 meters on average. The detected locations of the landmarks can be used to further estimate the viewing directions in which the photos were taken. In the experiments, we show that the average error to the real viewing directions is less than 14 degrees. The estimated viewing directions enable illustrating landmarks from different views and recommending users the popular views for taking photos on the landmarks.

We present a scalable mobile video Question-Answering system with locally aggregated descriptors and random projection using user-generated videos all around the world for ACM Multimedia 2011 Technicolor challenge: "precise event recognition and description from video excerpts." Our proposed system takes a video excerpt as a query and explores its canonical semantics of that. We collect a public events video dataset containing 7 topics with 1963 YouTube videos to evaluate our proposed system. The experiment results show that our proposed video feature representation outperforms a state-of-the-art near-duplicate retrieval based on color histogram. The signature generated by random projection not only ensures real-time efficiency but also achieves a competitive MAP performance with original feature.

We propose a novel model for landmark discovery that locates region-based landmarks on map in contrast to the traditional point-based landmarks. The proposed method preserves more information and automatically identifies candidate regions on map by crowdsourcing geo-referenced photos. Gaussian kernel convolution is applied to remove noises and generate detected region. We adopt F1 measure to evaluate discovered landmarks and manually check the association between tags and regions. The experiment results show that more than 90% of attractions in the selected city can be correctly located by this method.

Semi-supervised learning is to exploit the vast amount of unlabeled data in the world. This paper proposes a scalable graph-based technique leveraging the distributed computing power of the MapReduce programming model. For a higher quality of learning, the paper also presents a multi-layer learning structure to unify both visual and textual information of image data during the learning process. Experimental results show the effectiveness of the proposed methods.

Efficient image search clustering is prominent for image search engines for exponentially growing photo collections. In this work, we propose an image search clustering approach which selects multiple canonical images from image search results and constructs image clusters in real time on an image sub-graph for the search results. The efficiency is achieved with the help of offline-computed image context graphs by distributed computing methods. Extending our prior works, we demonstrate the results of the proposed canonical image selection and preliminary outcomes of large-scale image graph construction in this proposal. We experiment in Flickr550 dataset, containing 540,321 Flickr photos.