There are many cues that can be used to curate media from social networking websites. Beyond metadata, group behavior provide a strong community-based signal for surfacing images, which we show in a user-defined curatorial task. In a departure from mirco-task crowdwork, we observe that the curation inherent in online photo communities guides the discoverability and consumption of the media, which in turn provides a strong signal that can be used in new editorial tasks in a community-supervised manner. We use this approach in tandem with other more conventional multimedia methods (i.e.\ computer vision and contextual metadata) to form a broad multimodal approach to retrieval and recommendation. We present a large-scale system implementation on a real-world curative task for weather images on a web-scale dataset. Finally, we conduct an evaluation of this system using professional editors and find substantial improvements in editorial efficiency.

This paper considers the problem of fine-grained image recognition with a growing vocabulary. Since in many real world applications we often have to add a new object category or visual concept with just a few images to learn from, it is crucial to develop a method that is able to generalize the recognition model from existing classes to new classes. Deep convolutional neural networks are capable of constructing powerful image representations; however, these networks usually rely on a logistic loss function that cannot handle the incremental learning problem. In this paper, we present a new method that can efficiently learn a new class given only a limited number of training examples, which we evaluate on the problems of food and clothing recognition. To illustrate the performance of our proposed method on the task of recognizing different kinds of food, when using only 1.3\% of training examples per category we achieved about 73\% of the performance (as measured by F1-score) compared to when using all available training data.

The world is a big place. At any given instant something is happening somewhere, but even when nothing is going on people still find ways to generate multimedia data, ranging from social media posts, to photos and videos. A substantial number of these media objects is associated with a location, and in an increasingly mobile and connected world (both in terms of people and devices), this number is only bound to get larger. Yet, in the multimedia literature we observe that many researchers often unwittingly treat the geospatial dimension as if it were a regular feature dimension, despite it requiring special attention. In order to avoid pitfalls and to steer clear of erroneous conclusions, this tutorial aims to teach researchers and students how geotagged multimedia data differs from regular data and to educate them on best practices when dealing with such data. We will cover the lifecycle of geotagged data in multimedia research, where the topics range from how this kind of data is represented, processed, analyzed, and visualized. The tutorial requires both passive and active involvement, where we not only present the material, but the attendees also get the opportunity to interact with it using a variety of open source data and tools that we have prepared using a virtual machine.

Leveraged wisely, new datasets can inspire new multimedia methods and algorithms, as well as catalyze innovations in how their efficacy, efficiency, and generalizability can be evaluated. The availability of very large multimedia datasets like the Yahoo-Flickr Creative Commons 100 Million has offered unique opportunities for advancing the state of the art in multimedia processing, analysis, search, and visualization. The Multimedia Commons Initiative has been developing a community around the YFCC100M, including associated annotation and evaluation efforts. In addition to research in several multimedia subfields, including computer vision, image processing, and video content analysis, the YFCC100M and Multimedia Commons resources have been used in various competitions and benchmarks, such as the MediaEval Placing Task and the ACM Multimedia Grand Challenge competition. With additional annotation and curation, the data has the potential to enable major leaps forward in research.

Points of interest are an important requirement for location-based services, yet they are editorially curated and maintained, either professionally or through community. Beyond the laborious manual annotation task, further complications arise as points of interest may appear, relocate, or disappear over time, and may be relevant only to specific communities. To assist, complement, or even replace manual annotation, we propose a novel method for the automatic localization of points of interest depicted in photos taken by people across the world. Our technique exploits the geographic coordinates and the compass direction supplied by modern cameras, while accounting for possible measurement errors due to the variability in accuracy of the sensors that produced them. We statistically demonstrate that our method significantly outperforms techniques from the research literature on the task of estimating the geographic coordinates and geographic footprints of points of interest in various cities, even when photos are involved in the estimation process that do not show the point of interest at all.

In this workshop we aim to bring together practitioners and researchers to discuss their recent breakthroughs and the challenges with addressing spatial and temporal information access, both from the algorithmic and the architectural perspectives.

Geotagged data can be used to describe regions in the world and discover local themes. However, not all data produced within a region is necessarily specifically descriptive of that area. To surface the content that is characteristic for a region, we present the geographical hierarchy model (GHM), a probabilistic model based on the assumption that data observed in a region is a random mixture of content that pertains to different levels of a hierarchy. We apply the GHM to a dataset of 8 million Flickr photos in order to discriminate between content (i.e. tags) that specifically characterizes a region (e.g. neighborhood) and content that characterizes surrounding areas or more general themes. Knowledge of the discriminative and non-discriminative terms used throughout the hierarchy enables us to quantify the uniqueness of a given region and to compare similar but distant regions. Our evaluation demonstrates that our model improves upon traditional Naive Bayes classification by 47% and hierarchical TF-IDF by 27%. We further highlight the differences and commonalities with human reasoning about what is locally characteristic for a neighborhood, distilled from ten interviews and a survey that covered themes such as time, events, and prior regional knowledge.

People take photos all over the world at all times of day; each photo depicting a place and a moment worth capturing. In the context of multimedia analysis and social computing, accurate location and time information about where and when these photos were taken is of importance for understanding event semantics, image content and many other purposes. While location information associated with photos is known to be relatively accurate, time is not. From a sample of 10 million public Flickr photos, we observe that 37% of the photos differ more than an hour between their camera timestamps and GPS timestamps with respect to local time at the locations where the photos were taken. Erroneous time information may adversely influence the correctness of any kind of temporal analysis that relies on camera timestamps, as well as research and real-world applications that require accurate knowledge of when and where photos were captured. In light of our observations we propose a simple yet effective metadata-only technique for improving the accuracy of camera timestamps.

The world wide web is filled with billions of images and duplicates of images can frequently be found on many websites. These duplicates can be exact copies or differ slightly in their visual content. In this paper we provide a comparative study on how well content-based duplicate image detection methods are able to detect the duplicates of a query image. We conduct a survey to better understand in which ways such images on the internet differ from each other and use these observations to form a realistic and challenging duplicate image detection scenario. The methods we evaluate in our study are representative techniques from the research literature. In our evaluation, we target the performance of each method in relation to their descriptor size, description time and matching time, to assess their feasibility of application to large image collections (> 1 million).

Social media responses to news have increasingly gained in importance as they can enhance a consumer's news reading experience, promote information sharing and aid journalists in assessing their readership's response to a story. Given that the number of responses to an online news article may be huge, a common challenge is that of selecting only the most interesting responses for display. This paper addresses this challenge by casting message selection as an optimization problem. We define an objective function which jointly models the messages' utility scores and their entropy. We propose a near-optimal solution to the underlying optimization problem, which leverages the submodularity property of the objective function. Our solution first learns the utility of individual messages in isolation and then produces a diverse selection of interesting messages by maximizing the defined objective function. The intuitions behind our work are that an interesting selection of messages contains diverse, informative, opinionated and popular messages referring to the news article, written mostly by users that have authority on the topic. Our intuitions are embodied by a rich set of content, social and user features capturing the aforementioned aspects. We evaluate our approach through both human and automatic experiments, and demonstrate it outperforms the state of the art. Additionally, we perform an in-depth analysis of the annotated ``interesting'' responses, shedding light on the subjectivity around the selection process and the perception of interestingness.

We propose a novel algorithm for uncovering the colloquial boundaries of locally characterizing regions present in collections of labeled geospatial data. We address the problem by first modeling the data using scale-space theory, allowing us to represent it simultaneously across different scales as a family of increasingly smoothed density distributions. We then derive region boundaries by applying localized label weighting and image processing techniques to the scale-space representation of each label. Important insights into the data can be acquired by visualizing the shape and size of the resulting boundaries for each label at multiple scales. We demonstrate our technique operating at scale by discovering the boundaries of the most geospatially salient tags associated with a large collection of georeferenced photos from Flickr and compare our characterizing regions that emerge from the data with those produced by a recent technique from the research literature.

We present a system that supports zoomable browsing and exploration of photos taken across the globe. Our system is based on a novel algorithm that automatically uncovers the colloquial boundaries of regions that are characteristic for individual tags used in a large collection of geo-referenced photos. We first model the data using scale-space theory, which allows us to represent it simultaneously across different scales as a family of increasingly smoothed density distributions, after which we derive the region boundaries by applying image analysis techniques to the scale-space representation of each tag. The interface visualizes the shape and size of the resulting boundaries for each tag along the dimensions of space and time across multiple scales, giving the user the ability to explore the world as patchwork of dynamic characterizing geographic tag regions and to browse through their associated photos.

TOP-SURF is an image descriptor that combines interest points with visual words, resulting in a high performance yet compact descriptor that is designed with a wide range of content-based image retrieval applications in mind. TOP-SURF offers the flexibility to vary descriptor size and supports very fast image matching. In addition to the source code for the visual word extraction and comparisons, we also provide a high level API and very large pre-computed codebooks targeting web image content for both research and teaching purposes.

Experiential image retrieval systems aim to provide the user with a natural and intuitive search experience. The goal is to empower the user to navigate large collections based on his own needs and preferences, while simultaneously providing him with an accurate sense of what the database has to offer. In this paper we integrate a new browsing mechanism called deep exploration with the proven technique of retrieval by relevance feedback. In our approach, relevance feedback focuses the search on relevant regions, while deep exploration facilitates transparent navigation to promising regions of feature space that would normally remain unreachable. Optimal feature weights are determined automatically based on the evidential support for the relevance of each single feature. To achieve efficient refinement of the search space, images are ranked and presented to the user based on their likelihood of being useful for further exploration.

Our goal is to determine if artificially imagined or synthesized images can be beneficial to interactive visual search. We present a novel approach for using artificially imagined images in relevance feedback. Since the search engine constructs the synthetic images itself, any feedback given by the user on these images allows it to obtain a better understanding of what the user is looking for than it would from feedback on database images alone. We evaluated and compared our image synthesis approach with a normal Rocchio-based system on a well-known texture database with real users.