Every day, vast amounts of data are uploaded to various social-sharing websites. Each social-sharing website has its own media dataset. Recently, mining media datasets has shown great potential for our daily lives, e.g., earthquake detection. Generally, different datasets have different characteristics. Combining different datasets is capable of achieving better performance than using any dataset independently, particularly if the datasets can compensate for each other. The resulting performance, however, depends on the fusion method. Effectively combining different datasets is challenging. As a solution to this challenge, this paper presents a generic two-stage framework for events of interest. Specifically, the first stage normalizes the contents of different datasets to make them comparable; then, the second stage combines the normalized contents for a ranked event list using graph-based algorithms. Practically, this paper unifies a flow-based media dataset and a check-in-based media dataset. Based on the precision for the top n events, the experimental results demonstrate that the proposed framework can achieve better performance in finding events associated with sports, local festivals, concerts, and exhibitions compared with a state-of-the-art approach that uses one dataset alone.

Nowadays multi-track conferences impose great difficulty to attendees in making a day plan of attending relevant sessions/papers, because a large number of sessions are scheduled on a single day and many of them are at the same time. To address this problem, we introduce MatPlanner, a mobile application, which provides an alternate way of scheduling a day in a conference by understanding the attendee's interest and preferences. MatPlanner learns event-topic interest matrix from few sampled events selected by each user and then recommend relevant events and display the final output. More importantly, MatPlanner's interface enables users to resolve conflicting events by showing them at the same panel which makes it easy to compare. Interview results suggested a high level of satisfaction among participants in scheduling a day.

The goal of the research is to discover and summarize data from the emerging social media into information of interests. Specifically, leveraging user-contributed data from cross-domain social media, the idea is to perform multi-modal learning for a given photo, aiming to present people’s description or comments, geographical information, and events of interest, closely related to the photo. These information then can be used for various purposes, such as being a real-time guide for the tourists to improve the quality of tourism. As a result, this research investigates modern challenges of image annotation, image retrieval, and cross-media mining, followed by presenting promising ways to conquer the challenges.

Recently, the fast growth of social media communities and mobile devices encourages more people to share their media data online than ever before. Analyzing data and summarizing data into useful information have become increasingly popular and important for modern society. Given a set of event keywords and a dataset, this paper performs event summarization, aiming to discover and summarize what people may concern for each event from the given dataset. More specifically, this paper extracts latent sub-events with diverse and representative attributes for each given event. This paper proposes effective methods on detecting events with (1) human attribute discovery, such as human pose and clothes, (2) scene analysis, (3) image aspect discovery, and (4) temporal and semantic analysis, to provide people different perspectives for the events they are interested in. For practical implementation, this paper studied and conducted experiments on YFCC100M, which is a dataset with 100 million of photos and videos, provided by Yahoo!. Finally, a comprehensive and complete system is created accordingly to support diverse event summarization.

This work attempts to tackle the IBM grand challenge - seeing the daily life of New York City (NYC) in various perspectives by exploring rich and diverse social media content. Most existing works address this problem relying on single media source and covering limited life aspects. Because different social media are usually chosen for specific purposes, multiple social media mining and integration are essential to understand a city comprehensively. In this work, we first discover the similar and unique natures (e.g., attractions, topics) across social media in terms of visual and semantic perceptions. For example, Instagram users share more food and travel photos while Twitter users discuss more about sports and news. Based on these characteristics, we analyze a broad spectrum of life aspects - trends, events, food, wearing and transportation in NYC by mining a huge amount of diverse and freely available media (e.g., 1.6M Instagram photos, 5.3M Twitter posts). Because transportation logs are hardly available in social media, the NYC Open Data (e.g., 6.5B subway station transactions) is leveraged to visualize temporal traffic patterns. Furthermore, the experiments demonstrate that our approaches can effectively overview urban life with considerable technical improvement, e.g., having 16% relative gains in food recognition accuracy by a hierarchy cross-media learning strategy, reducing the feature dimensions of sentiment analysis by 10 times without sacrificing precision.

In this work, we target at solving the Bing challenge provided by Microsoft. The task is to design an effective and efficient measurement of query terms in describing the images (image-query pairs) crawled from the web. We observe that the provided image-query pairs (e.g., text-based image retrieval results) are usually related to their surrounding text; however, the relationship between image content seems to be ignored. Hence, we attempt to integrate the visual information for better ranking results. In addition, we found that plenty of query terms are related to people (e.g., celebrity) and user might have similar queries (click logs) in the search engine. Therefore, in this work, we propose a relevance association by investigating the effectiveness of different auxiliary contextual cues (i.e., face, click logs, visual similarity). Experimental results show that the proposed method can have 16% relative improvement compared to the original ranking results. Especially, for people-related queries, we can further have 45.7% relative improvement.

Image graph attracts attention from researchers due to the empirical success of graph based semi-supervised learning (SSL) methods and tasks such as image clustering, image navigation. Despite its simple structure, overwhelming scale of online images makes image graph construction a difficult problem. The challenge lies in time-consuming computation, and the difficulty of storing, processing the resulted graphs of huge size. We propose a novel method of image graph construction on MapReduce for large-scale data. The method consists of two stages: the first stage separates images into overlapping groups called image pools by using hash method, and the second computes pairwise similarities for pairs of images that are grouped into common pools. Both stages are performed on MapReduce. Our experiments on large-scale data show that the proposed method generates more sparse image graphs that reserve same or improved accuracy when comparing with previous method.

With the growth of mobile devices, the needs for location-based services are emerging. Taking the advantage of the GPS information, we can roughly estimate a user's location. However, it is necessary to leverage extra information (e.g., photos) to precisely locate the object of interest through mobile devices for further applications such as mobile search. Users can simply take a picture (with GPS enabled) of an interesting target to retrieve the building information. Therefore, the raise of real-time building recognition or retrieval system becomes a challenging problem. The most recent approaches are to recognize buildings by the street-view images; however, the query photos from mobile devices usually contain different lighting conditions. In order to provide a more robust city-view image retrieval system, we propose to augment the visual diversity of database images by integrating the context-rich user-contributed photos from social media. Preliminary experimental results show that the street-view images can provide different angles of the target whereas the user-contributed photos can enhance the diversity of the target. Besides, for the real-time retrieval system, we also combine both visual and GPS constraints in the retrieval process on inverted indexing so that we can achieve a real-time retrieval system.

Semi-supervised learning is to exploit the vast amount of unlabeled data in the world. This paper proposes a scalable graph-based technique leveraging the distributed computing power of the MapReduce programming model. For a higher quality of learning, the paper also presents a multi-layer learning structure to unify both visual and textual information of image data during the learning process. Experimental results show the effectiveness of the proposed methods.

OBJECTIVE: User data and information about anatomy education were used to guide development of a learning environment that is efficient and effective. The research question focused on how to design instructional software suitable for the educational goals of different groups of users of the Visible Human data set. The ultimate goal of the study was to provide options for students and teachers to use different anatomy learning modules corresponding to key topics, for course work and professional training. DESIGN: The research used both qualitative and quantitative methods. It was driven by the belief that good instructional design must address learning context information and pedagogic content information. The data collection emphasized measurement of users' perspectives, experience, and demands in anatomy learning. MEASUREMENT: Users' requirements elicited from 12 focus groups were combined and rated by 11 researchers. Collective data were sorted and analyzed by use of multidimensional scaling and cluster analysis. RESULTS: A set of functions and features in high demand across all groups of users was suggested by the results. However, several subgroups of users shared distinct demands. The design of the learning modules will encompass both unified core components and user-specific applications. The design templates will allow sufficient flexibility for dynamic insertion of different learning applications for different users. CONCLUSION: This study describes how users' requirements, associated with users' learning experiences, were systematically collected and analyzed and then transformed into guidelines informing the iterative design of multiple learning modules. Information about learning challenges and processes was gathered to define essential anatomy teaching strategies. A prototype instrument to design and polish the Visible Human user interface system is currently being developed using ideas and feedback from users.