The popularity of mobile devices equipped with various cameras has revolutionized modern photography. People are able to take photos and share their experiences anytime and anywhere. However, taking a high quality photograph via mobile device remains a challenge for mobile users. In this paper we investigate a photography model to assist mobile users in capturing high quality photos by using both the rich context available from mobile devices and crowdsourced social media on the Web. The photography model is learned from community-contributed images on the Web, and dependent on user's social context. The context includes user's current geo-location, time (i.e., time of the day), and weather (e.g., clear, cloudy, foggy, etc.). Given a wide view of scene, our socialized mobile photography system is able to suggest the optimal view enclosure (composition) and appropriate camera parameters (aperture, ISO, and exposure time). Extensive experiments have been performed for eight well-known hot spot landmark locations where sufficient context tagged photos can be obtained. Through both objective and subjective evaluations, we show that the proposed socialized mobile photography system can indeed effectively suggest proper composition and camera parameters to help the user capture high quality photos.

The ongoing revolution in media consumption from traditional PCs to the pervasiveness of mobile devices is driving the adoption of social media in our daily lives. More and more people are using their mobile devices to enjoy social media content while on the move. However, mobile display constraints create challenges for presenting and authoring the rich media content on screens with limited display size. This paper presents an innovative system to automatically generate magazine-like social media visual summaries, which is called "snippet," for efficient mobile browsing. The system excerpts the most salient and dominant elements, i.e., a major picture element and a set of textual elements, from the original media content, and composes these elements into a text overlaid image by maximizing information perception. In particular, we investigate a set of aesthetic rules and visual perception principles to optimize the layout of the extracted elements by considering display constraints. As a result, browsing the snippet on mobile devices is just like quickly glancing at a magazine. To the best of our knowledge, this paper represents one of the first attempts at automatic social media snippet generation by studying aesthetic rules and visual perception principles. We have conducted experiments and user studies with social posts from news entities. We demonstrated that the generated snippets are effective at representing media content in a visually appealing and compact way, leading to a better user experience when consuming social media content on mobile devices.

Capturing a professional photo with high visual quality is always a challenging task for mobile users. This paper presents a crowd sourced learning to photograph approach to assist mobile users for composing high quality photos via their mobile devices. The proposed approach is able to leverage the camera and scene context to search related images with similar context and content from social media communities, and then mine composition knowledge to guide photographing on mobile devices. We develop a patch-based feature generation and selection process to discover salient patches and positions that dominate photo composition aesthetics in the input scene. We then build a regression model to map the composition of salient patches to photo-aesthetic scores. Finally, we develop an efficient hierarchical approach to search for the optimal view enclosure for photograph suggestion. We conducted extensive simulations and subjective evaluations to verify the proposed approach.

Automatic photo quality assessment emerged as a hot topic in recent years for its potential in numerous applications. Most existing approaches to photo quality assessment have predominantly focused on image content itself, while ignoring various contexts such as the associated geo-location and timestamp. However, such a universal aesthetic assessment model may not work well with significantly different contexts, since the photography rules are always scene and context dependent. In real cases, professional photographers use different photography knowledge when shooting various scenes in different places. Motivated by this observation, we leverage the geo-context information associated with photos for visual quality assessment. Specifically, we propose in this paper a Scene-Dependent Aesthetic Model (SDAM) to assess photo quality, by jointly leveraging the geo-context and visual content. Geo-contextual leveraged searching is performed to obtain relevant images with similar content to discover the scene-dependent photography principles for accurate photo quality assessment. To overcome the problem that in many cases the number of the contextually searched images is insufficient for learning the SDAM, we adopt transfer learning to utilize auxiliary photos within the same scene category from other locations for learning photography rules. Extensive experiments shows that the proposed SDAM scheme indeed improves the photo quality assessment accuracy via leveraging photo geo-contexts, compared with traditional universal aesthetic models.

This paper proposes a semantic image adaptation scheme for heterogeneous mobile display devices. This scheme aims to provide mobile users with the most desired image content by integrating the content semantic importance with user preferences under limited mobile display constraints. The main contributions of the proposed scheme are: 1) seamless integration of mobile user supplied query information with low level image features to identify semantically important image contents; 2) integration of semantic importance and user feedback to dynamically update mobile user preferences; and 3) perceptually optimized adaptation for image display on mobile devices. In order to bridge the semantic gap for adaptation, we design a Bayesian fusion approach to properly integrate low level features with high level semantics. To accommodate the variation of user preferences, the system involves mobile users in the adaptation process with only a few simple feedbacks so as to present to the users most interesting content on mobile devices. Eventually, perceptually optimized adaptation is performed to present the best image content for mobile users according to mobile display capacities. Extensive experiments have been carried out based on several common events defined in Kodak's consumer image database. These experiments show that by utilizing the proposed semantic adaptation scheme with integration of the semantics and mobile user preferences, perceptually relevant adaptation can be effectively carried out to tailor the image towards user intentions under the mobile environment constraints.

This paper proposes a novel semantics-based consumer photo adaptation scheme for users of small-display mobile devices. The main contributions of the proposed scheme are: (1) seamless integration of mobile user supplied semantic information with low level image features to identify se-mantically important regions-of-interest (ROI), and (2) perceptually optimized adaptation for photo display on mobile devices. In order to bridge the semantic gap in photo search in a consumer photo collection and to perform appropriate adaptation of perceptually important region-of-interest from each photo to fit small displays on the mobile devices, we design a Bayesian fusion approach to properly integrate low level features with high level semantics. Low level features are extracted in a bottom-up fashion while the high level semantics is applied in a top-down style. Extensive experiments have been carried out based on several common events defined in the Kodak consumer photo database. These experiments show that by utilizing the semantics provided by the mobile device users, perceptually consistent adaptation can be effectively carried out. This new semantic adaptation scheme is able to outperform the conventional attention-based scheme for the common events and produce desired regions for the small displays on mobile devices.

This paper proposes a novel event-aware semantic image adaptation framework for accessing consumer photos via mobile devices. There are two major challenges to develop such an image adaptation framework: (1) how to bridge the semantic gap in the search for the desired photos from a consumer database; (2) how to perform appropriate adaptation for various small displays in order to maximize user's perceptual experience according to the semantic significance of objects in photos. To meet the first challenge, an event semantics-guided extraction is developed based on statistical fusion of bottom-up low level features and top-down semantic features. To meet the second challenge, a key object-based semantic adaptation approach is designed to obtain the perceptually optimized Region of Interest (ROI). We conducted experiments based on the events defined in the Kodak consumer photo database. These experiments show that by utilizing the semantics of the photos with event-based a priori knowledge, the adaptation result outperforms the conventional attention based scheme. More importantly, the proposed framework can be easily extended to other events by adopting the event semantics based a priori knowledge into the adaptation system.