In this article, we work on generating fashion style images with deep neural network algorithms. Given a garment image, and single or multiple style images (e.g., flower, blue and white porcelain), it is a challenge to generate a synthesized clothing image with single or mix-and-match styles due to the need to preserve global clothing contents with coverable styles, to achieve high fidelity of local details, and to conform different styles with specific areas. To address this challenge, we propose a fashion style generator (FashionG) framework for the single-style generation and a spatially constrained FashionG (SC-FashionG) framework for mix-and-match style generation. Both FashionG and SC-FashionG are end-to-end feedforward neural networks that consist of a generator for image transformation and a discriminator for preserving content and style globally and locally. Specifically, a global-based loss is calculated based on full images, which can preserve the global clothing form and design. A patch-based loss is calculated based on image patches, which can preserve detailed local style patterns. We develop an alternating patch-global optimization methodology to minimize these losses. Compared with FashionG, SC-FashionG employs an additional spatial constraint to ensure that each style is blended only onto a specific area of the clothing image. Extensive experiments demonstrate the effectiveness of both single-style and mix-and-match style generations.

In this paper, we focus on the problem of generating realistic stylized fashion images on the human body under a given text description. Existing stylized fashion image generation works mainly require additional style images as input and guidance. Very few works focus on the generation only in the context of textual description. And the fashion images generated by these works usually lack stylization and authenticity. There are two main challenges to this problem. Firstly, the shape of the generated garment should vary based on the text description (e.g., long sleeves, short sleeves, sleeveless) and the body shape and posture. Second, it is difficult to generate real textures directly from text descriptions. Although we can consider borrowing texture information from the auxiliary real clothing images, the misalignment of clothing image shapes can degrade performance. We address these changes by proposing a two-stage stylized text-to-fashion image generation framework. In the first stage, we predict the segmentation map for the generated clothing image based on the input text description and original body shape and posture. In the second stage, we focus on rendering colors and textures. In particular, we investigate a novel spatial-style loss that enables the generator to learn style distribution from text-related real images, even if shapes and postures are misaligned. Extensive experiments demonstrate that our proposed method outperforms state-of-the-art methods quantitatively and qualitatively.

When planning a trip, users tend to browse Place-of-Interest (POI) information on the Internet and then depart. Many works aimed at summarizing POIs by visual and textual analysis, while many of them ignored the inter-relationship between different views offered by the community-contributed information. In this paper, we propose a City-POI-LOI (CPL) summarization method to automatically mine POIs from the city-level landmark images. And a Location-Appearance-Semantic-Temporal (LAST) clustering method is proposed to mine the popular viewpoints termed Location-Of-Interest (LOI) in each POI by taking location, appearance, semantic, and temporal feature into consideration. We perform text and image summarization for each LOI, and we further summarize the POIs based on season. We conduct a series of experiments based on DIV400 and ATCF Dataset. Experimental results show the effectiveness of the proposed POI summarization approach.

A real-world recommender usually adopts heterogeneous types of user feedbacks, for example, numerical ratings such as 5-star grades and binary ratings such as likes and dislikes. In this work, we focus on transferring knowledge from binary ratings to numerical ratings, facing a more serious data sparsity problem. Conventional Collective Factorization methods usually assume that there are shared user and item latent factors across multiple related domains, but may ignore the shared common knowledge of rating patterns. Furthermore, existing works may also fail to consider the hierarchical structures in the heterogeneous recommendation scenario (i.e., genre, sub-genre, detailed-category). To address these challenges, in this paper, we propose a novel Deep Low-rank Sparse Collective Factorization (DLSCF) framework for heterogeneous recommendation. Specifically, we adopt low-rank sparse decomposition to capture the common rating patterns in related domains while splitting the domain-specific patterns. We also factorize the model in multiple layers to capture the affiliation relation between latent categories and sub-categories. We propose both batch and Stochastic Gradient Descent (SGD) based optimization algorithms for solving DLSCF. Experimental results on MoviePilot, Netfilx, Flixter, MovieLens10M and MovieLens20M datasets demonstrate the effectiveness of the proposed algorithms, by comparing them with several state-of-the-art batch and SGD based approaches.

This paper introduces the problem of automatic font pairing. Font pairing is an important design task that is difficult for novices. Given a font selection for one part of a document (e.g., header), our goal is to recommend a font to be used in another part (e.g., body) such that the two fonts used together look visually pleasing. There are three main challenges in font pairing. First, this is a fine-grained problem, in which the subtle distinctions between fonts may be important. Second, rules and conventions of font pairing given by human experts are difficult to formalize. Third, font pairing is an asymmetric problem in that the roles played by header and body fonts are not interchangeable. To address these challenges, we propose automatic font pairing through learning visual relationships from large-scale human-generated font pairs. We introduce a new database for font pairing constructed from millions of PDF documents available on the Internet. We propose two font pairing algorithms: dual-space k-NN and asymmetric similarity metric learning (ASML). These two methods automatically learn fine-grained relationships from large-scale data. We also investigate several baseline methods based on the rules from professional designers. Experiments and user studies demonstrate the effectiveness of our proposed dataset and methods.

Instance transfer approaches consider source and target data together during the training process, and borrow examples from the source domain to augment the training data, when there is limited or no label in the target domain. Among them, boosting-based transfer learning methods (e.g., TrAdaBoost) are most widely used. When dealing with more complex data, we may consider the more complex hypotheses (e.g., a decision tree with deeper layers). However, with the fixed and high complexity of the hypotheses, TrAdaBoost and its variants may face the overfitting problems. Even worse, in the transfer learning scenario, a decision tree with deep layers may overfit different distribution data in the source domain. In this paper, we propose a new instance transfer learning method, i.e., Deep Decision Tree Transfer Boosting (DTrBoost), whose weights are learned and assigned to base learners by minimizing the data-dependent learning bounds across both source and target domains in terms of the Rademacher complexities. This guarantees that we can learn decision trees with deep layers without overfitting. The theorem proof and experimental results indicate the effectiveness of our proposed method.

This paper addresses the problem of geotagging images, i.e., assigning GPS coordinates (i.e., latitude, longitude) to images using image contents. Due to the huge appearance variability of visual features across the world, the images' contents and their GPS coordinates may be inconsistent. This means images captured from geographically close areas may appear visually distinct; and images with visually similar contents may be taken from geographically distant areas. In this paper, we propose a deep Geo-constrained Auto-encoder (DGAE) to solve these inconsistency problems. Using clustered GPS data and visual data, our approach identifies inconsistent data pairs (i.e., image, GPS). We then propose a novel deep learning framework that can learn similar feature representations for geographically close images and distinct feature representations for geographically distant images. We introduce two new constraints: the same-area constraint and the easy-confusing constraint to our feature learning networks. The former one penalizes images from the same area but with very distinct visual features, and the latter one penalizes images from distant areas but with very similar visual features. A deep architecture is developed to further improve learning discriminative features, which can disambiguate different geometric locations. Our approach is extensively evaluated on a newly-compiled large image geotagging dataset from large-scale community-contributed images with 664,720 images and outperforms comparison approaches.

Style classification (e.g., Baroque and Gothic architecture style) is grabbing increasing attention in many fields such as fashion, architecture, and manga. Most existing methods focus on extracting discriminative features from local patches or patterns. However, the spread out phenomenon in style classification has not been recognized yet. It means that visually less representative images in a style class are usually very diverse and easily getting misclassified. We name them weak style images. Another issue when employing multiple visual features towards effective weak style classification is lack of consensus among different features. That is, weights for different visual features in the local patch should have been allocated similar values. To address these issues, we propose a Consensus Style Centralizing Auto-Encoder (CSCAE) for learning robust style features representation, especially for weak style classification. First, we propose a Style Centralizing Auto-Encoder (SCAE) which centralizes weak style features in a progressive way. Then, based on SCAE, we propose both the non-linear and linear version CSCAE which adaptively allocate weights for different features during the progressive centralization process. Consensus constraints are added based on the assumption that the weights of different features of the same patch should be similar. Specifically, the proposed linear counterpart of CSCAE motivated by the "shared weights" idea as well as group sparsity improves both efficacy and efficiency. For evaluations, we experiment extensively on fashion, manga and architecture style classification problems. In addition, we collect a new dataset-Online Shopping, for fashion style classification, which will be publicly available for vision based fashion style research. Experiments demonstrate the effectiveness of the SCAE and CSCAE on both public and newly collected datasets when compared with the most recent state-of-the-art works.

In this article, we address the cross-domain (i.e., street and shop) clothing retrieval problem and investigate its real-world applications for online clothing shopping. It is a challenging problem due to the large discrepancy between street and shop domain images. We focus on learning an effective feature-embedding model to generate robust and discriminative feature representation across domains. Existing triplet embedding models achieve promising results by finding an embedding metric in which the distance between negative pairs is larger than the distance between positive pairs plus a margin. However, existing methods do not address the challenges in the cross-domain clothing retrieval scenario sufficiently. First, the intradomain and cross-domain data relationships need to be considered simultaneously. Second, the number of matched and nonmatched cross-domain pairs are unbalanced. To address these challenges, we propose a deep cross-triplet embedding algorithm together with a cross-triplet sampling strategy. The extensive experimental evaluations demonstrate the effectiveness of the proposed algorithms well. Furthermore, we investigate two novel online shopping applications, clothing trying on and accessories recommendation, based on a unified cross-domain clothing retrieval framework.

In this paper, we focus on a new problem: applying artificial intelligence to automatically generate fashion style images. Given a basic clothing image and a fashion style image (e.g., leopard print), we generate a clothing image with the certain style in real time with a neural fashion style generator. Fashion style generation is related to recent artistic style transfer works, but has its own challenges. The synthetic image should preserve the similar design as the basic clothing, and meanwhile blend the new style pattern on the clothing. Neither existing global nor patch based neural style transfer methods could well solve these challenges. In this paper, we propose an end-to-end feed-forward neural network which consists of a fashion style generator and a discriminator. The global and patch based style and content losses calculated by the discriminator alternatively back-propagate the generator network and optimize it. The global optimization stage preserves the clothing form and design and the local optimization stage preserves the detailed style pattern. Extensive experiments show that our method outperforms the state-of-the-arts.

In cross-domain recommendation, data sparsity becomes more and more serious when the ratings are expressed numerically, e.g., 5-star grades. In this work, we focus on borrowing the knowledge from other domains in the form of binary ratings, such as likes and dislikes for certain items. Most existing works conventionally assume that multiple domains share some common latent information across users and items. In practice, however, the related domains not only share the common latent feature of users and items, but also share some knowledge of rating patterns. Furthermore, conventional methods did not consider the hierarchical structures (i.e., genre, sub genre, detailed-category) in real-world recommendation system. In this paper, we propose a novel Deep Low-rank Sparse Collective Factorization (DLSCF) to facilitate the cross-domain recommendation. Specifically, the low-rank sparse decomposition is adopted to capture the most shared rating patterns with low-rank constraint while integrating the domain-specific patterns with group-sparse scheme. Furthermore, we factorize the rating pattern matrix in multiple layers to obtain the user/item latent category affiliation matrices, which could indicate the affiliation relation between latent categories and latent sub-categories. Experimental results on MoviePilot and Netfilx datasets demonstrate the effectiveness of our proposed algorithm at various sparsity levels, by comparing it with several state-of-the-art approaches.

Style classification (e.g., architectural, music, fashion) attracts an increasing attention in both research and industrial fields. Most existing works focused on low-level visual features composition for style representation. However, little effort has been devoted to automatic mid-level or high-level style features learning by reorganizing low-level descriptors. Moreover, styles are usually spread out and not easy to differentiate from one to another. In this paper, we call these less representative images as weak style images. To address these issues, we propose a consensus style centralizing auto-encoder (CSCAE) to extract robust style features to facilitate weak style classification. CSCAE is the ensemble of several style centralizing auto-encoders (SCAEs) with consensus constraint. Each SCAE centralizes each feature of certain category in a progressive way. We apply our method in fashion style classification and manga style classification as two example applications. In addition, we collect a new dataset, Online Shopping, for fashion style classification evaluation, which will be publicly available for vision based fashion style research. Experiments demonstrate the effectiveness of SCAE and CSCAE on both public and newly collected datasets when compared with the most recent state-of-the-art works.

In this paper, we address two practical problems when shopping online: 1) What will I look like when wearing this clothing on the street? 2) How to find the exact same or similar clothing that other people are wearing on the street or in a movie? In this paper, we jointly solve these two problems with one bi-directional shop-to-street street-to-shop clothing retrieval framework. There are three main challenges of cross-domain clothing retrieval task. First is to learn the discrepancy (e.g., background, pose, illumination) between street domain and shop domain clothing. Second, both intra-domain and cross-domain similarity need to be considered during feature embedding. Third, there is large bias between the number of matched and non-matched street and shop pairs. To solve these challenges, in this paper, we propose a deep bi-directional cross-triplet embedding algorithm by extending the start-of-the-art triplet embedding into cross-domain retrieval scenario. Extensive experiments demonstrate the effectiveness of the proposed algorithm.

Big data increasingly benefit both research and industrial area such as health care, finance service and commercial recommendation. This paper presents a personalized travel sequence recommendation from both travelogues and community contributed photos and the heterogeneous metadata (e.g., tags, geo-location, and date taken) associated with these photos. Unlike most existing travel recommendation approaches, our approach is not only personalized to user's travel interest but also able to recommend a travel sequence rather than individual Points of Interest (POIs). Topical package space including representative tags, the distributions of cost, visiting time and visiting season of each topic, is mined to bridge the vocabulary gap between user travel preference and travel routes. We take advantage of the complementary of two kinds of social media: travelogue and community contributed photos. We map both user's and routes' textual descriptions to the topical package space to get user topical package model and route topical package model (i.e., topical interest, cost, time and season). To recommend personalized POI sequence, first, famous routes are ranked according to the similarity between user package and route package. Then top ranked routes are further optimized by social similar users' travel records. Representative images with viewpoint and seasonal diversity of POIs are shown to offer a more comprehensive impression. We evaluate our recommendation system on a collection of 7 million Flickr images uploaded by 7,387 users and 24,008 travelogues covering 864 travel POIs in nine famous cities, and show its effectiveness. We also contribute a new dataset with more than 200 K photos with heterogeneous metadata in nine famous cities.

With the rapid development of social networks, more and more users choose to share their own photos with their friends. Especially, users prefer to share the photos they took during traveling, thus there emerges many user generated content for place-of-interests (POIs). So based on the user contributed photos, we can summarize each POI by mining location-of-interest (LOI, which represents the attractive viewpoints of POI) and selecting some representative images from them. It is important for scheduling a traveling, and in this paper, an effective POI summarization approach is proposed by an improved geo-clustering with visual and views verification, which helps us to have a representative and comprehensive perception for POI. In our approach, we firstly collect POI related photos from social media, and filter the raw data by the combination of tags and geo-locations. Secondly, we mine LOIs for each POI by the improved geo-location clustering method. Finally, we employ visual and views verification to select images from LOIs to summarize the POI. We conduct a series of experiments based on Flcoickr dataset. Experimental results demonstrate the effectiveness of our proposed method.

From social media has emerged continuous needs for automatic travel recommendations. Collaborative filtering (CF) is the most well-known approach. However, existing approaches generally suffer from various weaknesses. For example , sparsity can significantly degrade the performance of traditional CF. If a user only visits very few locations, accurate similar user identification becomes very challenging due to lack of sufficient information for effective inference. Moreover, existing recommendation approaches often ignore rich user information like textual descriptions of photos which can reflect users' travel preferences. The topic model (TM) method is an effective way to solve the “sparsity problem,” but is still far from satisfactory. In this paper, an author topic model-based collaborative filtering (ATCF) method is proposed to facilitate comprehensive points of interest (POIs) recommendations for social users. In our approach, user preference topics, such as cultural, cityscape, or landmark, are extracted from the geo-tag constrained textual description of photos via the author topic model instead of only from the geo-tags (GPS locations). Advantages and superior performance of our approach are demonstrated by extensive experiments on a large collection of data.

Traveling experience sharing has become pervasive. In this paper, we present a system which automatically generates a multimedia travelogue for mobile users. Multimedia travelogue shows user footprint with photos on the corresponding location on the map, which could offer the space distribution inner or between the locations. Meanwhile, location overviews with representative tags and images show comprehensive knowledge of the landmarks. The key challenges are: 1) when mapping footprint, some photos are without geo-tags; 2)user-contributed photos may be not clear and graceful; 3) to detect which landmark is actually on the photo to offer location overview. We solve these challenges by combining both geo-tags and tags to estimate the location. First we use the relationship of timestamp and geo-tag in the same trip group to estimate the footprints. Second we localize the landmark on the photo by tag match method which matching user tags with representative tags of the landmark. Experimental results on a Flickr image collection of nearly 2 million images of 6,581 users demonstrate the effectiveness of our approach.