We propose CLASS; a novel unified model for the syn-thesis and search for design layouts, two tasks that are often handled separately by prior works. We propose to learn a compact and coherent latent feature of a layout supporting joint search and synthesis. This allows vari-ous operations such style-conditioned layout generation, la-tent space manipulation and provides seamless integration of search and synthesis for an effective design workflow. We train CLASS with a dual decoder: a new transformer-based layout-conditioned decoder and a CNN-based raster decoder. The latent-conditioned decoder explicitly conditions upon a latent vector while generating a layout in an auto-regressive fashion. We train CLASS under variational framework which in conjunction with a raster-decoder en-hances the latent representation improving both generation and retrieval performances. We show the effectiveness of CLASS on the RICO and PubLayNet benchmarks, and demonstrate that CLASS is capable of high-quality synthe-sis from scratch, as well as performing self-completion, in-terpolation, project between design layouts, whilst achieving close to or better than state-of-the-art search performance.

We present a novel hierarchical modeling method for layout representation learning, the core of design documents (e.g., user interface, poster, template). Existing works on layout representation often ignore element hierarchies, which is an important facet of layouts, and mainly rely on the spatial bounding boxes for feature extraction. This paper proposes a Spatial-Structural Hierarchical Auto-Encoder (SSH-AE) that learns hierarchical representation by treating a hierarchically annotated layout as a tree format. On the one side, we model SSH-AE from both spatial (semantic views) and structural (organization and relationships) perspectives, which are two complementary aspects to represent a layout. On the other side, the semantic/geometric properties are associated at multiple resolutions/granularities, naturally handling complex layouts. Our learned representations are used for effective layout search from both spatial and structural similarity perspectives. We also newly involve the tree-edit distance (TED) as an evaluation metric to construct a comprehensive evaluation protocol for layout similarity assessment, which benefits a systematic and customized layout search. We further present a new dataset of POSTER layouts which we believe will be useful for future layout research. We show that our proposed SSH-AE outperforms the existing methods achieving state-of-the-art performance on two benchmark datasets. Code is available at github.com/yueb17/SSH-AE.

This paper proposes dynamically modulated metric learning (DMML) for learning a tiered similarity space to perform visual search. Existing methods often treat the training samples having different degree of information with equal importance which hinders in capturing the underlying granularities in visual similarity. Proposed DMML automatically exploits the informativeness of samples during training by leveraging correlation between image attributes and embedding that are learned jointly. The two tasks are interlinked by supervising signals where the predicted attribute vectors are used to dynamically learn the loss function. To this end, we propose a new soft-binomial deviance loss that helps to capture the feature similarity space at multiple granularities. Compared to recent ensemble and attention based methods, our DMML framework is conceptually simple yet effective, and achieves state-of-the-art performances on standard benchmark datasets; e.g. an improvement of 4% Recall@1 over the SOTA [1] on DeepFashion dataset.

Ethnicity information is an integral part of human identity, and a useful identifier for various applications ranging from video surveillance, targeted advertisement to social media profiling. In recent years, Convolutional Neural Networks (CNNs) have shown state-of-the-art performance in many visual recognition problems. Currently, there are a few CNN-based approaches on ethnicity classification [1], [2]. However, the approaches suffer from the following limitations: (i) most face datasets do not include ethnicity information, and those with ethnicity information are typically small to medium in size, thereby they do not provide sufficient samples for training of CNNs from the scratch, and (ii) the CNN methods often treat ethnicity classification as a multi-class classification where the likelihood of each class label is generated. However, it does not utilize the intermediate activation functions of CNNs which provide rich hierarchical features to assist in ethnicity classification. In view of this, this paper proposes a new hybrid supervised learning method to perform ethnicity classification that uses both the strength of CNN as well as the rich features obtained from the network. The method combines the soft likelihood of CNN classification output with an image ranking engine that leverages on matching of the hierarchical features between the query and dataset images. A supervised Support Vector Machine (SVM) hybrid learning is developed to train the combined feature vectors to perform ethnicity classification. The performance of the proposed method is evaluated using a dataset consisting of Bangladeshi, Chinese and Indian ethnicity groups, and it outperforms the state-of-the-art methods [2], [3] by up to 3% in recognition accuracy.

Brand plays a significant role in fashion clothing. Consumers are brand conscious during the clothing search and purchase. Existing visual fashion search methods [1], [8], [17]-[19] often do not explicitly consider the brand information such as logos. Brand logo in clothing images are quite small and often suffer various deformations, and hence pose a significant challenge for branded clothing search. In view of this, this paper presents a new Brand-Aware Fashion Search (BAFS) framework that explores the brand information during the visual search. We construct a new brand fashion dataset which consists of 10K images of branded clothing with trademark logos. The proposed framework first jointly detects the brand logo and clothing items in the images. Next, to extract the rich visual information from the clothing images, we propose a new deep feature encoding known as Principal Component Maximum Activation of Convolutions (PMAC) that leverages hierarchies of CNN activations. The PMAC feature aims to capture both low-level visual information and high-level global abstraction from the images. The proposed method further uses a brand-aware re-ranking technique to improve the search. Experiments conducted on the brand fashion dataset shows that the proposed framework achieves superior performance to other comparative methods.

Repetitive patterns are significant visual cues for matching and detecting objects in images. However, information from repetitive patterns is underutilized in computer vision algorithms as they cause burstiness issue in image similarity scoring. Existing similarity metrics do not take the repetitive patterns into account, and hence cannot handle images with repetitive patterns well. In view of this, this letter presents a new feature repetitiveness similarity (FRS) metric that not only addresses the burstiness issue, but also uses the information from the repetitive patterns to enhance the retrieval performance. The proposed FRS framework detects the repetitive patterns using descriptor and geometric information of local features in the images. Unique and repetitive features are handled separately and then fused at scoring stage using the FRS metric. Experiments conducted on the benchmark Oxford and Paris datasets show that the proposed method outperforms the state-of-the-art methods by a mean average precision of 6%. This demonstrates the effectiveness of FRS metric in matching and retrieval of images with repeated patterns.