Due to the large-scale and multiscale natures of social media data, sparse representation (SR) learning methods are widely followed. However, there are three problems associated with the existing SR methods: 1) they neglect the fact that the semantic features of data may change during iterative learning, which leads to weak semantic learning; 2) they often assume that the data are linearly separable, while the data might be nonlinear in many real-world applications; and 3) they cannot ensure the low-rank and discriminative properties of the data at the same time and might neglect the global properties of the data, leading to suboptimal solutions. To solve these problems, we propose a novel method, named kernel-based SR learning with global and local low-rank label (KSR-GL3) constraint, which strengthens the semantic information and ensures the semantic features invariant during learning. First, we map the data into a high-dimensional feature space to learn the linear representation of samples. Second, global and local low-rank label (GL3) constraint is used to ensure the semantic invariance, low-rankness, and discrimination of features during learning. Third, an$\ell _{2,1}$is imposed to explore the sparseness of the subspace. Mathematical analyses show that GL3 can retain the intrinsic properties of data during learning. By combining the above three components, a generalized power iteration (GPI) approach is applied to build the model and deal with the tricky optimization problem. By KSR-GL 3, a sparse, low-rank, and discriminative subspace is produced from the high-dimensional and orthogonal representation of the data under the guidance of semantics, while the intrinsic properties of data are preserved. Extensive experiments on six datasets compared with five advanced algorithms demonstrate its promising prospects.

Conventional hash-based retrieval method rely on the assumption that the query and database are of the identical domain. However, cross-domain problem often occurs in real-world applications, leading to the unsatisfactory performance of existing hashing methods. Recently, some researchers have put forward domain adaptation retrieval (DAR) under the perspective of domain adaptation (DA) and achieved promising results. But the following limitations still exist: 1) a single function is used to handle two challenges, i.e., domain adaptation and hashing, which is not flexible to explore enough underlying information for simultaneously accomplishing these challenges well; 2) non-dominant features in the sample are ignored; 3) the dissimilarity structure of dissimilar samples is not taken into account. To address the above problems, we propose a novel framework named two-step strategy (TSS) for domain adaptation retrieval, which advocates dividing DAR into two steps: DA step and hashing step. A DA function and a hash function are learned to handle the above two challenges, respectively, making the process more reasonable. Additionally, a discriminant semantic fusion loss is proposed to improve the discriminative ability among classes. Unlike other works that focus on discovering dominant features, we exploit the neglected non-dominant features and assign them attention with sinusoidal semantic embedding, actively creating a clear separation between classes. At last, we present an adaptive similarity preserving loss to preserve the similarity structure of the original data in all intra-domain and inter-domain hash codes. Extensive experiments on various datasets demonstrate that the proposed TSS achieves state-of-the-art performance.

Hashing-based techniques present appealing solutions for cross-modal retrieval due to its low storage requirements and excellent query efficiency. The majority of cross-modal hashing methods typically adopt equal-length encoding scheme to represent multimodal data and achieve cross-modal similarity search. However, such scheme can be regarded as a relatively strict limitation, because it sacrifices the flexible representation of multimodal data in reality and cannot always guarantee the optimal retrieval performance. To address the challenge, this paper focuses on encoding heterogeneous data with varying hash lengths. To achieve this purpose, we propose a flexible cross-modal hashing approach, named Two-stage Asymmetric Similarity Preserving Hashing, TASPH for short, which can be applied to both unequal-length and equal-length retrieval scenarios. Specifically, in the first stage, TASPH designs a novel discrete asymmetric strategy to learn the modality-specific hash codes with varying lengths, enabling a flexible representation of heterogeneous data. Simultaneously, TASPH utilizes two semantic transformation matrices to establish the semantic correlations between varying hash codes. Different from most of the existing approaches that employ relaxation solutions, TASPH satisfies the discrete constraints without any relaxation. In the second stage, the learned semantic transformation matrices are employed to alleviate cross-modal heterogeneity, which guarantees that TASPH can learn more powerful hash functions to improve the discriminative ability of hash codes. Abundant experiments conducted on three benchmark datasets demonstrate encouraging results compared with the state-of-the-art approaches under different retrieval scenarios.

Partial multi-label learning (PML) needs to address the problem of multi-label learning when the dataset contains redundant information. PML is more challenging compared to traditional multi-label learning, because PML needs not only to perform the multi classification task, but also to reduce the impact of noise information on the model. Existing PML methods suffer from the following problems. (1) Only single source of noise is considered. (2) Some methods ignore the label correlations. To solve the above problems, we proposes a new dual noise elimination and dynamic label correlation guided partial multi-label learning (PML-DNDC). Specifically, the hidden ground-truth label matrix is decomposed into two compressed matrices of instance and classifier, which are used to approximate the candidate label matrix, to eliminate the negative effects of label noise on the model. On one hand, the compressed instance matrix maintains local structural consistency with the original instances, eliminating noise in the feature. On the other hand, dynamic label correlation guidance is designed to help classifier training by dynamically exploring the potential label correlations, which encourages relevant labels to obtain similar classifiers. After extensive experiments and analyses, we conclude that the proposed PML-DNDC is superior to the state-of-the-art methods.

Hashing techniques have been extensively studied in cross-modal retrieval due to their advantages in high computational efficiency and low storage cost. However, existing methods unconsciously ignore the complementary information of multimodal data, thus failing to consider learning discriminative hash codes from the perspective of information complementarity while often involving time-consuming training overhead. To tackle the above issues, we propose an efficient discriminative hashing (EDH) with information complementarity consideration. Specifically, we reckon that multimodal features and their corresponding semantic labels describe heterogeneous data viewed from low-and high-level structures, which owns complementarity. To this end, low-level latent representation and high-level semantics representation are simply derived. Then, a joint learning strategy is formulated to simultaneously exploit the above two representations for generating discriminative hash codes, which is quite computationally efficient. Besides, EDH decomposes hash learning into two steps. To obtain powerful hash functions which are conductive to retrieval, a regularization term considering pairwise semantic similarity is introduced into hash functions learning. In addition, an efficient optimization algorithm is designed to solve the optimization problem in EDH. Extensive experiments conducted on benchmark datasets demonstrate the superiority of our EDH in terms of retrieval performance and training efficiency. The source code is available at https://github.com/hjf-hjf/EDH.

Partial multi-label learning (PML) aims to accurately predict multi-labels for unknown instances with noise labels in training set. Accurate identification of ground-truth labels is essential to optimize performance. The accuracy of ground-truth labels affects label correlation capture, which guides classifier learning. However, current methods often rely on approximation of ground-truth labels through intermediate variables, rather than restoring binary ground-truth labels directly. To solve this problem, we propose a partial multi-label learning algorithm by exploring binary ground-truth labels (PML-EBGL). First, the candidate label matrix is decomposed into a ground-truth label matrix and a noise matrix. Then, the noise matrix is constrained using the l1norm for sparsity, and the binary ground-truth label matrix is used to explore label correlation, which is transferred to classifier by using the Laplacian term. Finally, the rotation matrix is added after the classifier, and the instance is projected onto the binary label matrix. Extensive experiments show that PML-EBGL outperforms state-of-the-art methods.

In this paper, we propose a deep dense fusion network with multimodal residual (DFMR) to integrate multimodal information including language, acoustic speeches, and visual images for sentiment analysis. DFMR exploits a dense fusion (DF) block to fuse the multimodal features obtained by modality-specific sequence networks, which is achieved by modelling their unimodal, bimodal and trimodal interactions jointly. Instead of concatenating the multimodal features directly, DF block conducts fusion for any two paired modalities firstly, and the fused information will be integrated with the other modalities subsequently. Furthermore, DFMR stacks multiple DF blocks to capture high-level semantic information conveyed by the multimodal representations. In particular, DFMR adopts a multimodal residual (MR) block to integrate the modality-specific features and fused features in each DF blocks, to avoid forgetting the multi-aspect information and alleviate gradient vanishing during stacking. Extensive experiments conducted on four public benchmark datasets show that DFMR outperforms eleven state-of-the-art baselines.

A major assumption in data mining and machine learning is that the training set and test set come from the same domain. They share the same feature space and have the same distribution. However, in many real-world applications, the training set and test set usually come from different domains. Thus, there might be negative similarities between different domains so that the negative transfer problem caused by negative similarity may happen. In this paper, we propose a novel method named active transfer learning (ATL) to solve the above problem. Specifically, the orthogonal projection matrix and the weight coefficient vector are introduced to extend maximum mean discrepancy (MMD) so that it can minimize MMD and simultaneously eliminate the negative transfer. To find the informative and discriminative subsets from the source domain, we then propose an information diversity term by using the local geometric structure information of the source samples. Besides, by using the label information of source samples, our method can guarantee the selected subsets as discriminative as possible. Finally, to efficiently implement the proposed method, an alternating optimization approach, which is based on the alternating direction method of multipliers (ADMM), is designed to solve the optimization problem. To demonstrate the effectiveness of the proposed ATL model, experiments are conducted on five real-world data sets. The experimental results show the superiority of our method over the state-of-the-art methods.

In this article, we propose to learn shared semantic space with correlation alignment ( S 3 CA ) for multimodal data representations, which aligns nonlinear correlations of multimodal data distributions in deep neural networks designed for heterogeneous data. In the context of cross-modal (event) retrieval, we design a neural network with convolutional layers and fully connected layers to extract features for images, including images on Flickr-like social media. Simultaneously, we exploit a fully connected neural network to extract semantic features for text documents, including news articles from news media. In particular, nonlinear correlations of layer activations in the two neural networks are aligned with correlation alignment during the joint training of the networks. Furthermore, we project the multimodal data into a shared semantic space for cross-modal (event) retrieval, where the distances between heterogeneous data samples can be measured directly. In addition, we contribute a Wiki-Flickr Event dataset, where the multimodal data samples are not describing each other in pairs like the existing paired datasets, but all of them are describing semantic events. Extensive experiments conducted on both paired and unpaired datasets manifest the effectiveness of S 3 CA , outperforming the state-of-the-art methods.

In this paper, a novel Deep Semantic Space learning model with Intra-class Low-rank constraint (DSSIL) is proposed for cross-modal retrieval, which is composed of two subnetworks for modality-specific representation learning, followed by projection layers for common space mapping. In particular, DSSIL takes into account semantic consistency to fuse the cross-modal data in a high-level common space, and constrains the common representation matrix within the same class to be low-rank, in order to induce the intra-class representations more relevant. More formally, two regularization terms are devised for the two aspects, which have been incorporated into the objective of DSSIL. To optimize the modality-specific subnetworks and the projection layers simultaneously by exploiting the gradient decent directly, we approximate the nonconvex low-rank constraint by minimizing a few smallest singular values of the intra-class matrix with theoretical analysis. Extensive experiments conducted on three public datasets demonstrate the competitive superiority of DSSIL for cross-modal retrieval compared with the state-of-the-art methods.

In this paper, a Catboost-based framework is proposed to predict social media popularity. The framework is constituted by two components: feature representation and Catboost training. In the component of feature representation, numerical features are directly used, while categorical features are converted into numerical features by a method of order target statistics in Catboost. Besides, some additional user information is also tracked to enrich the feature space. In the other component, Catboost is adopted as the regression model which is trained by using post-related, user-related and additional user information. Moreover, to make full use of the dataset for model training, a dataset augmentation strategy based on pseudo labels is proposed. This strategy involves in two-stage training. In the first stage, it trains a first-stage model that is used to label the test set as pseudo labeled. In the next stage, a final model is trained based on the new training set that includes original validation set and the pseudo labeled test set. The proposed method achieves the 2nd place in the leader board of the Grand Challenge of Social Media Prediction.

Cross-domain image retrieval is always encountering insufficient labelled data in real world. In this paper, we propose unsupervised embedding learning (UEL) for cross-domain beauty and personal care product retrieval to finetune the convolutional neural network (CNN). More specifically, UEL utilizes the non-parametric softmax to train the CNN model as instance-level classification, which reduces the influence of some inevitable problems (e.g., shape variations). In order to obtain better performance, we integrate a few existing retrieval methods trained on different datasets. Furthermore, a query expansion strategy (i.e., diffusion) is adopted to improve the performance. Extensive experiments conducted on a dataset including half million images of beauty and personal product items (Perfect-500K) manifest the effectiveness of our proposed method. Our approach achieves the 2nd place in the leader board of the Grand Challenge of AI Meets Beauty in ACM Multimedia 2019. Our code is available at: https://github.com/RetrainIt/Perfect-Half-Million-Beauty-Product-Image-Recognition-Challenge-2019.

Social media headline prediction (SMHP) is a thriving application scenario, which aims to predict the popularity of the post data shared on social media. In this paper, we propose to use multi-aspect features combined with the random forest (RF) model for popularity predictions. Firstly, we extract features by combining both metadata of the posts and users' features. More specifically, we adopt the binary coding strategy for dimensionality reduction and deal with the missing values by using some strategies, i.e., estimating the missing geographic information according to the information of users, and filling the missing features with median. Furthermore, regression models can be used directly to make predictions. In particular, a random forest (RF) model is adopted since it does not require much effort in tuning hyper-parameters and performs effectively. Extensive experiments conducted on the SMHP dataset consisting of 340K image posts shared by 80K users manifest the effectiveness of our method. Our approach achieves the 4nd place in the leader board of the Grand Challenge of SMHP in ACM Multimedia 2018.