Environmental, social, and governance (ESG) serves as a crucial indicator for evaluating firms in terms of sustainable development. However, the existing ESG evaluation systems suffer from limitations, such as narrow coverage, subjective bias, and lack of timeliness. Therefore, there is a pressing need to leverage machine learning methods to predict the ESG performance of firms using their publicly available data. Traditional machine learning models encounter the feature imbalance problem due to the heterogeneity in ESG-related features. Common approaches typically involve unfolding all features, thereby granting highdimensional folding features greater exposure and accessibility to downstream models, which results in the neglect of lowdimensional features. To fill the research gap regarding fully using the heterogeneous features of enterprises to enhance AIbased ESG prediction performance, we propose the Feature Balancing Transformer (FeBT), a model based on autoencoders and Transformer blocks. FeBT incorporates a novel feature balancing technique that compresses and enhances high-dimensional features from imbalanced data into low-dimensional representations, thereby ensuring a more balanced impact of high-dimensional and low-dimensional features on the model's performance in the downstream ESG forecasting module. Extensive experiments verified the superior performance of FeBT compared with stateof-the-art methods in real-world ESG-related datasets and evidenced that our feature balancing module provides significant insights from high-dimensional folding features.

Vision-language pre-trained (VLP) models have been the foundation of numerous vision-language tasks. Given their prevalence, it becomes imperative to assess their adversarial robustness, especially when deploying them in security-crucial real-world applications. Traditionally, adversarial perturbations generated for this assessment target specific VLP models, datasets, and/or downstream tasks. This practice suffers from low transferability and additional computation costs when transitioning to new scenarios.

The training of the Deep Neural Network (DNN) has been seriously challenged by insidious noise in the dataset, including noise in raw data and errors in annotations. Existing methods usually limit their efforts to the defense of one particular kind of noise, which would be powerless when facing the coexistence of various noise. To deal with it, we propose a novel Multi-head Siamese Prototype Learning (MSPL) method to promote discriminative features and representative prototypes by modeling invariance in samples and sieving out incorrectness in labels. More specifically, a multi-head Siamese network structure is constructed, where prototype learning with the multi-consistency constraint is performed to improve the resilience of the model to noise. Under this regime, adversarial contrastive learning is performed to train the model with the dynamically generated vicious adversarial examples, further enhancing the invariant predictive ability against data noise. At the same time, to deal with label noise, an effective multi-granularity sample selection strategy is designed to filter out noisy labels by measuring the error distribution in both global and local (i.e., class-specific) perspectives. Semi-supervised learning is accordingly conducted to train the model with the resulting labelled data (i.e., data with clean labels) and unlabelled data (i.e., data with noisy labels). Extensive experiments on benchmarks demonstrate the effectiveness of the proposed method in the extremely noisy learning environment.

Deep cross-modal retrieval techniques have recently achieved remarkable performance, which also poses severe threats to data privacy potentially. Nowadays, enormous user-generated contents that convey personal information are released and shared on the Internet. One may abuse a retrieval system to pinpoint sensitive information of a particular Internet user, causing privacy leakage. In this article, we propose a data-centric Proactive Privacy-preserving Cross-modal Learning algorithm that fulfills the protection purpose by employing a generator to transform original data into adversarial data with quasi-imperceptible perturbations before releasing them. When the data source is infiltrated, the inside adversarial data can confuse retrieval models under the attacker’s control to make erroneous predictions. We consider the protection under a realistic and challenging setting where the prior knowledge of malicious models is agnostic. To handle this, a surrogate retrieval model is instead introduced, acting as the target to fool. The whole network is trained under a game-theoretical framework, where the generator and the retrieval model persistently evolve to fight against each other. To facilitate the optimization, a Gradient Reversal Layer module is inserted between two models, enabling a one-step learning fashion. Extensive experiments on widely used realistic datasets prove the effectiveness of the proposed method.

Despite great success achieved, deep learning methods are vulnerable to noise in the training dataset, including adversarial perturbations and annotation noise. These harmful factors significantly influence the learning process of deep models, leading to less confident models. However, existing methods have not yet studied this practical and challenging issue.

Modern deep neural networks (DNNs) have greatly facilitated the development of sequential recommender systems by achieving state-of-the-art recommendation performance on various sequential recommendation tasks. Given a sequence of interacted items, existing DNN-based sequential recommenders commonly embed each item into a unique vector to support subsequent computations of the user interest. However, due to the potentially large number of items, the over-parameterised item embedding matrix of a sequential recommender has become a memory bottleneck for efficient deployment in resource-constrained environments, e.g., smartphones and other edge devices. Furthermore, we observe that the widely-used multi-head self-attention, though being effective in modelling sequential dependencies among items, heavily relies on redundant attention units to fully capture both global and local item-item transition patterns within a sequence.

The hashing technique has recently sparked much attention in information retrieval community due to its high efficiency in terms of storage and query processing. For cross-modal retrieval tasks, existing supervised hashing models either treat the semantic labels as the ground truth and formalize the problem to a classification task, or further add a similarity matrix as supervisory signals to pursue hash codes of high quality to represent coupled data. However, these approaches are incapable of ensuring that the learnt binary codes preserve well the semantics and similarity relationships contained in the supervised information. Moreover, for sophisticated discrete optimization problems, it is always addressed by continuous relaxation or bit-wise solver, which leads to a large quantization error and inefficient computation. To relieve these issues, in this paper, we present a two-step supervised discrete hashing method, i.e., laBel ReconstrUcting and Similarity preserving Hashing (BRUSH). We formulate it as an asymmetric pairwise similarity-preserving problem by using two latent semantic embeddings deducted from decomposing semantics and reconstructing semantics, respectively. Meanwhile, the unified binary codes are jointly generated based on both embeddings with the affinity guarantee, such that the discriminative property of the obtained hash codes can be significantly enhanced alongside preserving semantics well. In addition, by adopting two-step hash learning strategy, our method simplifies the procedure of the hashing function and binary codes learning, thus improving the flexibility and efficiency. The resulting discrete optimization problem is also elegantly solved by the proposed alternating algorithm without any relaxation. Extensive experiments on benchmarks demonstrate that BRUSH outperforms the state-of-the-art methods, in terms of efficiency and effectiveness.

Deep learning techniques have ushered in significant progress in large-scale multi-modal retrieval. Nevertheless, the advanced techniques may be used nefariously to conduct a search that violates the privacy of individuals. In this paper, we propose a novel PrIvacy Protection method (PIP) against malicious multi-modal retrieval models, which proactively transfers original data into adversarial data with quasi-imperceptible perturbations before releasing them. Consequently, unauthorized malicious parties are not able to use deployed deep models to find out desired sensitive information with them. In addition to privacy preserving, PIP synchronously learns an effective multi-modal retrieval model to facilitate authorized uses, endowed with strong resilience to the perturbations. To the best of our knowledge, it is a very first attempt to consider privacy issues in multi-modal retrieval, and encapsulate both privacy protection against unauthorized retrieval and robust multi-modal learning for authorized uses into a unified framework. This work is conducted in the challenging no-box and unsupervised settings, where neither target malicious models nor supervised information is known. The optimization objective of our versatile PIP is achieved through a two-player game between different components with both the intra- and inter-modality graph alignments and the domain distribution alignment considered. Besides, a high-level similarity matrix is developed to obtain reliable guidance for learning. Empirically, we apply the proposed PIP to hashing based multi-modal retrieval scenarios and prove its effectiveness on a range of benchmarks and tasks.

Recently, cross-modal hashing (CMH) methods have attracted much attention. Many methods have been explored; however, there are still some issues that need to be further considered. 1) How to efficiently construct the correlations among heterogeneous modalities. 2) How to solve the NP-hard optimization problem and avoid the large quantization errors generated by relaxation. 3) How to handle the complex and difficult problem in most CMH methods that simultaneously learning the hash codes and hash functions. To address these challenges, we present a novel cross-modal hashing algorithm, named Asymmetric Discrete Cross-Modal Hashing (ADCH). Specifically, it leverages the collective matrix factorization technique to learn the common latent representations while preserving not only the cross-correlation from different modalities but also the semantic similarity. Instead of relaxing the binary constraints, it generates the hash codes directly using an iterative optimization algorithm proposed in this work. Based the learnt hash codes, ADCH further learns a series of binary classifiers as hash functions, which is flexible and effective. Extensive experiments are conducted on three real-world datasets. The results demonstrate that ADCH outperforms several state-of-the-art cross-modal hashing baselines.