With growing concerns over user privacy, federated recommendation (FedRec) has emerged as a mainstream solution for personalized recommendation services. FedRec trains user-private parameters on local clients while collaboratively updating global parameters on a centralized server. However, despite advances in optimizing these local and global parameters, existing methods overlook two key challenges: tradeoff training and distribution discrepancy . Tradeoff training balances timely local updates with diverse global parameters, limiting the model’s learning ability. Distribution discrepancy arises from the divergence between locally trained global parameters and those aggregated by the server, corrupting inference performance. To fill in the gap, we propose FedSC , a principled federated recommendation framework that boosts FedRec’s training and inference processes with minimal yet nontrivial efforts. During training, FedSC employs a stochastic aggregation strategy where all users participate in every round, while only a random subset is selected for aggregation, preserving the diversity of global parameters and ensuring timely local updates. During inference, FedSC makes recommendations with a consistency inference mechanism that uses the most recent locally trained global parameters of each user to improve the model’s understanding of user preferences. Extensive experiments on multiple benchmark datasets demonstrate the superiority of FedSC, achieving up to a 20% improvement in most evaluation scenarios.

Federated Recommendation (FedRec) systems are recognized as privacy-preserving solutions for collaboratively training recommender models without sharing users’ private data. However, recent studies have revealed that FedRec systems are vulnerable to interaction-level membership inference attacks. In such attacks, a semi-honest server can employ crafted methods to infer users’ interacted items. In this article, we identify that user preference information is predominantly stored in the user-uploaded parameters rather than in the local parameters after local training. Leveraging this insight, we expose a new interaction vulnerability and introduce the PubPara attack. Our experiments show that PubPara improves the inference performance by at least 40% over existing attacks, while requiring minimal inference time and remaining robust against current defense methods. To safeguard user privacy without compromising recommender performance, we propose MultiVerse, a novel countermeasure. MultiVerse utilizes untrained items outside the user’s local training data to obfuscate the server’s inference of interacted items. It includes a four-step strategy (training, optimization, refinement, and denoising) to achieve robust defense. Extensive experiments on three representative FedRec models (F-NCF, F-LightGCN, and FedRAP) across three real-world datasets validate that MultiVerse significantly degrades the attack’s inference performance to near the level of random guess while maintaining lossless recommender performance.

Crowdsourcing is a promising human-in-the-loop paradigm for processing computer hard tasks by harnessing crowd intelligence. However, canonical crowdsourcing systems mostly need to aggregate/transmit worker data and may lead to privacy-leakage. To tackle this problem, we propose a novel approach, called FedCS (Federated CrowdSourcing), to achieve privacy protection while ensuring quality. FedCS aggregates model parameters from clients to build a shared server model while keeping the training data locally on worker devices to protect data privacy. To mitigate the staleness of stragglers and boost efficiency, we introduce a semi-asynchronous federated crowdsourcing mechanism, where the parameter server performs global aggregation periodically. Moreover, due to the different frequencies of workers participating in asynchronous update, FedCS uses a staleness-aware grouping and weighted aggregation heuristic to balance the training process. To speed up the convergence rate and improve the training accuracy, FedCS deploys adaptive learning step size for worker devices by their participation frequency. We further present a task assignment algorithm to help workers choose worthy and suitable tasks for annotations and to save the budget. Extensive experiments on benchmark datasets and a real-world crowdsourcing project show that FedCS can complete secure crowdsourcing projects with high quality and low budget.

A recent trend of fair machine learning is to build a decision model subjected to causality-based fairness requirements, which concern with the causality between sensitive attributes and decisions. Almost all (if not all) solutions focus on a single fair decision model and assume no hidden confounder to model causal effects in a too simplified way. However, multiple interdependent decision models are actually used and discrimination may transmit among them. The hidden confounder is another inescapable fact and causal effects cannot be computed from observational data in the unidentifiable situation. To address these problems, we propose a method called CMFL (Causality-based Multiple Fairness Learning). CMFL parameterizes the causal model by response-function variables, whose distributions capture the randomness of causal models. CMFL treats each classifier as a soft intervention to infer the post-intervention distribution, and combines the fairness constraints with the classification loss to train multiple decision classifiers. In this way, all classifiers can make approximately fair decisions. Experiments on synthetic and benchmark datasets confirm its effectiveness, the response-function variables can deal with the unidentifiable issue and hidden confounders.

Discovery the causal structure graph among a set of variables is a fundamental but difficult task in many empirical sciences. Reinforcement learning based causal discovery from observed data achieves prominent results. However, previous algorithms lack interpretability and efficiency, and ignore the prior knowledge of causal structure. To solve these problems, we propose GARL that leverages graph attention network to embed the structure information and the prior knowledge, and reinforcement learning to search the variable ordering with the best score. GARL takes the structure information and prior knowledge as the computational skeleton of attention to obtain the embedded representation of variables, and then generates variable orderings through the designed ordering model. In addition, the structure information is used to form the DAG corresponding to the variable ordering, which reduces the computational difficulty and improves the efficiency. GARL generates DAGs in the reinforcement learning framework, and uses the score of DAG as the reward to optimize the network structure to search the DAG with the best score. Experimental results on synthetic and real datasets show that our GARL has obvious advantages in multi-node operation efficiency, and competitive results with competitive baselines.

Learning the directed acyclic graph (DAG) among causal variables is a fundamental pre-task in causal discovery. Available DAG learning solutions canonically focus on homogeneous nodes with multiple variables and assume i.i.d. samples, how to learn DAG on typical attributed heterogeneous network (AHN) composed with different types of inter-dependent nodes and diverse attributes is a practical but more difficult task. In this paper, we propose HetDAG to identify DAG among nodes from heterogeneous network. HetDAG first embeds different types of node attributes and aggregates these embeddings as the node's raw representation. Then it uses contrastive learning with prior network structure to explore latent relationships between nodes and update the representation. Next, HetDAG introduces an attention-based DAG learning module that takes node representations as input to search DAG and orient edges between nodes. To the best of our knowledge, HetDAG is the first study to learn DAG on heterogeneous networks. Extensive experiments on both semi-synthetic and real data show that HetDAG can learn DAG in an efficacy way and outperforms the state-of-the-art approaches. The results on real biological networks confirm that HetDAG can find out the causal relations between lncRNAs and miRNAs.

Multiple clusterings can explore the same set of data from different perspectives by discovering different and meaningful clusterings. However, most, if not all, of the existing approaches overwhelmingly focus on the diversity between clustering subspaces, and pay much less attention on the salience of the subspaces. As a consequence, the quality of the produced clusterings is an understudied aspect of the problem. Furthermore, existing methods cannot explain the unique internal subspace structure of each clustering, and cannot incorporate multi-facet knowledge to generate different clusterings. In this paper, we propose a solution namediMClusts(interpretableMultipleClusteringsby diversified attention). iMClusts makes use of the expressive representational power of deep autoencoders and multi-head attention to generate multiple salient embedding matrices, and multiple clusterings therein. In addition, it leverages multi-facet knowledge and enhances the diversity between heads to boost the quality and diversity of multiple clusterings. Experimental results on benchmark datasets show that iMClusts can generate multiple clusterings with quality, interpretability, and diversity.

Multi-label active learning (MAL) aims to learn an accurate multi-label classifier by selecting which examples (or example-label pairs) will be annotated and reducing query effort. MAL is a more complicated and expensive process than single-label active learning, due to one example can be associated with a set of non-exclusive labels and the annotator has to scrutinize the whole example and label space to provide correct annotations. Instead of scrutinizing the whole example for annotation, we may just examine some of its subexamples with respect to a label for annotation. In this way, we can not only save the annotation cost but also speedup the annotation process. Given this observation, we introduce CMAL, a two-stage Cost-effective MAL strategy (CMAL) by querying subexamples. CMAL first selects the most informative example-label pairs by leveraging uncertainty, label correlation and label space sparsity. Specifically, the uncertainty of a label to an example can be reduced if its correlated labels already annotated to the example, and its uncertainty can be reduced also if more examples annotated to this label. Next, CMAL greedily queries the most probable positive subexample-label pairs of the selected example-label pair. In addition, we propose rCMAL to account for the representative of examples to more reliably select example-label pairs in the first stage. Extensive experiments on multi-label datasets from diverse domains show that our proposed CMAL and rCMAL can better save the query cost than state-of-the-art MAL methods. The contribution of leveraging label correlation, label sparsity, and representative for saving cost is also confirmed.

Crowdsourcing is a popular and relatively economic way to harness human intelligence to process computer-hard tasks. Due to diverse factors (i.e., task difficulty, worker capability, and incentives), the collected answers from various crowd workers are of different quality. Many approaches have been proposed to manage high quality answers and to reduce the budget by modelling tasks, workers, or both. However, most of the existing approaches implicitly assume that the capability of workers is fixed during the crowdsourcing process. But in practice, such capability can be improved by gradually completing easy to hard tasks, alike human beings’ intrinsic self-paced learning ability. In this paper, we investigate crowdsourcing with self-paced workers, whose capability can be gradually boosted as he/she scrutinises and completes easy to hard tasks. Our proposed SPCrowd (Self-Paced Crowd worker) first asks workers to complete a set of golden tasks with known annotations; provides feedback to assist workers with capturing the raw modes of tasks and to spark the self-paced learning, which in turn facilitates the estimation of workers’ quality and tasks’ difficulty. It then introduces a task difficulty model to quantify the difficulty of tasks and rank them from easy to hard, and a benefit maximization criterion for task assignment, which can dynamically monitor the quality of self-paced workers and assign the sorted tasks to capable workers. In this way, a worker can successfully complete hard tasks after he/she completes easier and related tasks. Experimental results on semi-simulated and real crowdsourcing projects show that SPCrowd can better control the quality and save the budget compared to competitive baselines.

Multi-instance learning (MIL) can model complex bags (samples) that are further made of diverse instances (subsamples). In typical MIL, the labels of bags are known while those of individual instances are unknown and to be specified. In this paper we propose an imbalanced deep multi-instance learning approach (IDMIL-III) and apply it to predict genome-wide isoform–isoform interactions (IIIs). This prediction task is crucial for precisely understanding the interactome between proteoforms and to reveal their functional diversity. The current solutions typically formulate the prediction of IIIs as a MIL problem by pairing two genes as a “bag” and any two isoforms spliced from these two genes as “instances.” The key instances (interacting isoform pairs) trigger the label of the positive (interacting) gene bags, which is important for identifying the IIIs. Furthermore, the prediction task was simplified as a balanced classification problem, which in practice is a rather imbalanced one. To address these issues, IDMIL-III fuses RNA-seq, nucleotide sequence, amino acid sequence and exon array data, and further introduces a novel loss function to separately model the loss of positive pairs and of negative pairs, and thus to avoid the expected loss dominated by majority negative pairs. In addition, it includes an attention strategy to identify positive isoform pairs from a positive gene bag. Extensive experimental results prove the effectiveness of IDMIL-III on predicting IIIs. Particularly, IDMIL-III achieves an F1 value as 95.4%, at least 3.8% higher than those of competitive methods at the gene-level; and obtains an F1 as 29.8%, at least 2.4% higher than the state-of-the-art methods at the isoform-level. The code of IDMIL-III is available at http://mlda.swu.edu.cn/codes.php?name=IDMIL-III.

Crowdsourcing is a relatively inexpensive and efficient mechanism to collect annotations of data from the open Internet. Crowdsourcing workers are paid for the provided annotations, but the task requester usually has a limited budget. It is desirable to wisely assign the appropriate task to the right workers, so the overall annotation quality is maximized while the cost is reduced. In this article, we propose a novel task assignment strategy (CrowdWT) to capture the complex interactions between tasks and workers, and properly assign tasks to workers. CrowdWT first develops a Worker Bias Model (WBM) to jointly model the worker’s bias, the ground truths of tasks, and the task features. WBM constructs a mapping between task features and worker annotations to dynamically assign the task to a group of workers, who are more likely to give correct annotations for the task. CrowdWT further introduces a Task Difficulty Model (TDM), which builds a Kernel ridge regressor based on task features to quantify the intrinsic difficulty of tasks and thus to assign the difficult tasks to more reliable workers. Finally, CrowdWT combines WBM and TDM into a unified model to dynamically assign tasks to a group of workers and recall more reliable and even expert workers to annotate the difficult tasks. Our experimental results on two real-world datasets and two semi-synthetic datasets show that CrowdWT achieves high-quality answers within a limited budget, and has the best performance against competitive methods.<?vsp -1.5pt?>

Co-clustering aims at discovering groups of both objects and features from a given data matrix. Co-clustering ensembles can produce robust co-clusters by combining multiple base co-clusterings. However, current co-clustering ensemble solutions either ignore the constraints resulting from feature-to-feature and object-to-object relevance information, or ignore feature-to-object relevance information. In this paper, we advocate that all three information sources contribute to the achievement of good consensus solutions, and propose a co-clustering ensemble (CoCE) approach based on multiple relevance measures. CoCE first evaluates the quality of base co-clusters and consequently measures feature-to-object relevance. The latter, along with feature-to-feature and object-to-object relevance measures, contribute to the definition of a hybrid graph. The consensus process uses the resulting hybrid graph; it's formulated as a trace minimization problem and introduces a block-wise matrix multiplication technique to perform the optimization. Experimental results on various datasets show that CoCE not only frequently outperforms other related co-clustering ensembles, but also has reduced runtime cost and is more robust to poor base co-clusterings.

Multi-view clustering aims at exploiting information from multiple heterogeneous views to promote clustering. Most previous works search for only one optimal clustering based on the predefined clustering criterion, but devising such a criterion that captures what users need is difficult. Due to the multiplicity of multi-view data, we can have meaningful alternative clusterings. In addition, the incomplete multi-view data problem is ubiquitous in real world but has not been studied for multiple clusterings. To address these issues, we introduce a deep incomplete multi-view multiple clusterings (DiMVMC) framework, which achieves the completion of data view and multiple shared representations simultaneously by optimizing multiple groups of decoder deep networks. In addition, it minimizes a redundancy term to simultaneously control the diversity among these representations and among parameters of different networks. Next, it generates an individual clustering from each of these shared representations. Experiments on benchmark datasets confirm that DiMVMC outperforms the state-of-the-art competitors in generating multiple clusterings with high diversity and quality.

Multi-typed objects Multi-view Multi-instance Multi-label Learning (M4L) deals with interconnected multi-typed objects (or bags) that are made of diverse instances, represented with heterogeneous feature views and annotated with a set of non-exclusive but semantically related labels. M4L is more general and powerful than the typical Multi-view Multi-instance Multi-label Learning (M3L), which only accommodates single-typed bags and lacks the power to jointly model the naturally interconnected multi-typed objects in the physical world. To combat with this novel and challenging learning task, we develop a joint matrix factorization based solution (M4L-JMF). Particularly, M4L-JMF firstly encodes the diverse attributes and multiple inter(intra)-associations among multi-typed bags into respective data matrices, and then jointly factorizes these matrices into low-rank ones to explore the composite latent representation of each bag and its instances (if any). In addition, it incorporates a dispatch and aggregation term to distribute the labels of bags to individual instances and reversely aggregate the labels of instances to their affiliated bags in a coherent manner. Experimental results on benchmark datasets show that M4L-JMF achieves significantly better results than simple adaptions of existing M3L solutions on this novel problem.

Partial multi-label learning (PML) aims at learning a robust multi-label classifier from partial multi-label data, where a sample is annotated with a set of candidate labels, while only a subset of those labels is valid. The existing PML algorithms generally suffer from the high computational cost when learning with large label spaces. In this paper, we introduce a PML approach (PML-LCom) that uses Label Compression to efficiently learn from partial multi-label data. PML-LCom firstly splits the observed label data matrix into a latent relevant label matrix and an irrelevant one, and then factorizes the relevant label matrix into two low-rank matrices, one encodes the compressed labels of samples, and the other explores the underlying label correlations. Next, it optimizes the coefficient matrix of the multi-label predictor with respect to the compressed label matrix. In addition, it regularizes the compressed label matrix with respect to the feature similarity of samples, and optimizes the label matrix and predictor in a coherent manner. Experimental results on both semi-synthetic and real-world PML datasets show that PML-LCom achieves a performance superior to the state-of-the-art solutions on predicting the labels of unlabeled samples with a large label space. The label compression improves both the effectiveness and efficiency, and the coherent optimization mutually benefits the label matrix and predictor.

Crowdsourcing is a relatively economic and efficient solution to collect annotations from the crowd through online platforms. Answers collected from workers with different expertise may be noisy and unreliable, and the quality of annotated data needs to be further maintained. Various solutions have been attempted to obtain high-quality annotations. However, they all assume that workers' label quality is stable over time (always at the same level whenever they conduct the tasks). In practice, workers' attention level changes over time, and the ignorance of which can affect the reliability of the annotations. In this paper, we focus on a novel and realistic crowdsourcing scenario involving attention-aware annotations. We propose a new probabilistic model that takes into account workers' attention to estimate the label quality. Expectation propagation is adopted for efficient Bayesian inference of our model, and a generalized Expectation Maximization algorithm is derived to estimate both the ground truth of all tasks and the label-quality of each individual crowd worker with attention. In addition, the number of tasks best suited for a worker is estimated according to changes in attention. Experiments against related methods on three real-world and one semi-simulated datasets demonstrate that our method quantifies the relationship between workers' attention and label-quality on the given tasks, and improves the aggregated labels.

Hashing has been widely studied for big data retrieval due to its low storage cost and fast query speed. Zero-shot hashing (ZSH) aims to learn a hashing model that is trained using only samples from seen categories, but can generalize well to samples of unseen categories. ZSH generally uses category attributes to seek a semantic embedding space to transfer knowledge from seen categories to unseen ones. As a result, it may perform poorly when labeled data are insufficient. ZSH methods are mainly designed for single-modality data, which prevents their application to the widely spread multi-modal data. On the other hand, existing cross-modal hashing solutions assume that all the modalities share the same category labels, while in practice the labels of different data modalities may be different. To address these issues, we propose a general Cross-modal Zero-shot Hashing (CZHash) solution to effectively leverage unlabeled and labeled multi-modality data with different label spaces. CZHash first quantifies the composite similarity between instances using label and feature information. It then defines an objective function to achieve deep feature learning compatible with the composite similarity preserving, category attribute space learning, and hashing coding function learning. CZHash further introduces an alternative optimization procedure to jointly optimize these learning objectives. Experiments on benchmark multi-modal datasets show that CZHash significantly outperforms related representative hashing approaches both on effectiveness and adaptability.

Multiple clustering approaches aim at exploring alternative ways of organizing a given collection of data into various clusters from different perspectives. Although multiple one-way clusterings have been studied for more than a decade, how to explore alternative two-way clusterings (or co-clusterings) still remains an untouched topic, and an important one from an application standpoint. To solve this interesting but yet unexplored topic, we assume the existence of alternative co-clusterings embedded in different subspaces and simultaneously pursue multiple co-clusterings therein. We initially specify a subspace indicator matrix for each feature subspace, and employ matrix tri-factorization to seek row-wise and column-wise cluster indicator matrices in each subspace. To ensure diversity, we quantify the redundancy between pairwise co-clusterings using the cluster indicator and the subspace indicator matrices. We further introduce a unified objective function to simultaneously account for the two pursues, and an alternating optimization solution to iteratively optimize cluster indicator and feature indicator matrices. Our empirical study shows that the proposed solution can explore multiple meaningful co-clusterings and generally achieves better results than state-of-the-art methods.

Multi-label active learning addresses the scarce labeled example problem by querying the most valuable unlabeled examples, or example-label pairs, to achieve a better performance with limited query cost. Current multi-label active learning methods require the scrutiny of the whole example in order to obtain its annotation. In contrast, one can find positive evidence with respect to a label by examining specific patterns (i.e., subexample), rather than the whole example, thus making the annotation process more efficient. Based on this observation, we propose a novel two-stage cost effective multi-label active learning framework, called CMAL. In the first stage, a novel example-label pair selection strategy is introduced. Our strategy leverages label correlation and label space sparsity of multi-label examples to select the most uncertain example-label pairs. Specifically, the unknown relevant label of an example can be inferred from the correlated labels that are already assigned to the example, thus reducing the uncertainty of the unknown label. In addition, the larger the number of relevant examples of a particular label, the smaller the uncertainty of the label is. In the second stage, CMAL queries the most plausible positive subexample-label pairs of the selected example-label pairs. Comprehensive experiments on multi-label datasets collected from different domains demonstrate the effectiveness of our proposed approach on cost effective queries. We also show that leveraging label correlation and label sparsity contribute to saving costs.

Crowdsourcing is a useful and economic approach to data annotation. To obtain annotation of high quality, various aggregation approaches have been developed, which take into account different factors that impact the quality of aggregated answers. However, existing methods generally focus on single-label (multi-class and binary) tasks, and they ignore the inter-correlation between labels, and thus may have compromised quality. In this paper, we introduce a Multi-Label answer aggregation approach based on Joint Matrix Factorization (ML-JMF). ML-JMF selectively and jointly factorizes the sample-label association matrices collected from different annotators into products of individual and shared low-rank matrices. As such, it takes advantage of the robustness of low-rank matrix approximation to noise, and reduces the impact of unreliable annotators by assigning small (zero) weights to their annotation matrices. In addition, it takes advantage of the correlation among labels by leveraging the shared low-rank matrix, and of the similarity between annotators using the individual low-rank matrices to guide the factorization. ML-JMF pursues the low-rank matrices via a unified objective function, and introduces an iterative technique to optimize it. ML-JMF finally uses the optimized low-rank matrices and weights to infer the ground-truth labels. Our experimental results on multi-label datasets show that ML-JMF outperforms competitive methods in inferring ground truth labels. Our approach can identify unreliable annotators, and is robust against their misleading answers through the assignment of small (zero) weights to their annotation.

The goal of multiple clusterings is to discover multiple independent ways of organizing a dataset into clusters. Current approaches to this problem just focus on one-way clustering. In many real-world applications, though, it's meaningful and desirable to explore alternative two-way clustering (or co-clusterings), where both samples and features are clustered. To tackle this challenge and unexplored problem, in this paper we introduce an approach, called Multiple Co-Clusterings (MultiCC), to discover non-redundant alternative co-clusterings. MultiCC makes use of matrix tri-factorization to optimize the sample-wise and feature-wise co-clustering indicator matrices, and introduces two non-redundancy terms to enforce diversity among co-clusterings. We then combine the objective of matrix tri-factorization and two non-redundancy terms into a unified objective function and introduce an iterative solution to optimize the function. Experimental results show that MultiCC outperforms existing multiple clustering methods, and it can find interesting co-clusters which cannot be discovered by current solutions.

Current efforts on multi-label learning generally assume that the given labels of training instances are noise-free. However, obtaining noise-free labels is quite difficult and often impractical, and the presence of noisy labels may compromise the performance of multi-label learning. Partial multi-label learning (PML) addresses the scenario in which each instance is annotated with a set of candidate labels, of which only a subset corresponds to the ground-truth. The PML problem is more challenging than partial-label learning, since the latter assumes that only one label is valid and may ignore the correlation among candidate labels. To tackle the PML challenge, we introduce a feature induced PML approach called fPML, which simultaneously estimates noisy labels and trains multi-label classifiers. In particular, fPML simultaneously factorizes the observed instance-label association matrix and the instance-feature matrix into low-rank matrices to achieve coherent low-rank matrices from the label and the feature spaces, and a low-rank label correlation matrix as well. The low-rank approximation of the instance-label association matrix is leveraged to estimate the association confidence. To predict the labels of unlabeled instances, fPML learns a matrix that maps the instances to labels based on the estimated association confidence. An empirical study on public multi-label datasets with injected noisy labels, and on archived proteomic datasets, shows that fPML can more accurately identify noisy labels than related solutions, and consequently can achieve better performance on predicting labels of instances than competitive methods.

Weak-label learning is an important branch of multi-label learning; it deals with samples annotated with incomplete (weak) labels. Previous work on weak-label learning mainly considers data represented by a single view. An intuitive way to leverage multiple features obtained from different views is to concatenate the features into a single vector. However, this process is not only prone to over-fitting and often results in very high time-complexity, but also ignores the potentially useful complementary information spread across the different views. In this paper, we propose an approach based on Matrix Completion for multi-view Weak-label Learning (McWL). Matrix completion (MC) has sound theoretical properties and is robust to missing values in both feature and label spaces. Our method enforces the optimization of multiple view integration and of MC-based classification within a unified objective function. Specifically, a kernel target alignment technique and the loss function of an MC-based classifier are used to jointly and iteratively adjust the weights assigned to individual views, and to optimize the classifier. McWL can selectively integrate views and is able to assign small weights to views of low quality. Extensive experiments on a broad range of datasets validate the effectiveness of our approach against competitive algorithms.