Anomaly detection, the task of differentiating abnormal data points from normal ones, presents a significant challenge in the realm of machine learning. Numerous strategies have been proposed to tackle this task, with classification-based methods, specifically those utilizing a self-supervised approach via random affine transformations (RATs), demonstrating remarkable performance on both image and non-image data. However, these methods encounter a notable bottleneck, the overlap of constructed labeled datasets across categories, which hampers the subsequent classifiers' ability to detect anomalies. Consequently, the creation of an effective data distribution becomes the pivotal factor for success. In this article, we introduce a model called "self-supervised forest (sForest)," which leverages the random Fourier transform (RFT) and random orthogonal rotations to craft a controlled data distribution. Our model utilizes the RFT to map input data into a new feature space. With this transformed data, we create a self-labeled training dataset using random orthogonal rotations. We theoretically prove that the data distribution formulated by our methodology is more stable compared to one derived from RATs. We then use the self-labeled dataset in a random forest (RF) classifier to distinguish between normal and anomalous data points. Comprehensive experiments conducted on both real and artificial datasets illustrate that sForest outperforms other anomaly detection methods, including distance-based, kernel-based, forest-based, and network-based benchmarks.

Learning from label proportions (LLPs), which aims to learn an instance-level classifier using proportion-based grouped training data, has garnered increasing attention in the field of machine learning. Existing deep learning-based LLP methods employ end-to-end pipelines to derive proportional loss functions via the Kullback-Leibler (KL) divergence between bag-level prior and posterior class distributions. However, the optimal solutions of these methods often struggle to conform to the given proportions, inevitably leading to degradation in the final classification performance. In this article, we address this issue by proposing a novel progressive training method for LLP, termed PT-LLP, which strives to meet the proportion constraints from the bag level to the instance level. Specifically, we first train a model by using the existing KL-divergence-based LLP methods that are consistent with bag-level proportion information. Then, we impose additional constraints on strict proportion consistency to the classifier to further move toward a more ideal solution by reformulating it as a constrained optimization problem, which can be efficiently solved using optimal transport (OT) algorithms. In particular, the knowledge distillation is employed as a transition stage to transfer the bag-level information to the instance level using a teacher-student framework. Finally, our framework is model-agnostic and demonstrates significant performance improvements through extensive experiments on different datasets when incorporated into other deep LLP methods as the first training stage.

Abstract Recently, neural abstractive text summarization (NATS) models based on sequence-to-sequence architecture have drawn a lot of attention. Real-world texts that need to be summarized range from short news with dozens of words to long reports with thousands of words. However, most existing NATS models are not good at summarizing long documents, due to the inherent limitations of their underlying neural architectures. In this paper, we focus on the task of long document summarization (LDS). Based on the inherent section structures of source documents, we divide an abstractive LDS problem into several smaller-sized problems. In this circumstance, how to provide a less-biased target summary as the supervision for each section is vital for the model’s performance. As a preliminary, we formally describe the section-to-summary-sentence (S2SS) alignment for LDS. Based on this, we propose a novel NATS framework for the LDS task. Our framework is built based on the theory of unbalanced optimal transport (UOT), and it is named as UOTSumm. It jointly learns three targets in a unified training objective, including the optimal S2SS alignment, a section-level NATS summarizer, and the number of aligned summary sentences for each section. In this way, UOTSumm directly learns the text alignment from summarization data, without resorting to any biased tool such as ROUGE. UOTSumm can be easily adapted to most existing NATS models. And we implement two versions of UOTSumm, with and without the pretrain-finetune technique. We evaluate UOTSumm on three publicly available LDS benchmarks: PubMed , arXiv , and GovReport . UOTSumm obviously outperforms its counterparts that use ROUGE for the text alignment. When combined with UOTSumm, the performance of two vanilla NATS models improves by a large margin. Besides, UOTSumm achieves better or comparable performance when compared with some recent strong baselines.

Recently, the attention mechanism boosts the performance of many neural network models in Natural Language Processing (NLP). Among the various attention mechanisms, Multi-Head Attention (MHA) is a powerful and popular variant. MHA helps the model to attend to different feature subspaces independently which is an essential component of Transformer. Despite its success, we conjecture that the different heads of the existing MHA may not collaborate properly. To validate this assumption and further improve the performance of Transformer, we study the collaboration problem for MHA in this paper. First, we propose the Single-Layer Collaboration (SLC) mechanism to help each attention head improve its attention distribution based on the feedback of other heads. Furthermore, we extend SLC to the cross-layer Multi-Head Dense Collaboration (MHDC) mechanism. MHDC helps each MHA layer learn the attention distributions considering the knowledge from the other MHA layers. Both SLC and MHDC are implemented as lightweight modules with very few additional parameters. When equipped with these modules, our new framework, i.e., Collaborative TransFormer (CollFormer), significantly outperforms the vanilla Transformer on a range of NLP tasks, including machine translation, sentence semantic relatedness, natural language inference, sentence classification, and reading comprehension. Besides, we also carry out extensive quantitative experiments to analyze the properties of the MHDC in different settings. The experimental results validate the effectiveness and universality of MHDC as well asCollFormer.

Learning from label proportions (LLP) aims at learning an instance-level classifier with label proportions in grouped training data. Existing deep learning based LLP methods utilize end-to-end pipelines to obtain the proportional loss with Kullback-Leibler divergence between the bag-level prior and posterior class distributions. However, the unconstrained optimization on this objective can hardly reach a solution in accordance with the given proportions. Besides, concerning the probabilistic classifier, this strategy unavoidably results in high-entropy conditional class distributions at the instance level. These issues further degrade the performance of the instance-level classification. In this paper, we regard these problems as noisy pseudo labeling, and instead impose the strict proportion consistency on the classifier with a constrained optimization as a continuous training stage for existing LLP classifiers. In addition, we introduce the mixup strategy and symmetric cross-entropy to further reduce the label noise. Our framework is model-agnostic, and demonstrates compelling performance improvement in extensive experiments, when incorporated into other deep LLP models as a post-hoc phase.

Event-based social networks (EBSNs) are becoming popular in recent years. Users can publish a planned event on an EBSN website, calling for other users to participate in the event. When a user is making a decision on whether to participate in an event in EBSNs, one aspect for consideration is existing participants defined as users who have agreed to join this event. Existing participants of the event may affect the decision of the user, to which we refer as participant influence. However, participant influence is not well studied by previous works. In this article, we propose an event recommendation model that considers participant influence, and exploits the influence of existing participants on the decisions of new participants based on Poisson factorization. The effect of participant influence is associated with the target event, the host group of the event, and the location of the event. Furthermore, our proposed model can extract latent event topics from event text descriptions, and characterize events, groups, and locations by distributions of event topics. Associations between latent event topics and participant influence are exploited for improving event recommendation. Besides making event recommendation, the proposed model is able to reveal the semantic properties of the participant influence between two users semantically. We have conducted extensive experiments on some datasets extracted from a real-world EBSN. Our proposed model achieves superior event recommendation performance over several state-of-the-art models. The results demonstrate that the consideration of participant influence can improve event recommendation.