Evidence prediction is the cornerstone of litigation, which requires complex legal reasoning to bridge the gap between facts and the parties’ assertions. However, most existing large language model (LLM) benchmarks do not offer a systematic assessment for evidence planning. Therefore, we introduce CLER, a large-scale Chinese Litigation Evidence Reasoning benchmark. CLER is built through a four-stage pipeline from millions of real civil and criminal judgments. The pipeline includes (1) semi-structured data collection, (2) LLM-based evidence extraction, (3) purpose refinement through decomposition into legal elements, and (4) validation through a tri-party adversarial review method that imitates courtroom debate. To this end, we obtain over 100,000 samples and conduct extensive experiments on 23 baselines, including open-weight, commercial, domain-specific LLMs and retrieval-augmented generation (RAG) systems. The findings indicate that the task poses a major challenge for current LLMs. RAG strategies, particularly a hybrid method retrieving similar cases, achieve the best performance with an F1 score of 54.4%, surpassing state-of-the-art proprietary models and improving the base model by at least 22%. Our analysis reveals several key insights. There is a significant performance gap with an average difference F1 of 16.2% exists between element checking in civil cases and narrative completeness in criminal cases. There is an inverted U-shaped correlation between case length and model performance, as insufficient or overloaded information leads to abandonment of precise reasoning, causing an average drop of approximately 10% in F1.

Multi-Label Learning (MLL) involves the task of assigning a set of relevant labels to a given instance. Recently, Label Enhancement (LE) has gained significant attention in various MLL tasks, as it allows for effective mining the implicit relative importance information of different labels. However, in existing LE-based MLL methods, the LE process is decoupled from the MLL process. Consequently, the label distribution recovered by the LE process may not be suitable for training the predictive model, thus affecting the overall learning system. In this study, we propose a novel approach named interactive Fusion Label Enhancement for Multi-Label Learning ( Flem ) that seamlessly integrates the LE process with the MLL process. Specifically, we introduce a matching and interaction mechanism comprising a novel interaction label enhancement loss and a contrastive alignment approach to prevent object mismatch. Furthermore, we present a unified label distribution loss that establishes the relationship between the recovered label distribution and the training of the predictive model. By leveraging these losses, the label distributions obtained from the LE process can be efficiently utilized for training the predictive model. Experimental results on multiple benchmark datasets demonstrate the effectiveness of the proposed method.

Label Distribution Learning (LDL) offers a promising solution to label ambiguity by employing Label Distributions (LDs) instead of traditional logical labels. However, acquiring LDs for real-world data is both expensive and challenging. To address this issue, Label Enhancement (LE) techniques have been proposed to derive LDs from readily available logical labels. While much of the prior work has focused on enhancing LE for better recovery performance, the ultimate objective remains improving LDL’s overall effectiveness. In this paper, we introduce a novel LE method, Topological Label Enhancement via Optimal Transport (TLEOT), which integrates Optimal Transport (OT) theory with topological space analysis. This method goes beyond improving LE, targeting the enhancement of LDL performance by aligning the feature and label distributions within a unified topological framework. Additionally, we present two innovative topological techniques designed to further improve LDL. Extensive experimental evaluations on real-world datasets demonstrate that TLEOT consistently outperforms nine state-of-the-art methods in predictive tasks. Furthermore, the proposed topological techniques significantly enhance LDL’s performance, validating their practical utility in real-world applications.

Numerous researches have proved that deep neural networks (DNNs) can fit almost everything even given data with noisy labels, and result in poor generalization performance. However, recent studies suggest that DNNs tend to gradually memorize the data, moving from correct data to mislabeled data. Inspired by this finding, we propose a novel method namedDynamic Loss Thresholding (DLT). During the training process, DLT records the loss value of each sample and calculates dynamic loss thresholds. Specifically, DLT compares the loss value of each sample with the current loss threshold. Samples with smaller losses can be considered as clean samples with higher probability and vice versa. Then, DLT discards the potentially corrupted labels and further leverages self-training semi-supervised learning techniques. Experiments on CIFAR-10/100, WebVision and Clothing1M demonstrate substantial improvements over recent state-of-the-art methods. In addition, we investigate two real-world problems. Firstly, we propose a novel approach to estimate the noise rates of datasets based on the loss difference between the early and late training stages of DNNs. Secondly, we explore the effect of hard samples (which are difficult to be distinguished) on the process of learning from noisy labels.

Multi-label text classification (MLTC) refers to the problem of tagging a given document with the most relevant subset of labels. One of the biggest challenges for MLTC is the existence of class imbalance. Most advanced MLTC models suffer from this issue, which limits the performance of the models. In this paper, we propose a model-agnostic framework named variational continuous label distribution learning (VCLDL) to address this problem. VCLDL theoretically builds a corresponding relationship between the feature space and the label space to mine the information hidden in the observable logical labels. Specifically, VCLDL regards label distribution as a continuous density function in latent space and forms a flexible variational approach to approximate the density function of the labels with the collaboration of the feature space. Combined with VCLDL, MLTC models can pay more attention to the distribution of the whole label set, rather than specific labels with maximum response values, thus the class imbalance problem can be well overcome. Experimental results on multiple benchmark datasets demonstrate that VCLDL can bring significant performance improvements over the existing MLTC models.

Label Distribution Learning (LDL) has attracted increasing research attentions due to its potential to address the label ambiguity problem in machine learning and success in many real-world applications. In LDL, it is usually expensive to obtain the ground-truth label distributions of data, but it is relatively easy to obtain the logical labels of data. How to use training instances only with logical labels to learn an effective LDL model is a challenging problem. In this paper, we propose a two-step framework to address this problem. Specifically, we firstly design an efficient recovery model to recover the latent label distributions of training instances, named Fast Label Enhancement (FLE). Our idea is to use non-negative matrix factorization (NMF) to mine the label distribution information from the feature space. Moreover, we take the instance-class similarities into consideration to discover the importance of each label to training instances, which is useful for learning precise label distributions. Then, we train a predictive model for testing instances based on generated label distributions of training instances and an existing LDL method (e.g., SA-BFGS). Experimental results on fifteen benchmark datasets show the effectiveness of the proposed two-step framework and verify the superiority of FLE over several state-of-the-art approaches.

Multi-view partial multi-label learning (MVPML) aims to learn a multi-label predictive model from the training examples, each of which is presented by multiple feature vectors while associated with a set of candidate labels where only a subset is correct. Generally, existing techniques work simply by identifying the ground-truth label via aggregating the features from all views to train a final classifier, but ignore the cause of the incorrect labels in the candidate label sets, i.e., the diverse property of the representation from different views leads to the incorrect labels which form the candidate labels alone with the essential supervision. In this paper, a novel MVPML approach is proposed to learn the predictive model and the incorrect-labeling model jointly by incorporating the graph-fusion-based topological structure of the feature space. Specifically, the latent label distribution and the incorrect labels are identified simultaneously in a unified framework under the supervision of candidate labels. In addition, a common topological structure of the feature space from all views is learned via the graph fusion for further capturing the latent label distribution. Experimental results on the real-world datasets clearly validate the effectiveness of the proposed approach for solving multi-view partial multi-label learning problems.

We propose a novel probabilistic label enhancement algorithm, called PLEA, to solve challenging label distribution learning (LDL) for multi-label classification problems. We adopt the well-known maximum entropy model based label distribution learner. However, unlike the existing LDL algorithms based on the maximum entropy model, we propose to use manifold learning to enhance the label distribution learner. Specifically, the supervised information in the label manifold is utilized in the feature manifold space construction to improve the accuracy of feature extraction, while dramatically reducing the feature dimension. Then the robust linear regression is employed to estimate the label distributions associated with the extracted reduced-dimension features. Using the enhanced reduced-dimension features and their associated estimated label distributions in the maximum entropy model, the unknown true label distributions can be estimated more accurately, while imposing considerably lower computational complexity. We evaluate the proposed PLEA method on a wide-range artificial and high-dimensional real-world datasets. Experimental results obtained demonstrate that our proposed PLEA method has advantages in LDL accuracy and runtime performance, compared to the latest multi-label LDL approaches. The results also show that our PLEA compares favourably with the state-of-the-arts multi-label learning algorithms for classification tasks.

Label distribution is more general than both single-label annotation and multi-label annotation. It covers a certain number of labels, representing the degree to which each label describes the instance. The learning process on the instances labeled by label distributions is called label distribution learning (LDL). Unfortunately, many training sets only contain simple logical labels rather than label distributions due to the difficulty of obtaining the label distributions directly. To solve this problem, one way is to recover the label distributions from the logical labels in the training set via leveraging the topological information of the feature space and the correlation among the labels. Such process of recovering label distributions from logical labels is defined as label enhancement (LE), which reinforces the supervision information in the training sets. This paper proposes a novel LE algorithm called Graph Laplacian Label Enhancement (GLLE). Experimental results on one artificial dataset and fourteen real-world LDL datasets show clear advantages of GLLE over several existing LE algorithms. Furthermore, experimental results on eleven multi-label learning datasets validate the advantage of GLLE over the state-of-the-art multi-label learning approaches.

Multi-label learning deals with training examples each represented by a single instance while associated with multiple class labels, and the task is to train a predictive model which can assign a set of proper labels for the unseen instance. Existing approaches employ the common assumption of equal labeling-importance, i.e., all associated labels are regarded to be relevant to the training instance while their relative importance in characterizing its semantics are not differentiated. Nonetheless, this common assumption does not reflect the fact that the importance degree of each relevant label is generally different, though the importance information is not directly accessible from the training examples. In this article, we show that it is beneficial to leverage the implicit relative labeling-importance (RLI) information to help induce multi-label predictive model with strong generalization performance. Specifically, RLI degrees are formalized as multinomial distribution over the label space, which can be estimated by either global label propagation procedure or local k-nearest neighbor reconstruction. Correspondingly, the multi-label predictive model is induced by fitting modeling outputs with estimated RLI degrees along with multi-label empirical loss regularization. Extensive experiments clearly validate that leveraging implicit RLI information serves as a favorable strategy to achieve effective multi-label learning.

The task of multi-label learning is to predict a set of relevant labels for the unseen instance. Traditional multi-label learning algorithms treat each class label as a logical indicator of whether the corresponding label is relevant or irrelevant to the instance, i.e., +1 represents relevant to the instance and -1 represents irrelevant to the instance. Such label represented by -1 or +1 is called logical label. Logical label cannot reflect different label importance. However, for real-world multi-label learning problems, the importance of each possible label is generally different. For the real applications, it is difficult to obtain the label importance information directly. Thus we need a method to reconstruct the essential label importance from the logical multilabel data. To solve this problem, we assume that each multi-label instance is described by a vector of latent real-valued labels, which can reflect the importance of the corresponding labels. Such label is called numerical label. The process of reconstructing the numerical labels from the logical multi-label data via utilizing the logical label information and the topological structure in the feature space is called Label Enhancement. In this paper, we propose a novel multi-label learning framework called LEMLL, i.e., Label Enhanced Multi-Label Learning, which incorporates regression of the numerical labels and label enhancement into a unified framework. Extensive comparative studies validate that the performance of multi-label learning can be improved significantly with label enhancement and LEMLL can effectively reconstruct latent label importance information from logical multi-label data.

Although multi-label learning can deal with many problems with label ambiguity, it does not fit some real applications well where the overall distribution of the importance of the labels matters. This paper proposes a novel learning paradigm named label distribution learning (LDL) for such kind of applications. The label distribution covers a certain number of labels, representing the degree to which each label describes the instance. LDL is a more general learning framework which includes both single-label and multi-label learning as its special cases. This paper proposes six working LDL algorithms in three ways: problem transformation, algorithm adaptation, and specialized algorithm design. In order to compare the performance of the LDL algorithms, six representative and diverse evaluation measures are selected via a clustering analysis, and the first batch of label distribution datasets are collected and made publicly available. Experimental results on one artificial and 15 real-world datasets show clear advantages of the specialized algorithms, which indicates the importance of special design for the characteristics of the LDL problem.

In multi-label learning, each training example is represented by a single instance while associated with multiple labels, and the task is to predict a set of relevant labels for the unseen instance. Existing approaches learn from multi-label data by assuming equal labeling-importance, i.e. all the associated labels are regarded to be relevant while their relative importance for the training example are not differentiated. Nonetheless, this assumption fails to reflect the fact that the importance degree of each associated label is generally different, though the importance information is not explicitly accessible from the training examples. In this paper, we show that effective multi-label learning can be achieved by leveraging the implicit relative labeling-importance (RLI) information. Specifically, RLI degrees are formalized as multinomial distribution over the label space, which are estimated by adapting an iterative label propagation procedure. After that, the multi-label prediction model is learned by fitting the estimated multinomial distribution as regularized with popular multi-label empirical loss. Comprehensive experiments clearly validate the usefulness of leveraging implicit RLI information to learn from multi-label data.

Visual methods have been widely studied and used in data cluster analysis. Given a pairwise dissimilarity matrix {\schmi D} of a set of n objects, visual methods such as the VAT algorithm generally represent {\schmi D} as an n\times n image {\rm I}(\tilde{{\schmi D}}) where the objects are reordered to reveal hidden cluster structure as dark blocks along the diagonal of the image. A major limitation of such methods is their inability to highlight cluster structure when {\schmi D} contains highly complex clusters. This paper addresses this limitation by proposing a Spectral VAT algorithm, where {\schmi D} is mapped to {\schmi D}^{\prime } in a graph embedding space and then reordered to {{\tilde{\schmi D}^{\prime }}} using the VAT algorithm. A strategy for automatic determination of the number of clusters in {\rm I}({\tilde{{\schmi D}^{\prime }}}) is then proposed, as well as a visual method for cluster formation from {\rm I}({\tilde{{\schmi D}^{\prime }}}) based on the difference between diagonal blocks and off-diagonal blocks. A sampling-based extended scheme is also proposed to enable visual cluster analysis for large data sets. Extensive experimental results on several synthetic and real-world data sets validate our algorithms.

Given a pairwise dissimilarity matrix D of a set of objects, visual methods such as the VAT algorithm (for visual analysis of cluster tendency) represent (D macr )as an image (D macr ) where the objects are reordered to highlight cluster structure as dark blocks along the diagonal of the image. A major limitation of such visual methods is their inability to highlight cluster structure in 1(D macr ) when D contains clusters with highly complex structure. In this paper, we address this limitation by proposing a Spectral VAT (SpecVAT) algorithm, where D is mapped to D' in an embedding space by spectral decomposition of the Laplacian matrix, and then reordered to D' using the VAT algorithm. We also propose a strategy to automatically determine the number of clusters in (D macr '), as well as a method for cluster formation from (D macr ') based on the difference between diagonal blocks and off-diagonal blocks. We demonstrate the effectiveness of our algorithms on several synthetic and real-world data sets that are not amenable to analysis via traditional VAT.