The surge in scientific submissions has placed increasing strain on the traditional peer-review process, prompting the exploration of large language models (LLMs) for automated review generation. While LLMs demonstrate competence in producing structured and coherent feedback, their capacity for critical reasoning, contextual grounding, and quality sensitivity remains limited. To systematically evaluate these aspects, we propose a comprehensive evaluation framework that integrates semantic similarity analysis and structured knowledge graph metrics to assess LLM-generated reviews against human-written counterparts. We construct a large-scale benchmark of 1,683 papers and 6,495 expert reviews from ICLR and NeurIPS in multiple years, and generate reviews using five LLMs. Our findings show that LLMs perform well in descriptive and affirmational content, capturing the main contributions and methodologies of the original work, with GPT-4o highlighted as an illustrative example, generating 15.74% more entities than human reviewers in the strengths section of good papers in ICLR 2025. However, they consistently underperform in identifying weaknesses, raising substantive questions, and adjusting feedback based on paper quality. GPT-4o produces 59.42% fewer entities than real reviewers in the weaknesses and increases node count by only 5.7% from good to weak papers, compared to 50% in human reviews. Similar trends are observed across all conferences, years, and models, providing empirical foundations for understanding the merits and defects of LLM-generated reviews and informing the development of future LLM-assisted reviewing tools. Data, code, and more detailed results are publicly available at https://github.com/RichardLRC/Peer-Review.

Figure 1: Generate corresponding document images based on the existing layout.One layout can generate an infinite number of diverse document images with complex layout and authentic-looking text.

Several recently developed neural network models have shown their potential for automated text summarization. However, the evaluation results of these models on summarization of long text are fairly close in almost every major evaluation parameter. None of these models including large language models GPT-3.5 and GPT-4 can well summarize long text without manual interventions. In this paper, we report the evaluation results of several state-of-the-art neural network models on text summarization under different configurations of the input, which includes 1630 U.S. patent documents. Based on the evaluation results, we proposed a strategy for improving the text summarization in long text and demonstrated its effectiveness with new cases.

BACKGROUND: Biomedical sciences, with their focus on human health and disease, have attracted unprecedented attention in the 21st century. The proliferation of biomedical sciences has also led to a large number of scientific articles being produced, which makes it difficult for biomedical researchers to find relevant articles and hinders the dissemination of valuable discoveries. To bridge this gap, the research community has initiated the article recommendation task, with the aim of recommending articles to biomedical researchers automatically based on their research interests. Over the past two decades, many recommendation methods have been developed. However, an algorithm-level comparison and rigorous evaluation of the most important methods on a shared dataset is still lacking. METHOD: In this study, we first investigate 15 methods for automated article recommendation in the biomedical domain. We then conduct an empirical evaluation of the 15 methods, including six term-based methods, two word embedding methods, three sentence embedding methods, two document embedding methods, and two BERT-based methods. These methods are evaluated in two scenarios: article-oriented recommenders and user-oriented recommenders, with two publicly available datasets: TREC 2005 Genomics and RELISH, respectively. RESULTS: Our experimental results show that the text representation models BERT and BioSenVec outperform many existing recommendation methods (e.g., BM25, PMRA, XPRC) and web-based recommendation systems (e.g., MScanner, MedlineRanker, BioReader) on both datasets regarding most of the evaluation metrics, and fine-tuning can improve the performance of the BERT-based methods. CONCLUSIONS: Our comparison study is useful for researchers and practitioners in selecting the best modeling strategies for building article recommendation systems in the biomedical domain. The code and datasets are publicly available.

A high-quality corpus is essential for building an effective legal intelligence system. The quality of a corpus includes both the quality of original data and the quality of its corresponding labeling. The major quality dimensions of a legal corpus include comprehensiveness, freshness, and correctness. However, building a comprehensive, correct, and fresh legal corpus is a grand challenge. In this article, we propose a semiautomated machine learning framework to address the challenge. We first created an initial corpus with 4937 instances that were manually labeled. Several strategies were implemented to assure its quality. The initial results showed that class imbalance and insufficiency of training data are the two major quality issues that negatively impacted the quality of the system that was built on the data. We experimented and compared three class-imbalance-handling techniques and found that the mixed-sampling method, which combines upsampling and downsampling, was the most effective way to address the issue. In order to address the insufficiency of training data, we experimented several machine learning methods for automated data augmentation including pseudolabeling, co-training, expectation-maximization, and generative adversarial network (GAN). The results showed that GAN with deep learning models achieved the best performance. Finally, ensemble learning of different classifiers was proposed and experimented with for the construction of a legal corpus, which achieves higher quality in comprehensiveness, freshness, and correctness compared to existing work. The semiautomated machine learning framework and the data quality evaluation method developed in this research can be used for data augmentation and quality evaluation of a large dataset as well as a reference for the selection of machine learning methods for data augmentation and generation. The machine learning models, the training data, and the legal corpus are published and publicly accessible at [Online]. Available:https://github.com/haihua0913/legalArgumentmining.

Poor data quality has a direct impact on the performance of the machine learning system that is built on the data. As a demonstrated effective approach for data quality improvement, transfer learning has been widely used to improve machine learning quality. However, the “quality improvement” brought by transfer learning was rarely rigorously validated, and some of the quality improvement results were misleading. This article first exposed the hidden quality problem in the datasets used to build a machine learning system for normalizing medical concepts in social media text. The system was claimed to have achieved the best performance compared to existing work on a machine learning task. However, the results of our experiments showed that the “best performance” was due to the poor quality of the datasets and the defective validation process. To address the data quality issue and build a high-performance medical concept normalization system, we developed a transfer-learning-based strategy for data quality enhancement and system performance improvement. The results of the experiments showed a strong correlation between the quality of the datasets and the performance of the machine learning system. The results also demonstrated that a rigorous evaluation of data quality is necessary for guiding the quality improvement of machine learning. Therefore, we propose a data quality evaluation framework that includes the quality criteria and their corresponding evaluation approaches. The data validation process, the performance improvement strategy, and the data quality evaluation framework discussed in this article can be used for machine learning researchers and practitioners to build high-performance machine learning systems.

Knowledge graph has become an essential tool for semantic analysis with the development of natural language processing and deep learning. A high-quality knowledge graph is handy for building a high-performance knowledge-driven application. Despite recent advances in information extraction (IE) techniques, no suitable automated methods can be applied to constructing a domain-specific, comprehensive, and high-quality knowledge graph. However, a semi-automatic strategy, which can ensure the basic quality requirements of a knowledge graph, has been successfully implemented in the elementary science domain. This paper presents a semantic annotation system developed for building a high-quality legal knowledge graph (SALKG) using the semi-automatic strategy. We introduce its system design, architecture, algorithms, functions, and implementation. To investigate the effectiveness of SALKG, we conduct a preliminary annotation experiment with 280 legal texts which were collected from the Harvard Caselaw Access Project. The user evaluation from 32 graduate students demonstrates the high usability of SALKG in semantic annotation and the potential for building a high-quality legal knowledge graph. The system can also be adapted to other fields for constructing domain-specific knowledge graphs.

The poor quality of a dataset may produce low quality machine learning system. Therefore, transfer learning as a demonstrated effective approach for data quality improvement has been widely used for improving the quality of machine learning. However, the "quality improvement" brought by transfer learning in some studies was not rigorously validated or was even misleading. In this paper, we first investigate the quality problem of the datasets that were used for building a machine learning system. The system was claimed to have achieved the best performance comparing to existing work on a machine learning task. However, the "best performance" was due to the poor quality of the datasets as well as the incorrect validation process. Then we described an experimental study to demonstrate the effectiveness of transfer learning for improving the quality of datasets. However, the experiment results also show the quality improvement of transfer learning is not guaranteed, and a set of requirements have to be meet before applying the approach. Based on the investigation and experiment results, we propose a group of data quality criteria and evaluation approaches for quality improvement of machine learning. We investigated the research problem and explained the results through studying a machine learning system for normalizing medical concepts in social media text with open datasets.

Precision medicine information retrieval (PM IR) is about matching the most relevant scientific articles to an individual patient for reliable disease treatment. To achieve effectiveness and efficiency, the task usually consists of two stages: conventional information retrieval and reranking. Many approaches have been proposed for reranking. However, the performance is still far from satisfactory. In this work, we propose a regression-based reranking scheme for PM IR which uses labelled data regardless of empirical knowledge from similar but not identical documents set. Experiments validate that the performance of our approach is significantly better than that of the state-of-the-art approaches.