In NLG meta-evaluation, evaluation metrics are typically assessed based on their consistency with humans.However, we identify some limitations in traditional NLG meta-evaluation approaches, such as issues in handling human ratings and ambiguous selections of correlation measures, which undermine the effectiveness of meta-evaluation.In this work, we propose a dual-perspective NLG meta-evaluation framework that focuses on different evaluation capabilities, thereby providing better interpretability.In addition, we introduce a method of automatically constructing the corresponding benchmarks without requiring new human annotations.Furthermore, we conduct experiments with 16 representative LLMs as the evaluators based on our proposed framework, comprehensively analyzing their evaluation performance from different perspectives.

Large Language Models (LLMs) are increasingly integrated into academic peer review, prompting debates between full automation and purely human evaluation. Emerging evidence suggests optimal peer review leverages both human expertise and AI capabilities, and several major conferences have already adopted AI-assisted reviewing practices. However, effectively integrating these reviewers requires an aspect-based understanding of LLM vulnerabilities, clearly identifying specific dimensions where AI is most prone to error. Prior studies broadly caution against LLM biases but lack precise, aspect-specific insights necessary for informed human-AI partnerships in peer-review processes. We propose an aspect-guided, multi-level perturbation framework to systematically diagnose LLM weaknesses in automated peer review. By introducing targeted perturbations across key review components (papers, reviews, rebuttals) and evaluating impacts along critical quality dimensions (contribution, soundness, presentation, tone, completeness), our framework functions as a diagnostic tool: deviations from expected rating shifts after perturbation directly reveal specific LLM vulnerabilities. Our empirical analyses uncover recurring weaknesses, including misclassification of methodological flaws, disproportionate influence of strong rejection recommendations, inadequate responses to incomplete or negatively toned rebuttals, and misinterpretation of incorrect critiques as rigorous evaluations. These vulnerabilities consistently persist across diverse prompting strategies and a broad set of widely-used LLMs (e.g., GPT-4o, Gemini 2.0, LLaMA 3). This diagnostic framework provides granular insights into LLM limitations, empowering conference organizers to establish pragmatic, aspect-specific guidelines and enabling balanced, informed, and robust peer-review practices.

Mingqi Gao, Xinyu Hu, Li Lin, Xiaojun Wan. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Human evaluation is viewed as a reliable evaluation method for NLG which is expensive and time-consuming. To save labor and costs, researchers usually perform human evaluation on a small subset of data sampled from the whole dataset in practice. However, different selection subsets will lead to different rankings of the systems. To give a more correct inter-system ranking and make the gold standard human evaluation more reliable, we propose a Constrained Active Sampling Framework (CASF) for reliable human judgment. CASF operates through a Learner, a Systematic Sampler and a Constrained Controller to select representative samples for getting a more correct inter-system ranking. Experiment results on 137 real NLG evaluation setups with 44 human evaluation metrics across 16 datasets and 5 NLG tasks demonstrate CASF receives 93.18\% top-ranked system recognition accuracy and ranks first or ranks second on 90.91\% of the human metrics with 0.83 overall inter-system ranking Kendall correlation. Code and data are publicly available online.

Some prior work has shown that LLMs perform well in NLG evaluation for different tasks.However, we discover that LLMs seem to confuse different evaluation criteria, which reduces their reliability.For further verification, we first consider avoiding issues of inconsistent conceptualization and vague expression in existing NLG quality criteria themselves.So we summarize a clear hierarchical classification system for 11 common aspects with corresponding different criteria from previous studies involved.Inspired by behavioral testing, we elaborately design 18 types of aspect-targeted perturbation attacks for fine-grained analysis of the evaluation behaviors of different LLMs.We also conduct human annotations beyond the guidance of the classification system to validate the impact of the perturbations.Our experimental results reveal confusion issues inherent in LLMs, as well as other noteworthy phenomena, and necessitate further research and improvements for LLM-based evaluation. * Equal contribution. Prompt:Your task is to evaluate the summary written for a dialogue on the given criterion.

Research on automated text summarization typically uses human and automatic evaluation methods. While most recent studies focus on intrinsic evaluation, which assesses the general quality of summaries, e.g. coherence and informativeness, we concentrate on task-based extrinsic evaluation to determine the usefulness of summaries. We incorporate three downstream tasks, namely question answering, text classification, and text similarity assessment, and measure the usefulness of summaries for these tasks by several metrics. Our findings reveal that summaries are generally useful in tasks that require a comprehensive grasp of the text but are less useful in tasks requiring a more specific understanding of the text. We also analyze the usefulness and inherent properties of summaries from different models, and find that fine-tuned models consistently produce more useful summaries across all three tasks. In contrast, zero-shot models tend to lean towards text classification and similarity assessment, providing more general and less detailed summaries. Additionally, we assess the correlation between 14 intrinsic automatic metrics and human judgments. Intrinsic metrics perform well in evaluating summaries for question answering but are less effective in the other two tasks. This highlights the limitations of relying solely on intrinsic metrics for assessing summary performance and usefulness.

The evaluation of natural language generation (NLG) tasks is a significant and longstanding research area.With the recent emergence of powerful large language models (LLMs), some studies have turned to LLM-based automatic evaluation methods, which demonstrate great potential to become a new evaluation paradigm following traditional string-based and modelbased metrics.However, despite the improved performance of existing methods, they still possess some deficiencies, such as dependency on references and limited evaluation flexibility.Therefore, in this paper, we meticulously construct a large-scale NLG evaluation corpus NLG-Eval with annotations from both human and GPT-4 to alleviate the lack of relevant data in this field.Furthermore, we propose Themis, an LLM dedicated to NLG evaluation, which has been trained with our designed multi-perspective consistency verification and rating-oriented preference alignment methods.Themis can conduct flexible and interpretable evaluations without references, and it exhibits superior evaluation performance on various NLG tasks, simultaneously generalizing well to unseen tasks and surpassing other evaluation models, including GPT-4.

Factuality is important to dialogue summarization.Factual error correction (FEC) of modelgenerated summaries is one way to improve factuality.Current FEC evaluation that relies on factuality metrics is not reliable and detailed enough.To address this problem, we are the first to manually annotate a FEC dataset for dialogue summarization containing 4000 items and propose FERRANTI, a fine-grained evaluation framework based on reference correction that automatically evaluates the performance of FEC models on different error categories.Using this evaluation framework, we conduct sufficient experiments with FEC approaches under a variety of settings and find the best training modes and significant differences in the performance of the existing approaches on different factual error categories. 1 Corrected (ref) Original Summary Corrected (hypo) Align Align Edits (hypo) Edits (ref) Compare Miley needs some rest and wants to go to work tomorrow.Miley needs some rest and does not want to go to work tomorrow.Miley doesn't want to go to work tomorrow.U "needs some rest and" → "" R "wants" → "doesn't want" R "wants" → "does not want" NegE Pred:VerbE TP: 1 TP: 0 FP: 0 FP: 1 FN: 0 TN: 0 Classify Classify Edits (hypo) Edits (ref) U:Pred:VerbE "needs some rest and" → "" R:Pred:NegE "wants" → "doesn't want"

Dialogue summarization is receiving increasing attention from researchers due to its extraordinary difficulty and unique application value.We observe that current dialogue summarization models have flaws that may not be well exposed by frequently used metrics such as ROUGE.In our paper, we re-evaluate 18 categories of metrics in terms of four dimensions: coherence, consistency, fluency and relevance, as well as a unified human evaluation of various models in dialogue summarization for the first time.Some noteworthy trends which are different from the conventional summarization tasks are identified.We will release DialSummEval, a multi-faceted dataset of human judgments containing the outputs of 14 models on SAMSum.