Existing LLM-as-a-Judge approaches for evaluating text generation suffer from rating inconsistencies, with low agreement and high rating variance across different evaluator models.We attribute this to subjective evaluation criteria combined with Likert scale scoring in existing protocols.To address this issue, we introduce CheckEval, a checklist-based evaluation framework that improves rating reliability via decomposed binary questions.Through experiments with 12 evaluator models across multiple datasets, we first demonstrate that CheckEval strongly correlates with human judgments.More importantly, CheckEval dramatically improves the average agreement across evaluator models by 0.45 and reduces the score variance.CheckEval scores furthermore have the benefit of being more interpretable because it decomposes evaluation criteria into traceable binary decisions, allowing analyses of specific attributes driving quality judgments.

Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, Nako Sung. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Approximate message-passing (AMP) method is a simple and efficient framework for the linear inverse problems. In this letter, we propose a faster AMP to solve the L1-Split-Analysis for the 2-D sparsity separation, which is referred to as MixAMP. We develop the MixAMP based on the factor graphical modeling and the min-sum message-passing. Then, we examine MixAMP for two types of the sparsity separation: separation of the direct-and-group sparsity, and that of the direct-and-finite-difference sparsity. This case study shows that the MixAMP method offers computational advantages over the conventional first-order method, TFOCS.