Rubric-augmented verification guides reward models with explicit evaluation criteria, yielding more reliable judgments than single-model verification.However, most existing methods require costly rubric annotations, limiting scalability.Moreover, we find that rubric generation is vulnerable to a failure of cooperation; lowquality rubrics actively mislead reward models rather than help.Inspired by the principle of cooperative communication, we propose Cooperative yet Critical reward modeling (C2), a framework that significantly improves reward model judgments by having the reward model critically collaborate with a rubric generator trained solely from binary preferences.In C2, we synthesize helpful and misleading rubric pairs by measuring how each rubric shifts the reward model toward or away from the correct preference.Using these contrastive pairs, we train a cooperative rubric generator to propose helpful rubrics, and a critical verifier to assess rubric validity before making its judgment, following only rubrics it deems helpful at inference time.C2 outperforms reasoning reward models trained on the same binary preferences, with gains of up to 6.5 points on RM-Bench and 6.0 points length-controlled win rate on Al-pacaEval 2.0.Without external rubric annotations, C2 enables an 8B reward model to match performance achieved with rubrics from a 4× larger model.Overall, our work demonstrates that eliciting deliberate cooperation in rubricaugmented verification makes reward models more trustworthy in a scalable way. 1

Answer verification identifies correct solutions among candidates generated by large language models (LLMs).Current approaches typically train verifier models by labeling solutions as correct or incorrect based solely on whether the final answer matches the gold answer.However, this approach neglects any flawed rationale in the solution yielding the correct answer, undermining the verifier's ability to distinguish between sound and flawed rationales.We empirically show that in StrategyQA, only 19% of LLM-generated solutions with correct answers have valid rationales.Furthermore, we demonstrate that training a verifier on valid rationales significantly improves its ability to distinguish valid and flawed rationales.To make a better verifier without extra human supervision, we introduce REPS (Rationale Enhancement through Pairwise Selection), a method for selecting valid rationales from candidates by iteratively applying pairwise self-evaluation using the same LLM that generates the solutions.Verifiers trained on solutions selected by REPS outperform those trained using conventional training methods on three reasoning benchmarks (ARC-Challenge, DROP, and StrategyQA).Our results suggest that training reliable verifiers requires ensuring the validity of rationales in addition to the correctness of the final answers, which would be critical for models assisting humans in solving complex reasoning tasks.

To precisely evaluate a language model's capability for logical reading comprehension, we present a dataset for testing the understanding of the rationale behind critical reasoning.For questions taken from an existing multiplechoice logical reading comprehension dataset, we crowdsource rationale texts that explain why we should select or eliminate answer options, resulting in 3,003 multiple-choice subquestions that are associated with 943 main questions.Experiments on our dataset show that recent large language models (e.g., InstructGPT) struggle to answer the subquestions even if they are able to answer the main questions correctly.We find that the models perform particularly poorly in answering subquestions written for the incorrect options of the main questions, implying that the models have a limited capability for explaining why incorrect alternatives should be eliminated.These results suggest that our dataset encourages further investigation into the critical reasoning ability of language models while focusing on the elimination process of relevant alternatives.In a given context, you'll be given a question, an answer, and four rationales.Your task is to identify the rationale that explains the correctness of the provided option the best.If the option is wrong, choose the rationale that explains why it is wrong.Conversely, if the option is correct, choose the rationale that explains why it is correct.Context: Teachers should not do anything to cause their students to lose respect for them.And students can sense when someone is trying to hide his or her ignorance.Therefore, a teacher who does not know the answer to a question a student has asked should not pretend to know the answer.Question: The conclusion is properly drawn if which one of the following is assumed?Question: The conclusion is properly drawn if which one of the following is assumed?Option: Students' respect for a teacher is independent of the amount of knowledge they attribute to that teacher.Rationale0: The ranking of students' respect for honesty is not relevant to the conclusion of a teacher shouldn't pretend to know an answer to question they don't know the answer to. Rationale1:The assumption is that students' respect for the teacher is based on how much knowledge the teacher has.Rationale2: The conclusion is that teachers shouldn't pretend to know the answer to a question that they don't know, so the assumption is that student's respect for a teacher is interlinked to the student's perceived knowledge of the teacher.Rationale3: The conclusion does not have anything to do with a teacher being effective.Answer: The answer is Rationale2Context: Miguel has four family members who plan to come to his graduation on Sunday afternoon, but it is likely that only three of them will be allowed to attend.Normally graduation is held in the football stadium, where there is no limit on the number of family members who can attend.However, the ceremony is relocated to the gymnasium if it rains, and each graduate receives just three admission tickets for use by family members.Question: The conclusion of the argument is most strongly supported if which one of the following is assumed?Option: The weather service has indicated that there is a very high likelihood of rain on Sunday afternoon.Rationale0: No mention is made of whether un-needed spaces can be transferred between students, and so this cannot be assumed to impact the number of spaces available to Miguel's family.Rationale1: Abnormally large class size may not preclude Miguel from having more than three family members attend, as the football stadium is a possible venue and has no limitation on the number who may attend.Rationale2: A family member who cannot attend the graduation has no relevance to how many may be allowed to attend.Rationale3: Rain would preclude the use of the stadium which has no limit of the number of family members attending and force the use of the gymnasium, which limits the number attending to three.