评估GRPO训练的NLI模型的可扩展性与对抗性泛化能力
Evaluating the Scalability and Adversarial Generalization of GRPO-Trained NLI Models
AI总结：
本研究利用GRPO强化学习对7B至32B语言模型进行NLI思维链训练，无需人工标注理由，在对抗性基准上表现优于监督基线，且通过量化适配22GB内存，提供了可扩展的稳健NLI框架。
摘要：
自然语言推理（NLI）是自然语言理解中的核心任务，在事实核查、问答和信息检索中均有应用。尽管其重要性不言而喻，当前的NLI系统严重依赖监督学习，而所使用的数据集往往包含标注伪影和偏差，限制了泛化能力和实际应用性。在本工作中，我们采用基于强化学习的方法，将组相对策略优化（GRPO）应用于NLI中的思维链（CoT）学习，从而无需人工标注的理由，并使得此类训练能够在ANLI等具有挑战性的数据集上进行。我们使用参数高效技术（LoRA和QLoRA）微调了7B、14B和32B的语言模型，在标准及对抗性NLI基准上展示了强劲的性能。在32B规模下，GRPO训练的模型在对抗性数据集上的泛化能力优于其他监督基线。借助AWQ量化，32B模型可适配于22GB的CUDA内存。本工作为构建稳健的NLI系统提供了一个可扩展且实用的框架，且不牺牲推理质量。
英文摘要：
Natural Language Inference (NLI) is a central task in natural language understanding with applications in fact-checking, question answering, and information retrieval. Despite its importance, current NLI systems heavily rely on supervised learning with datasets that often contain annotation artifacts and biases, limiting generalization and real-world applicability. In this work, we apply a reinforcement learning-based approach using Group Relative Policy Optimization (GRPO) for Chain-of-Thought (CoT) learning in NLI, eliminating the need for human-labeled rationales and enabling this type of training on challenging datasets such as ANLI. We fine-tune 7B, 14B, and 32B language models using parameter-efficient techniques (LoRA and QLoRA), demonstrating strong performance across standard and adversarial NLI benchmarks. At the 32B scale, GRPO-trained models generalize better than other supervised baselines in adversarial sets. With AWQ quantization, the 32B model fits within 22GB of CUDA memory. This work provides a scalable and practical framework for building robust NLI systems without sacrificing inference quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
