采纳第二意见是有益的：通过选择性理由优化教导较小的LLM相互审议
It Helps to Take a Second Opinion: Teaching Smaller LLMs to Deliberate Mutually via Selective Rationale Optimisation
AI总结：
提出COALITION框架，通过同一SLM两个变体的交叉通信与选择性理由优化（SRO）相互审议生成并完善理由，在五项数据集上性能提升达5%，适用于4B-14B多种模型。
摘要：
诸如GPT-4等超大规模语言模型（LLM）已展现出通过生成和自我完善逐步理由来处理复杂任务的能力。较小语言模型（SLM，通常参数量<13B）通过使用超大规模LM生成的数据进行知识蒸馏而得到改进。然而，API成本、版权、法律和伦理政策等各种实际限制，制约了使用大型（通常是不透明的）模型来训练供商业使用的较小模型。在提升SLM通过自我审议来探索可能理由空间并自行评估它们的能力方面，取得的成功有限。为了解决这个问题，我们提出了COALITION，这是一个可训练的框架，它促进同一SLM的两个变体之间的交互，并训练它们生成和优化针对最终任务的完善理由。这两个变体在生成和完善步骤中表现出不同的行为，从而产生一组多样化的候选理由。然后通过选择性理由优化（SRO）训练模型，使其倾向于生成能够最大化产生真实答案可能性的候选理由。在推理阶段，COALITION采用一个控制器来选择合适的变体用于生成和完善理由。在涵盖数学问题、常识推理和自然语言推理的五个不同数据集上，COALITION的性能比几个基线高出多达5%。我们的消融研究表明，两个变体之间的交叉通信比使用单一模型自我完善理由表现更好。我们还展示了COALITION在不同规模（4B到14B参数）和模型家族（Mistral、Llama、Qwen、Phi）的LM上的适用性。我们在https://github.com/Sohanpatnaik106/coalition发布了本工作的代码。
英文摘要：
Very large language models (LLMs) such as GPT-4 have shown the ability to handle complex tasks by generating and self-refining step-by-step rationales. Smaller language models (SLMs), typically with < 13B parameters, have been improved by using the data generated from very-large LMs through knowledge distillation. However, various practical constraints such as API costs, copyright, legal and ethical policies restrict using large (often opaque) models to train smaller models for commercial use. Limited success has been achieved at improving the ability of an SLM to explore the space of possible rationales and evaluate them by itself through self-deliberation. To address this, we propose COALITION, a trainable framework that facilitates interaction between two variants of the same SLM and trains them to generate and refine rationales optimized for the end-task. The variants exhibit different behaviors to produce a set of diverse candidate rationales during the generation and refinement steps. The model is then trained via Selective Rationale Optimization (SRO) to prefer generating rationale candidates that maximize the likelihood of producing the ground-truth answer. During inference, COALITION employs a controller to select the suitable variant for generating and refining the rationales. On five different datasets covering mathematical problems, commonsense reasoning, and natural language inference, COALITION outperforms several baselines by up to 5%. Our ablation studies reveal that cross-communication between the two variants performs better than using the single model to self-refine the rationales. We also demonstrate the applicability of COALITION for LMs of varying scales (4B to 14B parameters) and model families (Mistral, Llama, Qwen, Phi). We release the code for this work at https://github.com/Sohanpatnaik106/coalition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
