arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.01185cs.CLcs.AI
收藏

利用 Soft RLLF 平衡 LLM 中的探索与利用以增强否定理解

Balancing Exploration and Exploitation in LLM using Soft RLLF for Enhanced Negation Understanding

National Institute of Informatics(信息学研究所)

Ha-Thanh Nguyen, Ken Satoh

AI总结:

提出用 Soft RLLF 在 LLM 微调中平衡探索与利用,并通过基准对比、迁移学习和法律 AI 评估验证其对否定理解与逻辑可靠性的提升。

摘要:

自然语言处理中的微调方法通常侧重于利用而非探索,这可能导致模型达到次优效果。鉴于自然语言庞大的搜索空间,这种有限的探索会限制模型在复杂、高风险领域中的表现,而在这些领域中,准确的否定理解和逻辑推理能力至关重要。为解决这一问题,我们利用来自逻辑反馈的强化学习(Reinforcement Learning from Logical Feedback,RLLF),在大语言模型(LLM)中有效平衡探索与利用。我们的方法采用适当的基准数据集进行训练和评估,突出了探索在增强否定理解能力方面的重要性。我们将经 RLLF 增强的 LLM 与未使用 RLLF 训练的基线模型进行性能比较,证明了这种平衡方法的价值。此外,我们通过采用迁移学习并评估其对否定理解的影响,展示了该方法在法律人工智能应用中的潜力。实验结果表明,利用 RLLF 平衡探索与利用能够有效提升 LLM 的否定理解能力。这对于在高风险领域开发更准确、可靠且逻辑一致的语言模型具有启示意义。

英文摘要:

Finetuning approaches in NLP often focus on exploitation rather than exploration, which may lead to suboptimal models. Given the vast search space of natural language, this limited exploration can restrict their performance in complex, high-stakes domains, where accurate negation understanding and logical reasoning abilities are crucial. To address this issue, we leverage Reinforcement Learning from Logical Feedback (RLLF) to create an effective balance between exploration and exploitation in LLMs. Our approach employs an appropriate benchmark dataset for training and evaluation, highlighting the importance of exploration in enhancing negation understanding capabilities. We compare the performance of our RLLF-enhanced LLMs with baseline models trained without RLLF, demonstrating the value of this balanced approach. Furthermore, we showcase the potential of our method in legal AI applications by employing transfer learning and evaluating its impact on negation understanding. Our experimental results exhibit the effectiveness of balancing exploration and exploitation with RLLF in improving LLMs' negation capabilities. This has implications for the development of more accurate, reliable, and logically consistent language models in high-stakes domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑