CLAWS：利用各部分注意力窗口检测 LLM 生成解答的创造性
CLAWS:Creativity detection for LLM-generated solutions using Attention Window of Sections
AI总结：
本文提出 CLAWS，通过 prompt 各部分与输出间的注意力权重，将 LLM 数学解答自动分为典型、创造性和幻觉三类，并在多个 7-8B 模型与 4545 道竞赛题上验证其优于五种白盒方法。
摘要：
近期在提升大型语言模型（LLM）推理能力方面的进展取得了显著成功。经过强化学习（RL）训练以进行推理的 LLM，即使模型规模相对较小，也在数学和编程等具有挑战性的任务中表现出强大性能。然而，尽管任务准确率有所提升，与写作任务相比，推理任务中对 LLM 生成内容创造性的评估在很大程度上被忽视。推理领域创造性评估研究的缺乏主要源于两个挑战：（1）界定创造性范围的难度；（2）评估过程中必须依赖人工评估。为应对这些挑战，我们提出 CLAWS，该方法通过利用 prompt 各部分与输出之间的注意力权重，在无需人工评估的情况下，将数学解答定义并分类为典型、创造性和幻觉三类。在五个 7-8B 数学 RL 模型（DeepSeek、Qwen、Mathstral、OpenMath2 和 Oreal）上，CLAWS 的表现优于五种现有白盒检测方法（Perplexity、Logit Entropy、Window Entropy、Hidden Score 和 Attention Score）。我们在从 181 场数学竞赛（AJHSME、AMC、AIME）收集的 4545 道数学题上验证了 CLAWS。
英文摘要：
Recent advances in enhancing the reasoning ability of large language models (LLMs) have been remarkably successful. LLMs trained with reinforcement learning (RL) for reasoning demonstrate strong performance in challenging tasks such as mathematics and coding, even with relatively small model sizes. However, despite these improvements in task accuracy, the assessment of creativity in LLM generations has been largely overlooked in reasoning tasks, in contrast to writing tasks. The lack of research on creativity assessment in reasoning primarily stems from two challenges: (1) the difficulty of defining the range of creativity, and (2) the necessity of human evaluation in the assessment process. To address these challenges, we propose CLAWS, a method that defines and classifies mathematical solutions into typical, creative, and hallucinated categories without human evaluation, by leveraging attention weights across prompt sections and output. CLAWS outperforms five existing white-box detection methods (Perplexity, Logit Entropy, Window Entropy, Hidden Score, and Attention Score) on five 7-8B math RL models (DeepSeek, Qwen, Mathstral, OpenMath2, and Oreal). We validate CLAWS on 4545 math problems collected from 181 math contests (AJHSME, AMC, AIME).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
