LLM 如何压缩自身的思维链？一种 Token 复杂度方法
How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approach
AI总结：
本研究首次系统探讨了LLM推理长度与性能的关系，提出“token复杂度”概念以量化准确率与压缩的信息论极限，发现现有提示压缩策略远未达理论上限，并强调自适应压缩的重要性。
摘要：
思维链提示已成为一种使大型语言模型（LLMs）能够解决复杂推理任务的强大技术。然而，这些推理链可能过于冗长，引发了对效率的担忧。作为回应，近期的研究试图通过简单的提示策略（例如“保持简洁”）来减少响应长度。在本研究中，我们首次系统地研究了在多种压缩指令（例如“使用 10 个词或更少”或“删除所有标点符号”）下，推理长度与模型性能之间的关系。在此过程中，我们发现推理长度与准确率之间存在一种普遍的权衡关系，即使在截然不同的推理链中也持续存在。我们证明，这种权衡源于问题层面的一种急剧阈值行为：每个任务都有一个内在的“token 复杂度”——即成功解决问题所需的最少 token 数量。我们展示了 token 复杂度如何使我们能够计算准确率-压缩权衡的信息论极限，并发现基于提示的压缩策略远未达到这些理论极限。这表明可能存在巨大的改进空间，我们的框架提供了一个基准，以帮助研究人员评估推理效率方面的进展。我们的工作还强调了自适应压缩的重要性——即对较简单的问题给出更短的响应——并且我们表明 token 复杂度是衡量这一能力的有用工具。
英文摘要：
Chain-of-thought prompting has emerged as a powerful technique for enabling large language models (LLMs) to solve complex reasoning tasks. However, these reasoning chains can be verbose, raising concerns about efficiency. In response, recent works have sought to decrease response lengths through simple prompting strategies (e.g. 'be concise'). In this work, we conduct the first systematic study of the relationship between reasoning length and model performance across a diverse range of compression instructions (e.g. 'use 10 words or less' or 'remove all punctuation'). In doing so, we discover a universal tradeoff between reasoning length and accuracy that persists across even very distinct reasoning chains. We demonstrate that this tradeoff emerges from a sharp threshold behavior at the question level: each task has an intrinsic 'token complexity' - a minimal number of tokens required for successful problem-solving. We show how token complexity enables us to compute information-theoretic limits on the accuracy-compression tradeoff, and find that prompt-based compression strategies operate far from these theoretical limits. This suggests there may be significant room for improvement and our framework provides a benchmark to help researchers evaluate progress in reasoning efficiency. Our work also highlights the importance of adaptive compression -- giving shorter responses for easier questions -- and we show that token complexity is a useful tool for measuring this capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
