量化大型语言模型的泛化复杂度
Quantifying Generalization Complexity for Large Language Models
AI总结:
提出 Scylla 动态评估框架,通过分布内与分布外任务解耦泛化和记忆,发现“泛化谷”与临界复杂度现象,并基于 28 个 LLMs 的基准测试揭示模型规模与泛化能力上界的关系。
摘要:
虽然大型语言模型(LLMs)在理解复杂查询和执行复杂任务方面表现出卓越能力,但其泛化能力往往与记忆深度交织,因此需要更精确的评估。为应对这一挑战,我们提出 Scylla,一个动态评估框架,用于定量测量 LLMs 的泛化能力。Scylla 通过在 5 个复杂度级别、共 20 个任务上评估模型在分布内(ID)和分布外(OOD)数据上的表现,将泛化与记忆解耦。通过大量实验,我们发现任务复杂度与 ID 和 OOD 数据之间的性能差距之间存在非单调关系,并将其称为泛化谷。具体而言,这一现象揭示了一个关键阈值——称为临界复杂度——在该阈值处对不可泛化行为的依赖达到峰值,表明 LLMs 泛化能力的上界。随着模型规模增大,临界复杂度向更高任务复杂度水平移动,表明更大的模型在过度依赖记忆之前能够处理更复杂的推理任务。利用 Scylla 和临界复杂度概念,我们对 28 个 LLMs 进行基准测试,包括 LLaMA 和 Qwen 系列等开源模型,以及 Claude 和 GPT 等闭源模型,从而提供更稳健的评估,并建立对 LLMs 泛化能力更清晰的理解。
英文摘要:
While large language models (LLMs) have shown exceptional capabilities in understanding complex queries and performing sophisticated tasks, their generalization abilities are often deeply entangled with memorization, necessitating more precise evaluation. To address this challenge, we introduce Scylla, a dynamic evaluation framework that quantitatively measures the generalization abilities of LLMs. Scylla disentangles generalization from memorization via assessing model performance on both in-distribution (ID) and out-of-distribution (OOD) data through 20 tasks across 5 levels of complexity. Through extensive experiments, we uncover a non-monotonic relationship between task complexity and the performance gap between ID and OOD data, which we term the generalization valley. Specifically, this phenomenon reveals a critical threshold - referred to as critical complexity - where reliance on non-generalizable behavior peaks, indicating the upper bound of LLMs' generalization capabilities. As model size increases, the critical complexity shifts toward higher levels of task complexity, suggesting that larger models can handle more complex reasoning tasks before over-relying on memorization. Leveraging Scylla and the concept of critical complexity, we benchmark 28LLMs including both open-sourced models such as LLaMA and Qwen families, and close-sourced models like Claude and GPT, providing a more robust evaluation and establishing a clearer understanding of LLMs' generalization capabilities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。