自我提升往往是突然的：大规模模型的顿悟式微调

中文摘要 AI 辅助

在大规模基础模型时代，追求自主自我提升的模型引发了越来越多的关注。从人类大脑中“顿悟”或“灵光一闪”的概念中获得灵感，我们假设大型模型表现出类似的顿悟现象——一种突然提升能力的潜在能力。然后，我们提出了顿悟（Enlightenment），一种用于大规模模型的新型无训练后微调范式。我们的方法在不更新权重的情况下修改关键模块/层的捷径，而现有的无训练方法主要操纵注意力权重。我们引入了两种特定于架构的实例化：i）对于大语言模型，我们提出了注意力头混合捷径，通过将初始注意力头的输出链接到所有其他目标头，并由自适应缩放因子初始化策略进行调制，来重新校准注意力权重。ii）对于视觉语言模型，我们在解码器层的残差连接中应用轻量级标量调制因子，调节信息流。大量实验表明，顿悟有效地解锁了预训练网络的潜在潜力，在各种基准和模型上产生了显著的性能提升。

英文摘要

The pursuit of autonomously self-improving models has attracted growing interest in the era of large-scale foundation models. Drawing inspiration from the concept of "enlightenment" or "aha moment" in human brain, we hypothesize that large models exhibit an analogous enlightenment phenomenon-a latent capacity for sudden capability boost. Then, we propose Enlightenment, a novel training-free post-tuning paradigm for large-scale models. Our approach modifies shortcuts for key modules/layers without weight updates, while existing training-free ones predominantly manipulate attention weights. We introduce two architecture-specific instantiations: i) For large language models, we propose attention head-mixing shortcuts that recalibrate attention weights by linking the initial attention head's output to all other target heads, modulated by an adaptive scaling factor initialization strategy. ii) For vision-language models, we apply a lightweight scalar-modulated factor to residual connections in the decoder layers, regulating information flow. Extensive experiments show that Enlightenment efficiently unlocks the latent potential of pre-trained networks, yielding remarkable performance improvements across diverse benchmarks and models.

发表机构

• School of Computer Science and Artificial Intelligence, Guangdong University of Education（广东第二师范学院计算机科学与人工智能学院）
• School of Instrumentation Science and Engineering, Harbin Institute of Technology（哈尔滨工业大学仪器科学与工程学院）

机构由 AI 辅助整理，请以论文原文为准。