StreamAdapter:从上下文流中实现高效的测试时适应
StreamAdapter: Efficient Test Time Adaptation from Contextual Streams
AI总结:
StreamAdapter 是一种在测试时直接从上下文更新模型参数的新方法,通过上下文映射和权重吸收机制将 ICL 示例转化为参数更新,显著降低推理成本并实现恒定时间复杂度的高效推理。
摘要:
上下文学习(ICL)允许大型语言模型(LLMs)直接从给定的示例中适应新任务,而无需梯度更新。尽管近期的进展扩大了上下文窗口以容纳更多示例,但这种方法增加了推理成本,且未必能提升性能。为了缓解这些问题,我们提出了 StreamAdapter,这是一种在测试时直接从上下文更新模型参数的新颖方法,消除了对显式上下文示例的需求。StreamAdapter 采用了上下文映射和权重吸收机制,以最少的额外参数将 ICL 示例动态转化为参数更新。通过减少对大量上下文示例的依赖,StreamAdapter 显著降低了推理成本,并允许以恒定的时间复杂度进行高效推理,而不受示例数量的影响。在多种任务和模型架构上的广泛实验表明,StreamAdapter 实现了与 ICL 相当或更优的适应能力,同时所需的示例数量显著减少。StreamAdapter 在语言理解和生成任务上的卓越任务适应和上下文编码能力,为在测试时利用上下文适应 LLMs 提供了新视角,从而实现了跨场景的更高效适应和更具成本效益的推理。
英文摘要:
In-context learning (ICL) allows large language models (LLMs) to adapt to new tasks directly from the given demonstrations without requiring gradient updates. While recent advances have expanded context windows to accommodate more demonstrations, this approach increases inference costs without necessarily improving performance. To mitigate these issues, We propose StreamAdapter, a novel approach that directly updates model parameters from context at test time, eliminating the need for explicit in-context demonstrations. StreamAdapter employs context mapping and weight absorption mechanisms to dynamically transform ICL demonstrations into parameter updates with minimal additional parameters. By reducing reliance on numerous in-context examples, StreamAdapter significantly reduce inference costs and allows for efficient inference with constant time complexity, regardless of demonstration count. Extensive experiments across diverse tasks and model architectures demonstrate that StreamAdapter achieves comparable or superior adaptation capability to ICL while requiring significantly fewer demonstrations. The superior task adaptation and context encoding capabilities of StreamAdapter on both language understanding and generation tasks provides a new perspective for adapting LLMs at test time using context, allowing for more efficient adaptation across scenarios and more cost-effective inference
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。