arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.23629cs.LG
收藏

通过激活方向引导代码LLMs实现语言与库的控制

Steering Code LLMs with Activation Directions for Language and Library Control

Iowa State University(爱荷华州立大学);Northeastern University(东北大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室)

Md Mahbubur Rahman, Arjun Guha, Harshitha Menon

AI总结:

研究通过激活空间中的线性方向调整,提升代码LLMs在中性提示下的生成倾向性,发现不同模型和目标生态系统的引导效果存在差异。

摘要:

代码LLMs在中性提示下常倾向于特定编程语言和库。本文探讨这些偏好是否以激活空间中的近似线性方向编码,可通过推理时调整实现控制。使用差异均值法,我们估计了五种语言/库配对的层间引导向量,并在生成过程中将其添加至模型隐藏状态。在三个开源代码LLMs上,这些干预显著提高了中性提示下的生成倾向性,即使在提示明确要求相反选择时仍有效。引导强度因模型和目标而异,常见生态系统更易诱导,过度强的干预可能降低输出质量。总体而言,研究结果表明LLMs中的代码风格偏好部分由激活空间中的紧凑、可调节结构表示。

英文摘要:

Code LLMs often default to particular programming languages and libraries under neutral prompts. We investigate whether these preferences are encoded as approximately linear directions in activation space that can be manipulated at inference time. Using a difference-in-means method, we estimate layer-wise steering vectors for five language/library pairs and add them to model hidden states during generation. Across three open-weight code LLMs, these interventions substantially increase generation toward the target ecosystem under neutral prompts and often remain effective even when prompts explicitly request the opposite choice. Steering strength varies by model and target, with common ecosystems easier to induce than rarer alternatives, and overly strong interventions can reduce output quality. Overall, our results suggest that code-style preferences in LLMs are partly represented by compact, steerable structure in activation space.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑