arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.17867cs.LGcs.CL
收藏

ADAPT:LLM特征可视化中的混合提示优化

ADAPT: Hybrid Prompt Optimization for LLM Feature Visualization

João N. Cardoso, Arlindo L. Oliveira, Bruno Martins

AI总结:

ADAPT通过结合束搜索与自适应梯度突变,改进LLM特征可视化方法,实现更高效的特征激活优化。

摘要:

理解由学习方向在LLM激活空间中编码的特征需要识别强烈激活它们的输入。特征可视化,通过优化输入以最大限度地激活目标方向,为成本高昂的数据集搜索方法提供了替代方案,但因文本的离散性质在LLM中仍处于探索阶段。此外,现有提示优化技术在该领域表现不佳,该领域极易陷入局部极小值。为克服这些限制,我们引入ADAPT,一种结合束搜索初始化与自适应梯度引导突变的混合方法,针对这些失败模式进行设计。我们在Gemma 2 2B的稀疏自编码器潜伏层上进行评估,提出基于数据集激活统计的指标以实现严格比较,并展示ADAPT在各层和潜伏类型上均优于先前方法。我们的结果表明,LLM特征可视化是可行的,但需要针对该领域设计的假设。

英文摘要:

Understanding what features are encoded by learned directions in LLM activation space requires identifying inputs that strongly activate them. Feature visualization, which optimizes inputs to maximally activate a target direction, offers an alternative to costly dataset search approaches, but remains underexplored for LLMs due to the discrete nature of text. Furthermore, existing prompt optimization techniques are poorly suited to this domain, which is highly prone to local minima. To overcome these limitations, we introduce ADAPT, a hybrid method combining beam search initialization with adaptive gradient-guided mutation, designed around these failure modes. We evaluate on Sparse Autoencoder latents from Gemma 2 2B, proposing metrics grounded in dataset activation statistics to enable rigorous comparison, and show that ADAPT consistently outperforms prior methods across layers and latent types. Our results establish that feature visualization for LLMs is tractable, but requires design assumptions tailored to the domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑