基于人类相似性判断的转向技术评估
Evaluating Steering Techniques using Human Similarity Judgments
AI总结：
本研究通过人类相似性判断任务评估LLM转向技术，发现基于提示的转向在准确性和人类对齐上最优，并揭示模型对种类相似性的偏向及大小对齐的困难。
摘要：
当前对大型语言模型（LLM）转向技术的评估侧重于任务特定性能，忽视了转向表示与人类认知的契合程度。我们采用一种成熟的三角相似性判断任务，评估了转向后的LLM根据大小或种类灵活判断概念间相似性的能力，这两个维度是组织人类心理表征的核心维度。我们发现，基于提示的转向方法在转向准确性和模型与人类对齐两方面均优于其他方法。我们还发现，LLM偏向于“种类”相似性，而在“大小”对齐上存在困难。这种基于人类认知的评估方法，进一步支持了基于提示的转向的有效性，并揭示了转向前LLM中特权的表征轴。
英文摘要：
Current evaluations of Large Language Model (LLM) steering techniques focus on task-specific performance, overlooking how well steered representations align with human cognition. Using a well-established triadic similarity judgment task, we assessed steered LLMs on their ability to flexibly judge similarity between concepts based on size or kind, two central dimensions organizing human mental representations. We found that prompt-based steering methods outperformed other methods both in terms of steering accuracy and model-to-human alignment. We also found LLMs were biased towards `kind' similarity and struggled with `size' alignment. This evaluation approach, grounded in human cognition, adds further support to the efficacy of prompt-based steering and reveals privileged representational axes in LLMs prior to steering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
