在舌尖上：用反向词典探针分析大型语言模型中的概念表征
On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe
AI总结：
本文利用反向词典任务探测LLM的概念推理能力，发现该能力可预测模型在多个基准上的通用推理表现，并表明描述到词语的提示有助于促进更广泛的常识推理。
摘要：
探究和增强大型语言模型的推理能力仍然是一个至关重要的开放问题。在此，我们重新利用反向词典任务作为案例研究，以探测LLM进行概念推理的能力。我们使用上下文学习来引导模型生成语言描述中所隐含的对象概念的术语。模型在该任务中稳健地取得了高准确率，并且它们的表征空间编码了关于对象类别和细粒度特征的信息。进一步的实验表明，通过反向词典任务所探测的概念推理能力能够预测模型在多个基准测试中的一般推理表现，尽管不同模型在句法泛化行为上表现相似。探索性分析表明，用“描述$\Rightarrow$词语”的示例来提示LLM可能会诱导超越任务理解中表面差异的泛化，并促进模型在更广泛的常识推理问题上的表现。
英文摘要：
Probing and enhancing large language models' reasoning capacity remains a crucial open question. Here we re-purpose the reverse dictionary task as a case study to probe LLMs' capacity for conceptual inference. We use in-context learning to guide the models to generate the term for an object concept implied in a linguistic description. Models robustly achieve high accuracy in this task, and their representation space encodes information about object categories and fine-grained features. Further experiments suggest that the conceptual inference ability as probed by the reverse-dictionary task predicts model's general reasoning performance across multiple benchmarks, despite similar syntactic generalization behaviors across models. Explorative analyses suggest that prompting LLMs with description$\Rightarrow$word examples may induce generalization beyond surface-level differences in task construals and facilitate models on broader commonsense reasoning problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
