AnaloBench：面向抽象与长上下文类比识别的基准测试
AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
AI总结：
本研究提出用于测定语言模型类比推理能力的基准AnaloBench，聚焦长场景与大规模信息回忆两类人类常见类比能力，测试发现模型规模扩大在这两类场景中增益极小，为相关研究提供参考。
摘要：
人类经常进行类比思维，将个人经历与当前情境关联起来（X与Y类似，原因是Z）。类比思维让人类能以创造性方式解决问题、理解难的概念，以及更有效地表达想法。语言模型（LMs）能做到同样的事吗？为回答这个问题，我们提出AnaloBench，一个用于测定语言模型类比推理能力的基准。我们的基准测试方法聚焦于人类具备的两类常见类比能力：（i）从大量信息中回忆相关经历，以及（ii）将类比推理应用于复杂冗长的场景。我们测试了广泛的专有模型（例如GPT系列、Claude V2）以及LLaMA2这类开源模型。和既往结果一致，扩大语言模型规模能带来一定性能提升。但令人意外的是，在以下两种情况中，规模扩大带来的增益极小：（i）类比涉及冗长场景，或（ii）需要从大量信息池中回忆相关场景——这一过程类似于大海捞针。我们希望这些观察能推动该领域的进一步研究。
英文摘要：
Humans regularly engage in analogical thinking, relating personal experiences to current situations (X is analogous to Y because of Z). Analogical thinking allows humans to solve problems in creative ways, grasp difficult concepts, and articulate ideas more effectively. Can language models (LMs) do the same? To answer this question, we propose AnaloBench, a benchmark to determine analogical reasoning ability in LMs. Our benchmarking approach focuses on aspects of this ability that are common among humans: (i) recalling related experiences from a large amount of information, and (ii) applying analogical reasoning to complex and lengthy scenarios. We test a broad collection of proprietary models (e.g., GPT family, Claude V2) and open source models such as LLaMA2. As in prior results, scaling up LMs results in some performance boosts. Surprisingly, scale offers minimal gains when, (i) analogies involve lengthy scenarios, or (ii) recalling relevant scenarios from a large pool of information, a process analogous to finding a needle in a haystack. We hope these observations encourage further research in this field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
