多跳推理的亲属数据基准
Kinship Data Benchmark for Multi-hop Reasoning
AI总结：
KinshipQA通过生成文化特定的家谱数据，评估多跳推理能力，揭示模型在不同文化背景下的推理差异。
摘要：
大型语言模型（LLMs）越来越多地被评估其进行多跳推理的能力，即结合多条信息进行连贯推理。我们引入了KinshipQA基准，通过推理亲属关系来探测这种能力。我们的核心贡献是一个生成管道，能够按需生成大规模、真实且文化特定的家谱数据：互连的家庭树集合，满足与不同亲属系统相关的显式婚姻约束。这允许任务难度、文化假设和关系深度系统地控制和变化。从这些家谱中，我们推导出需要推理隐含关系链的文本推理任务。我们使用六个最先进的LLMs评估了所得到的基准，涵盖开源和闭源模型，在统一的零样本协议下进行确定性解码。性能通过精确匹配和集合基度量进行测量。我们的结果表明，KinshipQA产生了一种广泛的结果分布，并揭示了模型和文化设置之间多跳推理的系统性差异。
英文摘要：
Multi-hop kinship reasoning is a natural testbed for LLM compositionality, but existing benchmarks (notably CLUTRR) cover only the descriptive Eskimo system. We introduce KinshipQA, a procedurally-generated benchmark covering seven anthropologically-documented kinship systems (Eskimo, Sudanese, Hawaiian, Iroquois, Dravidian, Crow, Omaha) and up to six reasoning hops, with a tunable simulator horizon that eliminates exact-instance pretraining overlap. Evaluating six LLMs, we find a 40.9% accuracy drop when reasoning shifts from biological multi-hop to culturally-marked classification on the five non-descriptive systems. The drop holds for every non-descriptive system and is largest for the two skewing systems (Crow, Omaha), persists under chain-of-thought and few-shot prompting, and compounds with depth: at 5--6 hops cultural override falls to 10.6% while biological composition over the same chains remains at 58.6%. Under identical rule access humans reach 89.0% versus 50.7% for LLMs, so the questions are reliably solvable once the rule is supplied. Two follow-up experiments suggest distinct contributors. A fictional-rule control swapping system labels and kin terms for invented strings raises accuracy by 6.1%, implicating familiar English surface forms. An in-context-rule probe prepending the override rule helps skewing systems (+17.1%) but hurts non-skewing systems whose baseline already exceeds about 60% (-13.4%), consistent with a missing skewing prior alongside rule interference where the model already has a working approximation. Our code and data are publicly available on GitHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
