驾驭语义关系：语言模型在抽象常识推理中的挑战
Navigating Semantic Relations: Challenges for Language Models in Abstract Common-Sense Reasoning
AI总结：
该研究针对大语言模型抽象常识推理能力不足的问题，提出指令提示和少样本提示两种方法，基于ConceptNet的实验揭示了商用模型的性能差距与偏差，验证了选择性检索提示工程的优化潜力。
摘要：
大语言模型（LLM）在生成类人文本及解决中等复杂度推理任务（如问答和数学解题）方面已取得显著成效。然而，它们在需要更深层认知技能的任务（如常识理解和抽象推理）中的能力仍未得到充分探究。本文中，我们利用ConceptNet知识图谱系统评估了大语言模型的抽象常识推理能力。我们提出两种提示方法：指令提示（instruct prompting），即模型基于给定定义预测合理的语义关系；以及少样本提示（few-shot prompting），即模型以示例为指导识别关系。我们使用gpt-4o-mini模型的实验表明，在指令提示下，模型对多个关系排序时性能稳定，但当被限制仅预测一个关系时性能大幅下降。在少样本提示中，模型从5种关系（而非完整关系集合）中选择时准确率显著提升，不过对特定关系存在明显偏差。这些结果表明，即便商用大语言模型的抽象常识推理能力与人类水平的理解相比仍存在显著差距。但研究结果也凸显了基于选择性检索的精心提示工程有望实现更优性能。
英文摘要：
Large language models (LLMs) have achieved remarkable performance in generating human-like text and solving reasoning tasks of moderate complexity, such as question-answering and mathematical problem-solving. However, their capabilities in tasks requiring deeper cognitive skills, such as common-sense understanding and abstract reasoning, remain under-explored. In this paper, we systematically evaluate abstract common-sense reasoning in LLMs using the ConceptNet knowledge graph. We propose two prompting approaches: instruct prompting, where models predict plausible semantic relationships based on provided definitions, and few-shot prompting, where models identify relations using examples as guidance. Our experiments with the gpt-4o-mini model show that in instruct prompting, consistent performance is obtained when ranking multiple relations but with substantial decline when the model is restricted to predicting only one relation. In few-shot prompting, the model's accuracy improves significantly when selecting from five relations rather than the full set, although with notable bias toward certain relations. These results suggest significant gaps still, even in commercially used LLMs' abstract common-sense reasoning abilities, compared to human-level understanding. However, the findings also highlight the promise of careful prompt engineering, based on selective retrieval, for obtaining better performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
