arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.09174cs.CLcs.AI
收藏

AbsPyramid:用统一蕴含图基准评估语言模型的抽象能力

AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph

Tencent AI Lab(腾讯AI实验室)

Zhaowei Wang, Haochen Shi, Weiqi Wang, Tianqing Fang, Hongming Zhang, Sehyun Choi, Xin Liu, Yangqiu Song

AI总结:

本文提出包含221K抽象知识的统一蕴含图AbsPyramid,用于全面评估并增强大语言模型在开放领域中的抽象能力,实验表明训练后可获得基本抽象能力并泛化到未见事件。

摘要:

认知研究表明,抽象能力是人类智能中至关重要的部分,而语言模型在这方面的探索仍不充分。本文提出了 AbsPyramid,一个包含 221K 条抽象知识文本描述的统一蕴含图。现有资源仅涉及简化事件或特定领域中的名词或动词,而 AbsPyramid 收集了多样事件三个组成部分的抽象知识,以全面评估语言模型在开放领域中的抽象能力。实验结果表明,当前大语言模型在零样本和少样本设置下理解抽象知识仍面临挑战。通过在丰富的抽象知识上进行训练,我们发现大语言模型能够获得基本的抽象能力,并泛化到未见事件。同时,我们通过实验证明,该基准具有全面性,能够在两个先前的抽象任务上提升大语言模型的表现。

英文摘要:

Cognitive research indicates that abstraction ability is essential in human intelligence, which remains under-explored in language models. In this paper, we present AbsPyramid, a unified entailment graph of 221K textual descriptions of abstraction knowledge. While existing resources only touch nouns or verbs within simplified events or specific domains, AbsPyramid collects abstract knowledge for three components of diverse events to comprehensively evaluate the abstraction ability of language models in the open domain. Experimental results demonstrate that current LLMs face challenges comprehending abstraction knowledge in zero-shot and few-shot settings. By training on our rich abstraction knowledge, we find LLMs can acquire basic abstraction abilities and generalize to unseen events. In the meantime, we empirically show that our benchmark is comprehensive to enhance LLMs across two previous abstraction tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑