arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.00244cs.CL
收藏

我们是否设计出了可泛化的结构化知识提示?系统性评估与反思

Have We Designed Generalizable Structural Knowledge Promptings? Systematic Evaluation and Rethinking

Zhejiang University(浙江大学);Ant Group(蚂蚁集团);Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph(浙江大学-蚂蚁集团知识图谱联合实验室)

Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Shaokai Chen, Mengshu Sun, Binbin Hu, Zhiqiang Zhang, Lei Liang, Wen Zhang, Huajun Chen

AI总结:

本文从粒度、可迁移性、可扩展性和普适性四个视角系统评估结构化知识提示(SKP)的泛化能力,并提出多粒度多层次基准SUBARU以揭示其能力边界。

摘要:

大语言模型(LLMs)在当前自然语言处理(NLP)研究中展现出卓越的文本生成性能。然而,事实准确性不足仍然是笼罩在LLM摩天大楼上的一朵乌云。结构化知识提示(Structural Knowledge Prompting,SKP)是一种通过引入结构化表示将外部知识整合到LLM中的重要范式,已在许多知识密集型任务中取得了最先进的结果。然而,现有方法往往只关注特定问题,缺乏对SKP泛化能力和能力边界的全面探索。本文旨在从粒度(Granularity)、可迁移性(Transferability)、可扩展性(Scalability)和普适性(Universality)四个视角评估并反思SKP范式的泛化能力。为了提供全面的评估,我们引入了一个新颖的多粒度、多层次基准,称为SUBARU,该基准包含9个不同粒度和难度水平的任务。

英文摘要:

Large language models (LLMs) have demonstrated exceptional performance in text generation within current NLP research. However, the lack of factual accuracy is still a dark cloud hanging over the LLM skyscraper. Structural knowledge prompting (SKP) is a prominent paradigm to integrate external knowledge into LLMs by incorporating structural representations, achieving state-of-the-art results in many knowledge-intensive tasks. However, existing methods often focus on specific problems, lacking a comprehensive exploration of the generalization and capability boundaries of SKP. This paper aims to evaluate and rethink the generalization capability of the SKP paradigm from four perspectives including Granularity, Transferability, Scalability, and Universality. To provide a thorough evaluation, we introduce a novel multi-granular, multi-level benchmark called SUBARU, consisting of 9 different tasks with varying levels of granularity and difficulty.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑