arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.14712cs.CLcs.AI
收藏

评估用于图到文本生成的生成式模型

Evaluating Generative Models for Graph-to-Text Generation

Karlsruhe Institute of Technology (KIT)(卡尔斯鲁厄理工学院(KIT))

Shuzhou Yuan, Michael Färber

AI总结:

本文探讨了GPT-3和ChatGPT在零样本图到文本生成中的能力,发现其虽能生成流畅文本,但在理解语义关系和避免幻觉方面仍有不足。

摘要:

大型语言模型(LLMs)已被广泛应用于图到文本生成任务。然而,微调LLMs的过程需要大量的训练资源和标注工作。在本文中,我们探讨了生成式模型在零样本设置下从图数据生成描述性文本的能力。具体而言,我们在两个图到文本数据集上评估了GPT-3和ChatGPT,并将其性能与T5和BART等微调LLM模型进行了比较。结果表明,生成式模型能够生成流畅连贯的文本,在AGENDA和WebNLG数据集上分别取得了10.57和11.08的BLEU分数。然而,错误分析显示,生成式模型在理解实体间的语义关系方面仍有困难,且倾向于生成带有幻觉或无关信息的文本。作为错误分析的一部分,我们利用BERT检测机器生成的文本,并取得了较高的macro-F1分数。我们已将生成式模型生成的文本公开发布。

英文摘要:

Large language models (LLMs) have been widely employed for graph-to-text generation tasks. However, the process of finetuning LLMs requires significant training resources and annotation work. In this paper, we explore the capability of generative models to generate descriptive text from graph data in a zero-shot setting. Specifically, we evaluate GPT-3 and ChatGPT on two graph-to-text datasets and compare their performance with that of finetuned LLM models such as T5 and BART. Our results demonstrate that generative models are capable of generating fluent and coherent text, achieving BLEU scores of 10.57 and 11.08 for the AGENDA and WebNLG datasets, respectively. However, our error analysis reveals that generative models still struggle with understanding the semantic relations between entities, and they also tend to generate text with hallucinations or irrelevant information. As a part of error analysis, we utilize BERT to detect machine-generated text and achieve high macro-F1 scores. We have made the text generated by generative models publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑