arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.04053cs.CLcs.AIcs.LG
收藏

使用 OpenAI 的 GPT 评估大型语言模型生成的文本摘要

Evaluating Text Summaries Generated by Large Language Models Using OpenAI's GPT

University of Colorado - Colorado Springs(科罗拉多大学科罗拉多斯普林斯分校);University of Massachusetts Lowell(马萨诸塞大学洛厄尔分校);University of Kansas(堪萨斯大学);DataCrunch Lab, LLC(DataCrunch实验室有限责任公司);University of Florida(佛罗里达大学)

Hassan Shakil, Atqiya Munawara Mahi, Phuoc Nguyen, Zeydy Ortiz, Mamoun T. Mardini

AI总结:

本文将 GPT 作为独立评估者评估六个 transformer 模型生成的摘要,并与 ROUGE、LSA 等传统指标比较,验证其在相关性和连贯性评估上的互补价值。

摘要:

本研究考察了 OpenAI 的 GPT 模型作为独立评估者,评估由 Hugging Face 上六个基于 transformer 的模型(DistilBART、BERT、ProphetNet、T5、BART 和 PEGASUS)生成的文本摘要的有效性。我们基于高质量摘要的关键属性——简洁性、相关性、连贯性和可读性——使用 ROUGE 和潜在语义分析(LSA)等传统指标对这些摘要进行了评估。独特之处在于,我们还将 GPT 用作评估者而非摘要生成器,使其能够在没有预定义指标的情况下独立评估摘要质量。分析显示,GPT 评估与传统指标之间存在显著相关性,尤其是在评估相关性和连贯性方面。结果表明,GPT 有潜力成为评估文本摘要的强大工具,提供补充既有指标的见解,并为自然语言处理任务中基于 transformer 的模型的比较分析奠定基础。

英文摘要:

This research examines the effectiveness of OpenAI's GPT models as independent evaluators of text summaries generated by six transformer-based models from Hugging Face: DistilBART, BERT, ProphetNet, T5, BART, and PEGASUS. We evaluated these summaries based on essential properties of high-quality summary - conciseness, relevance, coherence, and readability - using traditional metrics such as ROUGE and Latent Semantic Analysis (LSA). Uniquely, we also employed GPT not as a summarizer but as an evaluator, allowing it to independently assess summary quality without predefined metrics. Our analysis revealed significant correlations between GPT evaluations and traditional metrics, particularly in assessing relevance and coherence. The results demonstrate GPT's potential as a robust tool for evaluating text summaries, offering insights that complement established metrics and providing a basis for comparative analysis of transformer-based models in natural language processing tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑