arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.15042cs.CLcs.AI
收藏

PROXYQA:一种使用大型语言模型评估长文本生成的替代框架

PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models

Huawei Hisilicon(华为海思);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Hong Kong University of Science and Technology(香港科技大学);Harbin Institute of Technology(哈尔滨工业大学);City University of Hong Kong(香港城市大学);City University of Hong Kong Shenzhen Research Institute(香港城市大学深圳研究院)

Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, Linqi Song

AI总结:

本文提出ProxyQA框架,通过人工策划的元问题和代理问题,利用评估器准确率评估LLMs的长文本生成质量,解决了现有方法效率低且与人类判断不一致的问题。

摘要:

大型语言模型(LLMs)在理解长文本方面取得了显著成功。然而,探索其生成报告和文章等长文本的能力相对较少被研究,且现有基准对其评估不足。主要依赖众包的主流评估方法被认为劳动密集且缺乏效率,而自动指标(如ROUGE分数)则显示出与人类判断标准不一致。本文提出ProxyQA,一种用于评估长文本生成的创新框架。ProxyQA包含涵盖多个领域的人工深度策划元问题,每个元问题配有带有预标注答案的特定代理问题。LLMs被要求针对这些元问题生成大量内容,通过引入评估器并将生成的文本作为上下文背景,ProxyQA通过评估器回答代理问题的准确率来评估生成内容的质量。我们考察了多个LLMs,强调ProxyQA作为高质量评估工具的高要求特性。人类评估表明,代理问题方法具有显著的自洽性,并与人类评估标准高度一致。数据集和排行榜可在\url{https://proxy-qa.com}获取。

英文摘要:

Large Language Models (LLMs) have succeeded remarkably in understanding long-form contents. However, exploring their capability for generating long-form contents, such as reports and articles, has been relatively unexplored and inadequately assessed by existing benchmarks. The prevalent evaluation methods, which predominantly rely on crowdsourcing, are recognized for their labor-intensive nature and lack of efficiency, whereas automated metrics, such as the ROUGE score, demonstrate discordance with human judgment criteria. In this paper, we propose ProxyQA, an innovative framework dedicated to assessing long-text generation. ProxyQA comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers. LLMs are tasked to generate extensive content in response to these meta-questions, by engaging an evaluator and incorporating the generated texts as contextual background, ProxyQA assesses the generated content's quality through the evaluator's accuracy in addressing the proxy-questions. We examine multiple LLMs, emphasizing ProxyQA's demanding nature as a high-quality assessment tool. Human evaluation demonstrates that the proxy-question method is notably self-consistent and aligns closely with human evaluative standards. The dataset and leaderboard is available at \url{https://proxy-qa.com}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑