arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.06883cs.CLcs.AIcs.LG
收藏

一个用于评估研究问题提取任务的基于大语言模型的评估函数的数据集

A Dataset for Evaluating LLM-based Evaluation Functions for Research Question Extraction Task

Japan Advanced Institute of Science and Technology(日本先进科学技术学院);National Institute of Advanced Industrial Science and Technology(日本国立先进工业科学技术研究所);OMRON SINIC X Corporation(欧姆龙SINIC X公司)

Yuya Fujisaki, Shiro Takagi, Hideki Asoh, Wataru Kumagai

AI总结:

本研究针对专业文档研究问题提取任务的评估缺口,构建了含机器学习论文、GPT-4提取的RQ及多维度人工标注的新数据集,测试发现现有LLM评估函数与人工评估相关性不足,为该任务的评估函数研发提供基础。

摘要:

文本摘要技术已取得显著进展,但从研究论文这类高度专业化的文档中准确提取并总结必要信息的任务尚未得到充分研究。我们聚焦于从研究论文中提取研究问题(RQ)的任务,构建了一个新数据集,内容包含机器学习论文、由GPT-4从这些论文中提取的RQ,以及从多个维度对提取出的RQ开展的人工评估。我们利用该数据集系统比较了近期提出的各类用于摘要任务的基于大语言模型的评估函数,发现没有任何一个函数与人工评估结果表现出足够高的相关性。我们期望该数据集能为进一步研究开发适配RQ提取任务的更优评估函数奠定基础,并助力提升该任务的性能。该数据集可在https://github.com/auto-res/PaperRQ-HumanAnno-Dataset获取。

英文摘要:

The progress in text summarization techniques has been remarkable. However the task of accurately extracting and summarizing necessary information from highly specialized documents such as research papers has not been sufficiently investigated. We are focusing on the task of extracting research questions (RQ) from research papers and construct a new dataset consisting of machine learning papers, RQ extracted from these papers by GPT-4, and human evaluations of the extracted RQ from multiple perspectives. Using this dataset, we systematically compared recently proposed LLM-based evaluation functions for summarizations, and found that none of the functions showed sufficiently high correlations with human evaluations. We expect our dataset provides a foundation for further research on developing better evaluation functions tailored to the RQ extraction task, and contribute to enhance the performance of the task. The dataset is available at https://github.com/auto-res/PaperRQ-HumanAnno-Dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑