arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16779cs.CL
收藏

与内容无关,与呈现方式有关:一种提升阅读理解的极简方法

It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension

University of Colorado Boulder(科罗拉多大学博尔德分校);Johannes Gutenberg University Mainz(美因茨约翰内斯·古腾堡大学)

Sagi Shaier, Lawrence E Hunter, Katharina von der Wense

AI总结:

该研究聚焦阅读理解任务,探究输入顺序与输入强调对大模型性能的影响,发现上下文前置、强调上下文可提升准确率,且仅拼接几个token的简单方法就能让小模型超越大模型。

摘要:

过去十年间,自然语言处理领域发展迅速。由于发展速度快,一些实践未经充分评估就成为通用做法。针对这一现象并聚焦阅读理解任务,我们提出第一个研究问题:1)输入顺序——即问题和上下文的顺序——如何影响模型性能?此外,结合近期输入强调方向的研究进展,我们提出第二个研究问题:2)强调问题、上下文或同时强调二者是否能提升性能?\n我们在3个数据集上对9个大语言模型开展实验,发现将上下文置于问题之前能提升模型性能,准确率最高提升31%。此外,强调上下文的效果优于强调问题;总体而言,强调输入部分内容对于解决模型缺乏参数知识的问题尤其有效。\n我们同时测试了基于提示和基于注意力的强调方法,还发现最佳方法简单得出人意料:仅需在输入中拼接几个token,就能带来最高36%的准确率提升,使较小的模型性能超越规模远大于它的模型。

英文摘要:

Natural language processing has seen rapid progress over the past decade. Due to the speed of developments, some practices get established without proper evaluation. Considering one such case and focusing on reading comprehension, we ask our first research question: 1) How does the order of inputs -- i.e., question and context -- affect model performance? Additionally, given recent advancements in input emphasis, we ask a second research question: 2) Does emphasizing either the question, the context, or both enhance performance? Experimenting with 9 large language models across 3 datasets, we find that presenting the context before the question improves model performance, with an accuracy increase of up to $31\%$. Furthermore, emphasizing the context yields superior results compared to question emphasis, and in general, emphasizing parts of the input is particularly effective for addressing questions that models lack the parametric knowledge to answer. Experimenting with both prompt-based and attention-based emphasis methods, we additionally find that the best method is surprisingly simple: it only requires concatenating a few tokens to the input and results in an accuracy improvement of up to $36\%$, allowing smaller models to outperform their significantly larger counterparts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑