arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.14641cs.CLcs.AI
收藏

相关信息片段之间的距离导致长上下文LLMs中的偏差

Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs

Tsinghua University(清华大学);Renmin University of China(中国人民大学);Apple Inc.(苹果公司)

Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Huadong Wang, Xiaojiang Liu

AI总结:

提出LongPiBench基准评估长上下文LLMs中多个相关信息片段的位置偏差,发现现有模型虽克服了“迷失在中间”问题,但仍存在与信息片段间距相关的显著偏差。

摘要:

大型语言模型(LLMs)中的位置偏差阻碍了它们有效处理长输入的能力。一个突出的例子是“迷失在中间”现象,即LLMs难以利用位于输入中间的相关信息。虽然先前的研究主要集中在单个相关信息片段上,但实际应用通常涉及多个相关信息片段。为了弥补这一差距,我们提出了LongPiBench,这是一个旨在评估涉及多个相关信息片段的位置偏差的基准。我们使用5个商业模型和6个开源模型进行了全面实验。这些实验表明,尽管大多数当前模型对“迷失在中间”问题具有鲁棒性,但存在与相关信息片段间距相关的显著偏差。这些发现强调了评估和减少位置偏差对于提升LLM能力的重要性。

英文摘要:

Positional bias in large language models (LLMs) hinders their ability to effectively process long inputs. A prominent example is the "lost in the middle" phenomenon, where LLMs struggle to utilize relevant information situated in the middle of the input. While prior research primarily focuses on single pieces of relevant information, real-world applications often involve multiple relevant information pieces. To bridge this gap, we present LongPiBench, a benchmark designed to assess positional bias involving multiple pieces of relevant information. Thorough experiments are conducted with five commercial and six open-source models. These experiments reveal that while most current models are robust against the "lost in the middle" issue, there exist significant biases related to the spacing of relevant information pieces. These findings highlight the importance of evaluating and reducing positional biases to advance LLM's capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑