Recent advances in large language models (LLMs) have enhanced their ability to process long input contexts. This development is particularly crucial for tasks that involve retrieving knowledge from an external datastore, which can result in long inputs. However, recent studies show a positional bias in LLMs, demonstrating varying performance depending on the location of useful information within the input sequence. In this study, we conduct extensive experiments to investigate the root causes of positional bias. Our findings indicate that the primary contributor to LLM positional bias stems from the inherent positional preferences of different models. We demonstrate that merely employing prompt-based solutions is inadequate for overcoming the positional preferences. To address this positional bias issue of a pre-trained LLM, we developed a Position-Aware Parameter Efficient Fine-Tuning (PAPEFT) approach which is composed of a data augmentation technique and a parameter efficient adapter, enhancing a uniform attention distribution across the input context. Our experiments demonstrate that the proposed approach effectively reduces positional bias, improving LLMs' effectiveness in handling long context sequences for various tasks that require externally retrieved knowledge.
HSE University（俄罗斯国立高等经济大学）；Lomonosov Moscow State University（莫斯科国立大学）；Harbin Institute of Technology（哈尔滨工业大学）；ISP RAS（俄罗斯科学院系统编程研究所）；RANEPA（俄罗斯总统国民经济与公共管理学院）；ITMO University（圣彼得堡国立信息技术机械与光学大学）；Skoltech（斯科尔科沃科学技术研究院）
Mitigating Coordinate Prediction Bias from Positional Encoding Failures
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；University of California, Los Angeles（加州大学洛杉矶分校）；The University of Queensland（昆士兰大学）；Shanghai University of Finance and Economics（上海财经大学）
Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Models
Huawei Noah's Ark Lab.（华为诺亚实验室）；City University of Hong Kong.（香港城市大学）；University of Sydney.（悉尼大学）；State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（北京大学人工智能国家重点实验室，智能科学与技术学院）
Layer-wise Positional Bias in Short-Context Language Modeling
Tehran Institute for Advanced Studies, Khatam University, Iran（德黑兰高级研究院，卡坦大学，伊朗）；School of Electrical and Computer Engineering, University of Tehran, Iran（德黑兰理工大学电气与计算机工程学院，伊朗）
VRoPE: Rotary Position Embedding for Video Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）；Basic Algorithm Center, Tencent（腾讯基础算法中心）
A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University（知识驱动人机智能工程研究中心）；International Center of Future Science, Jilin University（未来科学国际中心）
Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narratives
Department of Computer Science, Columbia University（哥伦比亚大学计算机科学系）；Department of Applied Physics and Applied Mathematics, Columbia University（哥伦比亚大学应用物理与应用数学系）；Department of Psychology, Northwestern University（西北大学心理学系）
Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan（国家台湾大学计算机科学与信息工程系）；Institute of Information Science, Academia Sinica, Taiwan（台湾学术院信息科学研究所）；AI Research Center (AINTU), National Taiwan University, Taiwan（国家台湾大学人工智能研究中心（AINTU））
Are LLMs Positionally Consistent Ordinal Classifiers? A Systematic Evaluation
Cornell University（康奈尔大学）；University of Washington（华盛顿大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of California, Davis（加州大学戴维斯分校）
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Microsoft Corporation（微软公司）；School of Computer Science, Peking University（北京大学计算机学院）；National Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）
Are LLMs Rational Investors? A Study on Detecting and Reducing the Financial Bias in LLMs
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Institute of Fintech, Fudan University（复旦大学金融科技研究院）；School of Electronics and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）；DataGrand Inc.（数据堂公司）
