测量大型语言模型在偏好排序中的不一致性
Measuring the Inconsistency of Large Language Models in Preferential Ranking
AI总结：
本研究基于序理论形式化定义了偏好排序的一致性标准，通过诊断实验揭示了LLMs在传递性、无关替代方案独立性等方面的显著不一致性及强烈的positional bias。
摘要：
尽管大型语言模型（LLMs）近期取得了进展，但其偏见和幻觉问题依然存在，且其提供一致偏好排序的能力仍有待探索。本研究调查了LLMs提供一致序数偏好的能力，这在决策空间密集或缺乏绝对答案的场景中至关重要。我们引入了基于序理论的 consistency 形式化定义，概述了传递性、不对称性、可逆性以及独立性（独立于无关替代方案）等标准。我们对选定的 state-of-the-art LLMs 进行的诊断实验表明，它们无法满足这些标准，显示出强烈的 positional bias 和较差的传递性，其偏好极易被无关替代方案左右。这些发现突显了 LLM 生成的偏好排序中存在显著的不一致性，强调了需要进一步研究以解决这些局限性。
英文摘要：
Despite large language models' (LLMs) recent advancements, their bias and hallucination issues persist, and their ability to offer consistent preferential rankings remains underexplored. This study investigates the capacity of LLMs to provide consistent ordinal preferences, a crucial aspect in scenarios with dense decision space or lacking absolute answers. We introduce a formalization of consistency based on order theory, outlining criteria such as transitivity, asymmetry, reversibility, and independence from irrelevant alternatives. Our diagnostic experiments on selected state-of-the-art LLMs reveal their inability to meet these criteria, indicating a strong positional bias and poor transitivity, with preferences easily swayed by irrelevant alternatives. These findings highlight a significant inconsistency in LLM-generated preferential rankings, underscoring the need for further research to address these limitations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
