在群体中评判:上下文批处理使大语言模型的裁决依赖于其他项目
Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items
AI总结:
该研究发现LLM的上下文批处理裁决会受其他项目影响,不同随机批次导致11-22%决策变化,多数投票仍无法消除多数翻转,建议对模糊项目单提示评分并报告与单项目评分的一致性。
摘要:
为了节省成本,从业者常要求语言模型在一个提示中对多个独立项目进行评分,这一做法假设裁决是可分离的,即对某一项目的裁决不依赖于提示中的其他项目或其在其中的位置。在三项裁决任务(社交媒体评论的危害性、简答题评分、评论星级评分)和五个参数规模为4-9B的开放权重模型上,将相同的2000个项目分配给不同的25个随机批次,会使危害性和评分任务的决策变化11-22%,比重命名项目键或改写指令的影响高出6.5-13.3个百分点,而针对人类标签的AUROC与单项目评分相比仅相差-0.033至+0.010;对于四个参数规模为14-32B的模型,决策变化率为6-15%,影响高出4.2-9.1个百分点。研究表明,翻转率可精确分解为顺序、位置和成员身份的组成部分,在阈值模型下,翻转率随上下文贡献的方差的平方根增长,因此对分数而言较小的效应对决策而言较大。分区之间的潜在差异主要来自邻居的顺序和位置,而哪些项目共享提示的主要影响(无法通过对批次排列投票消除)占差异的0.11-0.22,对9种排列的多数投票仍会留下52-60%的翻转。批处理裁决也与单项目裁决存在系统性差异:即使15个随机批次的多数裁决,仍有8.4-17.8%的决策与单项目裁决不一致。编辑单个生成结果显示,模型自身先前的裁决和邻居的文本都会产生该影响。在简单任务(星级评分)上,仅2.9-3.9%的决策发生变化。研究建议,对模糊项目应每次提示一个进行评分,通过缓存前缀恢复大部分成本,且当项目被批处理时,应报告与单项目评分的一致性。
英文摘要:
To save cost, practitioners often ask a language model to grade many independent items in one prompt. This assumes that verdicts are separable: that the verdict on an item does not depend on the other items in the prompt or on its position among them. Across three judgement tasks (harm in social-media comments, short-answer grading and review star ratings) and five open-weight models with 4-9B parameters, assigning the same 2,000 items to different random batches of 25 changes 11-22% of decisions on harm and grading, 6.5-13.3 points more than renaming the item keys or rewording the instruction, while AUROC against human labels differs from single-item grading only by -0.033 to +0.010; for four models with 14-32B parameters, 6-15% of decisions change, 4.2-9.1 points more. We show that flip rates decompose exactly into components of order, slot and membership, and that under a threshold model they grow with the square root of the variance that the context contributes, so that effects that are small for scores are large for decisions. Most of the latent dissimilarity between partitions comes from the order of the neighbours and the slot. The main effect of which items share the prompt, the part that voting over permutations of a batch cannot remove, accounts for 0.11-0.22 of it, and majority votes over nine permutations still leave 52-60% of the flips. Batched verdicts also differ systematically from single-item verdicts: even the majority of fifteen random batches disagrees with them on 8.4-17.8% of decisions. Editing a single generation shows that both the model's own earlier verdicts and the neighbours' texts carry the effect. On the easy task, star ratings, only 2.9-3.9% of decisions change. We recommend grading ambiguous items one per prompt, recovering most of the cost with a cached prefix, and reporting agreement with single-item grading whenever items are batched.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。