arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.22971cs.CYcs.AIcs.MA
收藏

多智能体大语言模型评估中的同伴身份偏差:使用TRUST民主话语分析流水线的实证研究

Peer Identity Bias in Multi-Agent LLM Evaluation: An Empirical Study Using the TRUST Democratic Discourse Analysis Pipeline

TRUST Project(TRUST项目)

Juergen Dietrich

AI总结:

本文通过TRUST民主话语分析流水线评估多智能体大语言模型的同伴身份偏差,发现单一通道匿名化几乎无偏差,而完整流水线匿名化揭示了同质化集合体放大身份驱动的阿谀行为,异质化生产配置则相反。

摘要:

TRUST民主话语分析流水线通过多种结构性渠道将大型语言模型(LLM)组件暴露于同伴模型身份——这是一种设计特征,其偏见影响此前尚未经过实证测试。我们提供了首次系统测量TRUST中所有活跃的身份暴露通道的依赖性评分偏差,跨四个模型家族和两个匿名化范围对30个政治陈述进行评估。核心发现是,单一通道匿名化产生近零偏见效应,因为个体通道相互抵消。只有完整流水线匿名化才能揭示真实模式:同质化集合体在模型身份完全可见时放大身份驱动的阿谀行为,而异质化生产配置则相反。模型选择独立起作用:一个测试模型在基线阿谀行为上比其他模型高两到三倍,而在意识形态话题上几乎无辩论冲突,使其在需要真实跨角色分歧的管道中结构上不适用。三个实践结论随之而来。首先,异质模型集合体比同质集合体更结构上稳健,达成更高的共识率和更低的身份放大。第二,完整流水线匿名化是偏见测量的必需条件——部分匿名化不足且积极误导。第三,这些发现对质量关键应用中多智能体LLM系统的验证有直接影响:在部分匿名化或同质集合体下验证的系统可能通过验证但保留结构上不可见的偏见。

英文摘要:

The TRUST democratic discourse analysis pipeline exposes its large language model (LLM) components to peer model identity through multiple structural channels -- a design feature whose bias implications have not previously been empirically tested. We provide the first systematic measurement of identity-dependent scoring bias across all active identity exposure channels in TRUST, crossing four model families with two anonymization scopes across 30 political statements. The central finding is that single-channel anonymization produces near-zero bias effects, because individual channels act in opposite directions and cancel each other out -- a result that would lead an evaluator to conclude that identity bias is absent when it is not. Only full-pipeline anonymization reveals the true pattern: homogeneous ensembles amplify identity-driven sycophancy when model identity is fully visible, while the heterogeneous production configuration shows the reverse. Model choice matters independently: one tested model exhibits baseline sycophancy two to three times higher than the others and near-zero deliberative conflict on ideological topics, making it structurally unsuitable for pipelines where genuine inter-role disagreement is the intended quality mechanism. Three practical conclusions follow. First, heterogeneous model ensembles are structurally more robust than homogeneous ones, achieving higher consensus rates and lower identity amplification. Second, full-pipeline anonymization is required for valid bias measurement -- partial anonymization is insufficient and actively misleading. Third, these findings have direct implications for the validation of multi-agent LLM systems in quality-critical applications: a system validated under partial anonymization or with a homogeneous ensemble may pass validation while retaining structural identity bias invisible to single-channel measurement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑