自适应LLM代理中评估器驱动偏好动态的诊断框架与多评估器审计
A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents
AI总结：
提出EPC诊断框架，包含多模态偏好崩溃指数、评估器索引耦合矩阵和JS散度，通过8个实验条件揭示GPT-4o版本漂移导致评估结论反转，证明单一快照评估研究不可靠。
摘要：
专有LLM评估器的测量结果可能在数周内失效——我们记录了一个案例并提供了检测该问题的诊断框架。我们引入EPC——包括多模态偏好崩溃指数（MPCI）、评估器索引耦合矩阵和Jensen-Shannon散度（JSD）——并将其应用于八个实验条件（N=112主实验 + N=10消融 = 122个独立重复，均已报告）。各条件平均耦合系数范围从0.00到1.18（变异系数约0.9，n=8个条件）。四个条件显示强耦合（N=36；GPT-4o May、GPT-4o-mini、Qwen3.7-plus、DashScope 30r）；四个条件崩溃至接近零（N=76；GPT-4o June、qwen-plus N=30、对称LR、DeepSeek自评估）。从May到June的GPT-4o漂移——一个N=8的重复实验使研究结论反转——是最具信息量的测量：一个诊断仪器检测到自身的不稳定性，证明了它旨在测量的脆弱性。自评估（97%为零，JSD=0.003）持续崩溃，尽管可能存在地板效应。输出格式混淆分析发现，策略层面聚合的rho=0.89，但实例层面rho=0.219（p=0.093）；PCI作为偏好收敛指标报告。我们发布EPC及所有数据。发现并非任何单一耦合强度，而是版本条件不稳定的模式，使得单一快照评估研究不可靠。
英文摘要：
Measurements of proprietary LLM evaluators can become invalid within weeks -- we document one case and provide the diagnostic framework to detect it. We introduce EPC -- comprising the Multimodal Preference Collapse Index (MPCI), evaluator-indexed coupling matrix, and Jensen-Shannon divergence (JSD) -- and apply it across eight experimental conditions (N=112 main + N=10 ablation = 122 unique repetitions, all reported). Coupling coefficients range from 0.00 to 1.18 across per-condition means (CV approx 0.9, n=8 conditions). Four conditions show strong coupling (N=36; GPT-4o May, GPT-4o-mini, Qwen3.7-plus, DashScope 30r); four collapse to near-zero (N=76; GPT-4o June, qwen-plus N=30, symmetric LR, DeepSeek self-eval). The May-to-June GPT-4o drift -- an N=8 re-replication inverting the study's conclusion -- is the most informative measurement: a diagnostic instrument detecting its own instability demonstrates the fragility it was designed to measure. Self-evaluation (97% zero, JSD=0.003) consistently collapses, though floor effects are possible. Output-format confound analysis finds per-strategy aggregate rho=0.89 but per-instance rho=0.219 (p=0.093); PCI reported as preference-convergence metric. We release EPC with all data. The finding is not any single coupling magnitude but the pattern of version-conditional instability that makes single-snapshot evaluator studies unreliable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
