相同规则,不同裁决:韩语支持对话LLM评判者中的评判标准语言与从句结构
Same Rule, Different Verdicts: Rubric Language and Clause Structure in LLM Judges for Korean Support Dialogues
AI总结:
该研究探究韩语支持对话的LLM评判者裁决是否受评判标准语言和从句结构影响,发现韩语标准下跨模型一致性更低,分歧集中在特定结构,且未支持两个相关假设。
摘要:
韩语服务通常会用韩语编写评估评判标准并交给LLM评判者。本研究测试评判者的裁决是否取决于评判标准的语言以及评判标准条目的表述方式。我们为支持对话编写了12条标准,采用4种从句结构(肯定、否定、条件、例外),其中两对结构含义相同,且每个条目分别提供韩语和英语版本。来自3家提供商的6个低成本模型,加上1个参考模型,在每种组合下评判60个翻译后的ESConv对话和144个构建的边界案例(7个模型共51744次请求)。分析计划在数据收集前已确定并承诺执行。跨提供商的一致性在韩语评判标准下低于英语评判标准(Cohen's kappa为0.687 vs 0.745;差值为-0.058,95%置信区间[-0.072,-0.045])。分歧集中在否定、条件和例外条目(与肯定条目相比的优势比为5.6)。单个模型在含义相同的两个条目间改变裁决的比例为11%-15%。两个假设未得到支持:同一提供商的模型之间的一致性并不高于不同提供商的模型,添加配对示例也未缩小语言差距。我们仅测量一致性;未使用人工标签,所有材料均由LLM起草。
英文摘要:
Services that operate in Korean often write their evaluation rubrics in Korean and hand them to an LLM judge. We test whether a judge's verdict depends on the language of the rubric and on how a rubric item is phrased. We write 12 criteria for support dialogues in four clause structures (affirmative, negated, conditional, exception) such that two pairs of structures have the same meaning, and we give each item in Korean and in English. Six low-cost models from three providers, plus one reference model, judge 60 translated ESConv dialogues and 144 constructed boundary cases under every combination (51,744 requests across the seven models). The analysis plan was fixed and committed before data collection. Cross-provider agreement is lower with Korean rubrics than with English rubrics (Cohen's kappa 0.687 vs. 0.745; difference -0.058, 95% CI [-0.072,-0.045]). Disagreement concentrates in negated, conditional, and exception items (odds ratio 5.6 relative to affirmative items). A single model changes its verdict between two items with the same meaning in 11-15% of cases. Two hypotheses are not supported: models from the same provider do not agree more than models from different providers, and adding paired examples does not reduce the language gap. We measure consistency only; no human labels are used, and all materials were drafted by an LLM.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。