基于大语言模型的客服代理质量保障系统的反事实公平性评估
Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System
AI总结：
本文研究了基于大语言模型的客服代理质量保障系统在公平性方面的表现，发现系统性偏见问题，指出需标准化的公平性审计流程。
摘要：
大型语言模型（LLMs）正越来越多地用于客服中心的质量保证（QA），以自动化代理绩效评估和辅导反馈。尽管LLMs提供了前所未有的可扩展性和速度，但它们对网络级训练数据的依赖引发了关于人口统计和行为偏见的担忧，这些偏见可能会扭曲劳动力评估。我们展示了基于LLM的QA系统在13个维度上的反事实公平性评估，涵盖身份、上下文和行为风格三大类别。公平性通过反事实翻转率（CFR）来量化，即二元判断反转的频率，以及平均绝对分数差异（MASD），即反事实配对之间辅导或信心分数的平均变化。在3000份真实的客服中心 transcripts 上评估18个LLMs，我们发现系统性差异，CFR范围从5.4%到13.0%，在信心、积极和改进分数上存在一致的MASD变化。较大的、更强烈对齐的模型显示出较低的不公平性，尽管公平性不跟踪准确性。上下文优先级历史表现导致最严重的退化（CFR高达16.4%），而隐含的语言身份线索仍然是持续的偏见来源。最后，我们分析了公平性意识提示的有效性，发现明确的指令仅在评估一致性上产生有限的改进。我们的发现强调了在部署LLMs用于高风险劳动力评估之前，需要标准化的公平性审计流程。
英文摘要：
Large Language Models (LLMs) are increasingly deployed in contact-center Quality Assurance (QA) to automate agent performance evaluation and coaching feedback. While LLMs offer unprecedented scalability and speed, their reliance on web-scale training data raises concerns regarding demographic and behavioral biases that may distort workforce assessment. We present a counterfactual fairness evaluation of LLM-based QA systems across 13 dimensions spanning three categories: Identity, Context, and Behavioral Style. Fairness is quantified using the Counterfactual Flip Rate (CFR), the frequency of binary judgment reversals, and the Mean Absolute Score Difference (MASD), the average shift in coaching or confidence scores across counterfactual pairs. Evaluating 18 LLMs on 3,000 real-world contact center transcripts, we find systematic disparities, with CFR ranging from 5.4% to 13.0% and consistent MASD shifts across confidence, positive, and improvement scores. Larger, more strongly aligned models show lower unfairness, though fairness does not track accuracy. Contextual priming of historical performance induces the most severe degradations (CFR up to 16.4%), while implicit linguistic identity cues remain a persistent bias source. Finally, we analyze the efficacy of fairness-aware prompting, finding that explicit instructions yield only modest improvements in evaluative consistency. Our findings underscore the need for standardized fairness auditing pipelines prior to deploying LLMs in high-stakes workforce evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
