刻画偏见：在简体中文与繁体中文中对大型语言模型进行基准测试
Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese
AI总结：
研究设计地区用词和姓名选择两个基准任务，审查11个LLM在简繁中文下的性能差异，发现偏见源于训练数据、字符偏好和分词，并开源数据集促进未来评估。
摘要：
尽管大型语言模型（LLMs）在简体中文和繁体中文方面的能力已被研究，但目前尚不清楚LLMs在这两种书面中文变体下被提示时是否表现出差异性表现。这一理解至关重要，因为LLM响应质量的差异可能会因忽视简体中文与繁体中文背后不同的文化背景而加剧表征性伤害，并可能在教育或招聘等领域加剧LLM辅助决策中的下游伤害。为了调查潜在的LLM性能差异，我们设计了两个反映真实世界场景的基准任务：地区用词选择（提示LLM为一个在大陆和台湾有不同称呼的描述物品命名）和地区姓名选择（提示LLM从简体和繁体中文的名单中选择雇佣谁）。对于这两个任务，我们审查了11个领先的商业LLM服务和开源模型的性能——涵盖主要在英语、简体中文或繁体中文上训练的模型。我们的分析表明，LLM响应中的偏见取决于任务和提示语言：虽然大多数LLM在地区用词选择任务中不成比例地偏爱简体中文响应，但令人惊讶的是，他们在地区姓名选择任务中偏爱繁体中文姓名。我们发现，这些差异可能源于训练数据表示、书写字符偏好以及简体中文与繁体中文分词的差异。这些发现凸显了进一步分析LLM偏见的必要性；因此，我们提供了一个开源基准数据集，以促进对未来LLM在中文语言变体中行为的可复现评估（https://github.com/brucelyu17/SC-TC-Bench）。
英文摘要：
While the capabilities of Large Language Models (LLMs) have been studied in both Simplified and Traditional Chinese, it is yet unclear whether LLMs exhibit differential performance when prompted in these two variants of written Chinese. This understanding is critical, as disparities in the quality of LLM responses can perpetuate representational harms by ignoring the different cultural contexts underlying Simplified versus Traditional Chinese, and can exacerbate downstream harms in LLM-facilitated decision-making in domains such as education or hiring. To investigate potential LLM performance disparities, we design two benchmark tasks that reflect real-world scenarios: regional term choice (prompting the LLM to name a described item which is referred to differently in Mainland China and Taiwan), and regional name choice (prompting the LLM to choose who to hire from a list of names in both Simplified and Traditional Chinese). For both tasks, we audit the performance of 11 leading commercial LLM services and open-sourced models -- spanning those primarily trained on English, Simplified Chinese, or Traditional Chinese. Our analyses indicate that biases in LLM responses are dependent on both the task and prompting language: while most LLMs disproportionately favored Simplified Chinese responses in the regional term choice task, they surprisingly favored Traditional Chinese names in the regional name choice task. We find that these disparities may arise from differences in training data representation, written character preferences, and tokenization of Simplified and Traditional Chinese. These findings highlight the need for further analysis of LLM biases; as such, we provide an open-sourced benchmark dataset to foster reproducible evaluations of future LLM behavior across Chinese language variants (https://github.com/brucelyu17/SC-TC-Bench).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
