P3B3：用于测量大语言模型中欧洲和巴西葡萄牙语变体偏差的多轮对话基准
P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs
AI总结：
提出P3B3基准，通过专家策划的对话提示和评估框架，测量大语言模型在葡萄牙语变体（欧洲vs巴西）上的偏差和可控性，发现多数模型偏向巴西葡萄牙语。
摘要：
随着大语言模型（LLMs）融入日常交流，捕捉区域语言变异对于可靠和公平的语言使用至关重要。在葡萄牙语中，欧洲（pt-PT）和巴西（pt-BR）变体仍然代表性不均，pt-BR在数据量上占主导地位，而LLM对葡萄牙语变体的偏好尚未得到充分探索。为弥补这一空白，我们引入了P3B3，一个由专家策划的语言变体无关的对话提示基准，以及一个用于测量变体偏差和可控性的评估框架。在多个模型上的实验表明，大多数LLM表现出对pt-BR的强烈偏差，且不同模型的可控性存在差异。这些结果凸显了需要在语言变体之间实现更平衡的多语言表示。
英文摘要：
As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portuguese, European (pt-PT) and Brazilian (pt-BR) varieties remain unevenly represented, with pt-BR dominating in data quantity, while LLM preference for Portuguese variants remains underexplored. To address this gap, we introduce P3B3, an expert-curated language variety agnostic benchmark of conversational prompts, along with an evaluation framework for measuring variety bias and controllability. Experiments on several models show that most LLMs exhibit a strong bias toward pt-BR, with variation in controllability across models. These results highlight the need for more balanced multilingual representation across language varieties.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
