MlingConf：大语言模型多语言置信度估计的综合研究
MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models
AI总结：
本文提出MlingConf基准，系统研究大语言模型在多语言置信度估计中的表现，发现英语在语言无关任务中占主导，而语言特定任务中相关语言提示更优，并提出母语语气提示策略以提升可靠性。
摘要：
大语言模型（LLMs）产生幻觉的倾向引发了对其可靠性的担忧。因此，能够表明生成内容可信程度的置信度估计变得至关重要。然而，当前针对英语以外语言的LLM置信度估计仍未得到充分探索。本文通过引入对大语言模型多语言置信度估计（MlingConf）的全面调查来填补这一空白，重点关注语言无关（LA）和语言特定（LS）任务，以探索多语言置信度估计在不同任务上的性能和语言主导效应。该基准包含四个经过仔细检查和人工评估的高质量多语言数据集用于LA任务，以及一个针对特定语言的社会、文化和地理背景量身定制的LS任务数据集。我们的实验表明，在LA任务上，英语在置信度估计中表现出比其他语言更显著的语言主导性；而在LS任务上，使用与问题相关的语言来提示LLMs在多语言置信度估计中表现出更好的语言主导性。这些现象启发了一种简单而有效的母语语气提示策略，即对LS任务采用语言特定的提示，有效提高了LLMs在LS任务上的可靠性和准确性。
英文摘要：
The tendency of Large Language Models (LLMs) to generate hallucinations raises concerns regarding their reliability. Therefore, confidence estimations indicating the extent of trustworthiness of the generations become essential. However, current LLM confidence estimations in languages other than English remain underexplored. This paper addresses this gap by introducing a comprehensive investigation of Multilingual Confidence estimation (MlingConf) on LLMs, focusing on both language-agnostic (LA) and language-specific (LS) tasks to explore the performance and language dominance effects of multilingual confidence estimations on different tasks. The benchmark comprises four meticulously checked and human-evaluate high-quality multilingual datasets for LA tasks and one for the LS task tailored to specific social, cultural, and geographical contexts of a language. Our experiments reveal that on LA tasks English exhibits notable linguistic dominance in confidence estimations than other languages, while on LS tasks, using question-related language to prompt LLMs demonstrates better linguistic dominance in multilingual confidence estimations. The phenomena inspire a simple yet effective native-tone prompting strategy by employing language-specific prompts for LS tasks, effectively improving LLMs' reliability and accuracy on LS tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
