迈向地理文化接地的 LLM 生成
Towards Geo-Culturally Grounded LLM Generations
AI总结：
研究比较了标准LLM、知识库接地和搜索接地在文化认知基准上的表现，发现搜索接地能显著提升命题知识测试表现，但会增加刻板判断风险且未提升开放式文化流利度。
摘要：
生成式大型语言模型（LLMs）在全球范围内表现出对不同文化认知的差距。我们研究了检索增强生成和搜索接地技术对 LLMs 展现对各国家文化熟悉度能力的影响。具体而言，我们比较了标准 LLMs、使用定制知识库检索增强的 LLMs（即 KB grounding）以及使用网络搜索检索增强的 LLMs（即 search grounding）在多个文化认知基准上的表现。我们发现，搜索接地显著提升了 LLMs 在测试命题知识（例如文化规范、人工制品和制度）的多项选择基准上的表现，而 KB grounding 的有效性则受限于知识库覆盖不足和检索器次优。然而，搜索接地也增加了语言模型做出刻板判断的风险，并且在具有足够统计功效的人类评估中，未能提升评估者对文化熟悉度的判断。这些结果突显了在评估 LLMs 的文化认知时，命题性文化知识与开放式文化流利度之间的区别。
英文摘要：
Generative large language models (LLMs) have demonstrated gaps in diverse cultural awareness across the globe. We investigate the effect of retrieval augmented generation and search-grounding techniques on LLMs' ability to display familiarity with various national cultures. Specifically, we compare the performance of standard LLMs, LLMs augmented with retrievals from a bespoke knowledge base (i.e., KB grounding), and LLMs augmented with retrievals from a web search (i.e., search grounding) on multiple cultural awareness benchmarks. We find that search grounding significantly improves the LLM performance on multiple-choice benchmarks that test propositional knowledge (e.g., cultural norms, artifacts, and institutions), while KB grounding's effectiveness is limited by inadequate knowledge base coverage and a suboptimal retriever. However, search grounding also increases the risk of stereotypical judgments by language models and fails to improve evaluators' judgments of cultural familiarity in a human evaluation with adequate statistical power. These results highlight the distinction between propositional cultural knowledge and open-ended cultural fluency when it comes to evaluating LLMs' cultural awareness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
