是生成还是判别？衡量LLM文化契合度的方法论考虑
To Generate or Discriminate? Methodological Considerations for Measuring Cultural Alignment in LLMs
AI总结：
本文通过反向社会人口提示方法探讨LLM文化契合度测量问题，发现真实行为表现优于模拟行为，但个体层面个性化存在局限。
摘要：
社会人口提示（SDP）- 使用人口统计数据代理来生成文化契合输出来提示大型语言模型（LLMs）- 通常会使LLM响应显得刻板和有偏见。尽管在评估LLM的文化能力方面有效，但SDP容易受到提示敏感性、解码参数和生成任务固有难度等因素的干扰，因为生成任务的输出空间更大。这些因素使解释变得复杂，难以确定较差表现是由于偏见还是任务设计问题。为了解决这一问题，我们使用反向社会人口提示（ISDP），其中我们提示LLM进行判别并预测人口统计数据代理，从不同用户的真实和模拟行为中进行预测。我们使用Goodreads-CSI数据集（Saha等人，2025），该数据集捕捉了来自印度、墨西哥和美国的用户在理解英文书评时的困难，并测试了四个LLM：Aya-23、Gemma-2、GPT-4o和LLaMA-3.1。结果显示，模型在真实行为上的表现优于模拟行为，与SDP的预测相反。然而，无论是哪种行为类型，表现都会下降并在个体层面变得几乎相等，表明个性化存在局限。
英文摘要：
Socio-demographic prompting (SDP) - prompting Large Language Models (LLMs) using demographic proxies to generate culturally aligned outputs - often shows LLM responses as stereotypical and biased. While effective in assessing LLMs' cultural competency, SDP is prone to confounding factors such as prompt sensitivity, decoding parameters, and the inherent difficulty of generation over discrimination tasks due to larger output spaces. These factors complicate interpretation, making it difficult to determine if the poor performance is due to bias or the task design. To address this, we use inverse socio-demographic prompting (ISDP), where we prompt LLMs to discriminate and predict the demographic proxy from actual and simulated user behavior from different users. We use the Goodreads-CSI dataset (Saha et al., 2025), which captures difficulty in understanding English book reviews for users from India, Mexico, and the USA, and test four LLMs: Aya-23, Gemma-2, GPT-4o, and LLaMA-3.1 with ISDP. Results show that models perform better with actual behaviors than simulated ones, contrary to what SDP suggests. However, performance with both behavior types diminishes and becomes nearly equal at the individual level, indicating limits to personalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
