SPLIT：英语和乌克兰语LLM回应中的跨语言共情与文化根基
SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses
AI总结：
提出SPLIT基准测试，评估LLM在英语和乌克兰语危机情境下的共情准确性、语言自然性和文化根基，发现模型在乌克兰语中表现下降，且人类与AI评估者一致性弱。
摘要：
大型语言模型越来越多地部署在情感支持和危机相关情境中。然而，它们在这些情况下的跨语言能力仍未得到充分探索。现有基准强调多语言性能，但很少考察低到中资源语言中的危机相关共情和文化根基。我们引入了SPLIT，一个500提示的基准，旨在评估LLM在五个类别（压力、恐慌、孤独、内部流离失所和紧张）中生成情感基础回应的一致性。我们评估了三个技术多样的LLM在三个维度上的表现：共情准确性、语言自然性以及情境与文化根基。该框架旨在评估和比较LLM在英语和乌克兰语两种语言中的回应质量，并探索LLM作为陪审团范式的可靠性。我们的发现表明，Gemini-2.5-Flash和LLaMA-3.3-70B-Instruct在转换到乌克兰语时性能下降，而DeepSeek-V3在我们的基准中保持相对稳定。我们还发现，人类和AI评估者在共情和自然性上一致性弱，但在文化根基上存在分歧。我们进一步论证，生成乌克兰语文本并不等同于生成乌克兰语情感支持。我们的发现可能有助于未来开发更具文化针对性的基准设计，并鼓励更加强调以人为中心的评估。
英文摘要：
Large Language Models are increasingly deployed in emotional-support contexts and crisis-related situations. Nevertheless, their cross-lingual abilities in these circumstances remain underexplored. Existing benchmarks emphasize multilingual performance but rarely examine crisis-related empathy and cultural grounding in low-to-mid-resource languages. We introduce SPLIT, a 500-prompt benchmark designed to evaluate LLM consistency in generating emotionally grounded responses across five categories: Stress, Panic, Loneliness, Internal Displacement, and Tension. We evaluate three technically diverse LLMs across three dimensions: Empathetic Accuracy, Linguistic Naturalness, and Contextual & Cultural Grounding. The framework aims to assess and compare the quality of LLM responses in both English and Ukrainian languages, as well as to explore the reliability of the LLM-as-a-jury paradigm. Our findings reveal that Gemini-2.5-Flash and LLaMA-3.3-70B-Instruct degrade when transitioning to Ukrainian, while DeepSeek-V3 remains comparatively stable within our benchmark. We additionally find that human and AI evaluators agree weakly on empathy and naturalness but diverge on cultural grounding. We further argue that producing Ukrainian text is not equivalent to producing Ukrainian emotional support. Our findings may assist in the future development of more culturally tailored benchmark designs, as well as encourage a stronger emphasis on human-centered evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
