BenSyc: 孟加拉语上下文中大语言模型对话谄媚与人类对齐的基准测试
BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts
AI总结：
提出BenSyc基准，基于孟加拉语社交数据构建五级标注集，评估15+模型在对话对齐分类与生成任务上的表现，发现前沿模型在区分共情与强化性认可上仍存在困难。
摘要：
大型语言模型（LLMs）越来越多地参与情感敏感的社交对话，其回应可能从平衡支持转向过度认可或升级性对齐。现有的谄媚研究主要关注事实一致性和指令遵循设置，而文化背景下的对话谄媚尚未得到充分探索。我们引入了BenSyc，这是首个用于研究孟加拉语社交语境中对话谄媚的基准。从孟加拉国和西孟加拉邦社区收集的11,840条Reddit帖子和170k条评论出发，我们构建了一个人工验证的基准，包含二元标签和一个细粒度的五级分类体系，涵盖无效化、中立、支持、认可和升级。我们在对话对齐分类和响应生成任务上评估了超过15个开源和专有LLM。结果表明，即使对于前沿的指令调优模型，区分共情性支持与强化导向的认可仍然具有挑战性：最佳系统在二元检测上仅达到61.8 Macro-F1，在五类分类上达到61.7 Macro-F1。在生成设置中，多个模型在情感激烈的情境下频繁产生强烈认可或升级性回应。我们的发现凸显了不同模型家族和对话行为之间的显著差异，强调了文化背景下的多语言基准对于评估社交对齐的对话AI系统的重要性。
英文摘要：
Large language models (LLMs) increasingly participate in emotionally sensitive social conversations, where responses may shift from balanced support toward excessive validation or escalatory alignment. Existing sycophancy research primarily focuses on factual agreement and instruction-following settings, leaving culturally grounded conversational sycophancy underexplored. We introduce BenSyc, the first benchmark for studying conversational sycophancy in Bengali social contexts. Starting from 11,840 Reddit posts and 170k comments collected from communities across Bangladesh and West Bengal, we construct a human-validated benchmark with binary labels and a fine-grained five-level taxonomy spanning Invalidation, Neutral, Support, Validation, and Escalation. We evaluate more than 15 open and proprietary LLMs on conversational alignment classification and response generation tasks. Results show that distinguishing empathetic support from reinforcement-oriented validation remains challenging even for frontier instruction-tuned models: the best system achieves only 61.8 Macro-F1 on binary detection and 61.7 Macro-F1 on five-class classification. In generation settings, several models frequently produce strongly validating or escalatory responses in emotionally charged situations. Our findings highlight substantial variation across model families and conversational behaviors, underscoring the importance of culturally grounded multilingual benchmarks for evaluating socially aligned conversational AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
