STEER-BENCH：评估大型语言模型可操控性的基准
STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models
AI总结：
本文提出 Steer-Bench 基准，利用对比性 Reddit 社区评估 LLMs 的社区敏感型可操控性，发现最佳模型准确率约 65%，显著落后于人类专家的 81%，揭示了现有模型在理解社区规范方面的不足。
摘要：
可操控性，即大型语言模型（LLMs）调整输出以符合不同社区特定规范、视角和沟通风格的能力，对现实世界应用至关重要，但仍未得到充分评估。我们引入 Steer-Bench，一个利用对比性 Reddit 社区评估特定人群操控能力的基准。Steer-Bench 涵盖 19 个领域的 30 对对比性子版块，包含超过 10,000 个指令-响应对和经过验证的 5,500 道多项选择题，并配有相应的银标签，用于测试与多样化社区规范的一致性。我们使用 Steer-Bench 对 13 个流行 LLMs 的评估表明，虽然人类专家在银标签上达到 81% 的准确率，但表现最好的模型根据领域和配置的不同仅达到约 65% 的准确率。一些模型落后于人类水平的一致性超过 15 个百分点，凸显了社区敏感型可操控性方面的显著差距。Steer-Bench 是一个基准，用于系统评估 LLMs 理解社区特定指令的有效性、它们对对抗性操控尝试的鲁棒性，以及它们准确代表多样化文化和意识形态视角的能力。
英文摘要：
Steerability, or the ability of large language models (LLMs) to adapt outputs to align with diverse community-specific norms, perspectives, and communication styles, is critical for real-world applications but remains under-evaluated. We introduce Steer-Bench, a benchmark for assessing population-specific steering using contrasting Reddit communities. Covering 30 contrasting subreddit pairs across 19 domains, Steer-Bench includes over 10,000 instruction-response pairs and validated 5,500 multiple-choice question with corresponding silver labels to test alignment with diverse community norms. Our evaluation of 13 popular LLMs using Steer-Bench reveals that while human experts achieve an accuracy of 81% with silver labels, the best-performing models reach only around 65% accuracy depending on the domain and configuration. Some models lag behind human-level alignment by over 15 percentage points, highlighting significant gaps in community-sensitive steerability. Steer-Bench is a benchmark to systematically assess how effectively LLMs understand community-specific instructions, their resilience to adversarial steering attempts, and their ability to accurately represent diverse cultural and ideological perspectives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
