EVALUESTEER：衡量奖励模型对价值观和偏好的可引导性
EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences
AI总结：
EVALUESTEER基准通过合成16.5万偏好对，评估LLM和奖励模型对用户价值观与风格偏好的可引导性，发现完整档案下最佳模型准确率低于75%，揭示当前模型在适应多样化偏好上的局限。
摘要：
随着大型语言模型（LLMs）在全球范围内的部署，创建能够适应全球用户多样化偏好和价值观的多元系统变得至关重要。我们引入了EVALUESTEER，一个基准测试，用于衡量LLMs和奖励模型（RMs）对基于心理学和人机交互文献的用户价值观和风格偏好档案的可引导性。为解决现有数据集不支持对RM引导进行受控评估的空白，我们综合生成了165,888个偏好对——系统地沿4个价值维度（传统、世俗理性、生存和自我表达）和4个风格维度（冗长性、可读性、自信度和温暖度）变化这些对。我们使用EVALUESTEER来评估，在给定用户档案和一对候选的带有价值观和风格的响应时，LLMs和RMs是否能够选择与用户偏好一致的输出。我们在十一种系统性提示条件和六种偏好比较场景下评估了六个开源和专有的LLMs和RMs。值得注意的是，我们的结果表明，当给定用户的完整价值观和风格偏好档案时，最佳模型在选择正确响应方面的准确率低于75%，而仅提供相关风格和价值观偏好时，准确率则超过99%。因此，EVALUESTEER凸显了当前RMs在识别和适应相关用户档案信息方面的局限性，并为开发能够被引导向多样化人类价值观和偏好的RMs提供了一个具有挑战性的测试平台。
英文摘要：
As large language models (LLMs) are deployed globally, creating pluralistic systems that can accommodate the diverse preferences and values of users worldwide becomes essential. We introduce EVALUESTEER, a benchmark to measure LLMs' and reward models' (RMs) steerability towards users' value and stylistic preference profiles grounded in psychology and human-LLM interaction literature. To address the gap in existing datasets that do not support controlled evaluations of RM steering, we synthetically generated 165,888 preference pairs -- systematically varying pairs along 4 value dimensions (traditional, secular-rational, survival, and self-expression) and 4 style dimensions (verbosity, readability, confidence, and warmth). We use EVALUESTEER to evaluate whether, given a user profile and a pair of candidate value-laden and style-laden responses, LLMs and RMs are able to select the output that aligns with the user's preferences. We evaluate six open-source and proprietary LLMs and RMs under eleven systematic prompting conditions and six preference comparison scenarios. Notably, our results show that, when given the user's full profile of values and stylistic preferences, the best models achieve <75% accuracy at choosing the correct response, in contrast to >99% accuracy when only relevant style and value preferences are provided. EVALUESTEER thus highlights the limitations of current RMs at identifying and adapting to relevant user profile information, and provides a challenging testbed for developing RMs that can be steered towards diverse human values and preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
