PrefDisco：主动个性化推理的基准测试
PrefDisco: Benchmarking Proactive Personalized Reasoning
AI总结：
PrefDisco提出了一种评估方法，通过心理基础的人设将静态基准转化为互动个性化任务，定义PrefAlign作为细粒度的偏好对齐度量，揭示个性化推理需要专门开发而非自然产生。
摘要：
当前大型语言模型（LLM）的发展将任务解决和偏好对齐视为两个独立的挑战，首先优化客观正确性，然后优化对聚合人类偏好的对齐。这种范式在面向人类的应用中失效，因为解决问题的正确性不足以在响应与用户需求不匹配时发挥作用。在即需场景中，由于冷启动条件或隐私限制，不存在先前的用户交互历史，这一挑战更加突出。LLM需要主动识别用户不了解的内容，通过提问战略性地获取偏好值，然后相应地调整其推理过程和响应——这是一系列复杂的认知过程，我们称之为个性化推理。我们引入了PrefDisco，一种评估方法，通过心理基础的人设将静态基准转换为互动个性化任务，使用稀疏、上下文相关的偏好，并定义PrefAlign为一种细粒度的基于标准的度量指标，用于衡量偏好对齐。PrefDisco构建了场景，其中相同的问题需要根据用户情境不同的推理链，因为最佳解释方法会因个人专业知识和偏好而异，同时保持事实准确性。在10个任务上对21个前沿模型进行评估，发现29.0%的朴素个性化尝试产生的偏好对齐比通用响应更差，但通用响应也未能满足个人用户需求。这些发现表明，个性化推理需要专门开发，而不是自然产生。PrefDisco为在教育、医疗和技术领域需要个性化应用的系统开发提供了基础。
英文摘要：
Current large language model (LLM) development treats task-solving and preference-alignment as separate challenges, optimizing first for objective correctness, then for alignment to aggregated human preferences. This paradigm fails in human-facing applications where solving a problem correctly is insufficient if the response mismatches the user's needs. This challenge intensifies in just-in-time scenarios where no prior user interaction history exists due to cold-start conditions or privacy constraints. LLMs need to proactively identify what they don't know about the user, strategically elicit preference values through questioning, then adapt their reasoning processes and responses accordingly -- a complicated chain of cognitive processes which we term personalized reasoning. We introduce PrefDisco, an evaluation methodology that transforms static benchmarks into interactive personalization tasks using psychologically-grounded personas with sparse, context-dependent preferences, and define PrefAlign as a fine-grained rubric-based metric for measuring preference alignment. PrefDisco builds scenarios where identical questions require different reasoning chains depending on user context, as optimal explanation approaches vary by individual expertise and preferences while maintaining factual accuracy. Evaluation of 21 frontier models across 10 tasks reveals 29.0% of naive personalization attempts produce worse preference alignment than generic responses, yet generic responses also fail to serve individual user needs. These findings suggest personalized reasoning requires dedicated development rather than emerging naturally. PrefDisco provides a foundation for developing systems that can adapt to individual users in education, healthcare, and technical domains where personalization is critical.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
