AlpsBench: 一个面向真实对话记忆与偏好对齐的LLM个性化基准
AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment
AI总结：
AlpsBench通过真实人类与LLM对话数据构建，包含2500个长期交互序列及验证的记忆，评估个性化信息提取、更新、检索与利用等核心任务，揭示LLM在记忆管理中的局限性。
摘要：
随着大语言模型（LLM）发展为终身AI助手，LLM个性化成为关键前沿。然而，当前进展受限于缺乏黄金标准评估基准。现有基准或忽视个性化信息管理，或依赖合成对话，存在现实对话分布差距。为弥合此差距，我们引入AlpsBench，一个基于真实人类-LLM对话的LLM个性化基准。AlpsBench包含2500个长期交互序列，源自WildChat，配以人工验证的结构化记忆，涵盖显性与隐性个性化信号。我们定义四个关键任务——个性化信息提取、更新、检索与利用，并建立评估记忆管理全生命周期的协议。对前沿LLM和记忆导向系统进行基准测试，揭示：(i) 模型难以可靠提取潜在用户特征；(ii) 记忆更新在最强模型中也面临性能瓶颈；(iii) 大规模干扰池下检索准确率急剧下降；(iv) 显性记忆机制提升召回率，但不保证更偏好对齐或情感共鸣的响应。AlpsBench旨在提供全面的评估框架。
英文摘要：
As Large Language Models (LLMs) evolve into lifelong AI assistants, LLM personalization has become a critical frontier. However, progress is currently bottlenecked by the absence of a gold-standard evaluation benchmark. Existing benchmarks either overlook personalized information management that is critical for personalization or rely heavily on synthetic dialogues, which exhibit an inherent distribution gap from real-world dialogue. To bridge this gap, we introduce AlpsBench, An LLM PerSonalization benchmark derived from real-world human-LLM dialogues. AlpsBench comprises 2,500 long-term interaction sequences curated from WildChat, paired with human-verified structured memories that encapsulate both explicit and implicit personalization signals. We define four pivotal tasks - personalized information extraction, updating, retrieval, and utilization - and establish protocols to evaluate the entire lifecycle of memory management. Our benchmarking of frontier LLMs and memory-centric systems reveals that: (i) models struggle to reliably extract latent user traits; (ii) memory updating faces a performance ceiling even in the strongest models; (iii) retrieval accuracy declines sharply in the presence of large distractor pools; and (iv) while explicit memory mechanisms improve recall, they do not inherently guarantee more preference-aligned or emotionally resonant responses. AlpsBench aims to provide a comprehensive framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
