arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.23231cs.AI
收藏

PERMA:通过事件驱动的偏好和现实任务环境评估个性化记忆代理

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

University of Science and Technology of China(中国科学技术大学);City University of Hong Kong(香港城市大学);Northeastern University(东北大学);MemTensor (Shanghai) Technology Co., Ltd.(MemTensor(上海)科技有限公司)

Shuochen Liu, Junyi Zhu, Long Shu, Junda Lin, Yuhao Chen, Haotian Zhang, Chao Zhang, Derong Xu, Jia Li, Bo Tang, Zhiyu Li, Feiyu Xiong, Enhong Chen, Tong Xu

AI总结:

本文提出PERMA基准,通过事件驱动的偏好和现实任务环境评估个性化记忆代理的长期一致性,引入文本变异和语言对齐以模拟真实数据中的不规则用户输入和个体语言风格,实验表明先进记忆系统能精准提取偏好并减少token消耗,但仍需更稳健的个性化记忆管理。

摘要:

为构建能适应用户不断变化需求的代理,增强大语言模型的长期记忆能力至关重要。现有评估通常将偏好相关对话与无关对话交织,使任务退化为needle-in-a-haystack检索,忽略了驱动用户偏好演变的事件之间的关系。此类设置忽视了现实世界个性化的一个基本特征:偏好是逐渐形成并在嘈杂环境中跨交互累积的。为弥合这一差距,我们引入PERMA,一个评估时间跨度内人格一致性的基准,超越静态偏好回忆。此外,我们引入(1)文本变异和(2)语言对齐,以模拟现实数据中的不规则用户输入和个体语言风格。PERMA包含跨多个会话和领域的时序排列交互事件,其中偏好相关查询随时间插入。我们设计了多选和交互任务以探测模型对人格的理解沿交互时间线。实验表明,通过关联相关交互,先进记忆系统能够精确提取偏好并减少token消耗,优于传统语义检索原始对话。然而,它们在时间和跨领域干扰中仍难以保持一致的人格,突显了代理中需要更稳健的个性化记忆管理的必要性。我们的代码和数据在https://github.com/PolarisLiu1/PERMA上开源。

英文摘要:

Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of this capability typically interleave preference-related dialogues with irrelevant conversations, reducing the task to needle-in-a-haystack retrieval while ignoring relationships between events driving user preference evolution. Such settings overlook a fundamental characteristic of real-world personalization: preferences emerge gradually and accumulate across interactions within noisy contexts. To bridge this gap, we introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall. Additionally, we incorporate (1) text variability and (2) linguistic alignment to simulate erratic user inputs and individual idiolects in real-world data. PERMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time. We design both multiple-choice and interactive tasks to probe the model's understanding of persona along the interaction timeline. Experiments demonstrate that by linking related interactions, advanced memory systems extract precise preferences and reduce token consumption, outperforming traditional semantic retrieval of raw dialogues. Nevertheless, they still struggle to maintain a coherent persona across temporal depth and cross-domain interference, highlighting the need for more robust personalized memory management in agents. Our code and data are open-sourced at https://github.com/PolarisLiu1/PERMA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑