PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments
University of Science and Technology of China(中国科学技术大学);City University of Hong Kong(香港城市大学);Northeastern University(东北大学);MemTensor (Shanghai) Technology Co., Ltd.(MemTensor(上海)科技有限公司)
Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of this capability typically interleave preference-related dialogues with irrelevant conversations, reducing the task to needle-in-a-haystack retrieval while ignoring relationships between events driving user preference evolution. Such settings overlook a fundamental characteristic of real-world personalization: preferences emerge gradually and accumulate across interactions within noisy contexts. To bridge this gap, we introduce PERMA, a benchmark designed to evaluate persona consistency over time beyond static preference recall. Additionally, we incorporate (1) text variability and (2) linguistic alignment to simulate erratic user inputs and individual idiolects in real-world data. PERMA consists of temporally ordered interaction events spanning multiple sessions and domains, with preference-related queries inserted over time. We design both multiple-choice and interactive tasks to probe the model's understanding of persona along the interaction timeline. Experiments demonstrate that by linking related interactions, advanced memory systems extract precise preferences and reduce token consumption, outperforming traditional semantic retrieval of raw dialogues. Nevertheless, they still struggle to maintain a coherent persona across temporal depth and cross-domain interference, highlighting the need for more robust personalized memory management in agents. Our code and data are open-sourced at https://github.com/PolarisLiu1/PERMA.
From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents
National University of Singapore(新加坡国立大学);University of Toronto(多伦多大学);University of Minnesota Twin Cities(明尼苏达大学双城分校);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));University of Oxford(牛津大学)
SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes
Midea Group(美的集团);Beijing University of Posts and Telecommunications(北京邮电大学);Donghua University(东华大学);The University of Sydney(悉尼大学);Peking University(北京大学)
PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian Language
School of Computer Science Institute for Research in Fundamental Sciences(计算机科学学院基础科学研究研究所);School of Electrical and Computer Engineering University of Tehran(电气与计算机工程学院塔里班大学);Department of Computer Science Faculty of Mathematical Sciences Alzahra University(计算机科学系数学科学学院阿扎赫拉大学)
Personalized Multimodal Large Language Models: A Survey
University of California, San Diego(加利福尼亚大学圣地亚哥分校);University of Rochester(罗切斯特大学);Dartmouth College(达特茅斯学院);Adobe Research(奥多比研究院);University of Massachusetts at Amherst(马萨诸塞大学阿默斯特分校);University of Southern California(南加利福尼亚大学);Virginia Tech(弗吉尼亚理工大学);Cisco Research(思科研究院);University of Oregon(俄勒冈大学);Meta AI;University of Arizona(亚利桑那大学);University of Maryland(马里兰大学)
AI PERSONA: Towards Life-long Personalization of LLMs
OPPO AI Center(OPPO AI中心);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Beihang University(北京航空航天大学);Tsinghua University(清华大学);Guangdong University of Technology(广东工业大学)
AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shopping
Department of Artificial Intelligence Yonsei University Seoul Republic of Korea(人工智能系 首尔国立庆尚大学 首尔 大韩民国);ParamitaAI Seoul Republic of Korea(ParamitaAI 首尔 大韩民国);Yonsei University(首尔国立庆尚大学);ParamitaAI
Benchmarking Data Heterogeneity Evaluation Approaches for Personalized Federated Learning
Beijing University of Posts and Telecommunications(北京邮电大学);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Institute of High Performance Computing, Agency for Science, Technology and Research(新加坡科技研究局高性能计算研究所);Centre for Frontier AI Research, Agency for Science, Technology and Research(新加坡科技研究局前沿人工智能研究中心);School of Engineering, Westlake University(西湖大学工程学院)
DischargeSim: A Simulation Benchmark for Educational Doctor-Patient Communication at Discharge
Center for Healthcare Organization and Implementation Research, VA Bedford Health Care(VA贝德福德医疗保健中心医疗组织与实施研究中心);Manning College of Information and Computer Sciences, UMass Amherst(马萨诸塞大学阿默斯特分校曼宁信息与计算机科学学院);Miner School of Computer and Information Sciences, UMass Lowell(马萨诸塞大学洛厄尔分校迈纳计算机与信息科学学院);Department of Internal Medicine, Chung-Ang University(韩国中央大学内科学系)
AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
Tsinghua University(清华大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学)
University of California, Riverside(加州大学河滨分校);Yale University(耶鲁大学);Georgia Institute of Technology(佐治亚理工学院);University of Michigan, Ann Arbor(密歇根大学安娜堡分校);University of Toronto(多伦多大学)
AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models
Institute for Clarity in Documentation(文档清晰度研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康库尔中心);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕默研究实验室);State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Institute of Physical Science and Information Technology, Anhui University(光电信息获取与防护技术国家重点实验室,物理科学与信息技术学院,安徽大学);School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, the Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(认知智能国家重点实验室,中国科学技术大学,人工智能研究院,合肥综合性国家科学中心)