语言模型不知道你想要什么：评估深度研究中个性化需求需要真实用户
Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users
AI总结：
本文提出MyScholarQA系统，通过用户兴趣分析、个性化查询响应和报告生成，评估深度研究中个性化需求，发现LLM评估存在不足，需真实用户参与以实现有效个性化。
摘要：
深度研究（DR）系统帮助研究人员应对出版量激增的问题。此类工具通过综合科学论文来回答研究查询，但缺乏对用户需求的理解。我们提出了MyScholarQA（MySQA），一个个性化DR代理，能够推断用户的研究兴趣，提出个性化操作建议，并根据用户批准的操作编写多部分报告。我们首先使用NLP的标准协议测试MySQA：构建了一个包含合成用户和LLM评委的基准测试，其中MySQA在引用指标和个性化操作跟随方面超过了基线。然而，我们怀疑这一过程并未涵盖所有个性化DR用户价值的方面，因此在MySQA的在线版本中采访用户以揭示其需求。我们揭示了九种复杂的个性化DR错误，这些错误无法被我们的LLM评委检测到，并研究了定性反馈以形成未来DR设计的教训。总之，我们主张一个个性化支柱，即易于使用的LLM评委可能使NLP忽视：只有真实用户才能实现真正的个性化进步。
英文摘要：
Deep Research (DR) systems help researchers cope with ballooning publishing counts. Such tools synthesize scientific papers to answer research queries, but lack understanding of their users. We address this with MyScholarQA (MySQA), a personalized DR agent that: 1) infers a profile with a user's research interests; 2) proposes personalized actions for a user's input query; and 3) writes a multi-section report for the query that follows user-approved actions. We first test MySQA with NLP's standard protocol: we build a benchmark with synthetic users and LLM judges, where MySQA beats baselines in citation metrics and personalized action-following. However, we suspect this process does not cover all aspects of personalized DR users value, so we interview users in an online version of MySQA to unmask them. We reveal nine nuanced errors of personalized DR undetectable by our LLM judges, and we study qualitative feedback to form lessons for future DR design. In all, we argue for a pillar of personalization that easy-to-use LLM judges can lead NLP to overlook: real progress in personalization is only possible with real users.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
