UserSumBench：用于评估用户摘要方法的基准框架
UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches
AI总结：
本文提出UserSumBench基准框架，通过无参考质量指标和结合时间分层摘要与自我批评验证的鲁棒方法，解决用户摘要缺乏标签、主观且人工评估昂贵的问题。
摘要：
大语言模型（LLM）在从一长串原始用户活动数据生成用户摘要方面展现出卓越能力。这些摘要捕捉偏好和兴趣等关键用户信息，因此对于基于LLM的个性化应用（如可解释推荐系统）具有重要价值。然而，新摘要技术的发展受到以下因素阻碍：缺乏真实标签、用户摘要固有的主观性，以及通常成本高昂且耗时的人工评估。为应对这些挑战，我们提出UserSumBench，这是一个旨在促进基于LLM的摘要方法迭代开发的基准框架。该框架提供两个关键组件：（1）一种无参考摘要质量指标。我们证明，该指标在三个不同数据集（MovieLens、Yelp和Amazon Review）上均有效，并且与人类偏好保持一致。（2）一种新颖的鲁棒摘要方法，该方法利用时间分层摘要器和自我批评验证器生成高质量摘要，同时消除幻觉。该方法可作为摘要技术进一步创新的强有力基线。
英文摘要：
Large language models (LLMs) have shown remarkable capabilities in generating user summaries from a long list of raw user activity data. These summaries capture essential user information such as preferences and interests, and therefore are invaluable for LLM-based personalization applications, such as explainable recommender systems. However, the development of new summarization techniques is hindered by the lack of ground-truth labels, the inherent subjectivity of user summaries, and human evaluation which is often costly and time-consuming. To address these challenges, we introduce \UserSumBench, a benchmark framework designed to facilitate iterative development of LLM-based summarization approaches. This framework offers two key components: (1) A reference-free summary quality metric. We show that this metric is effective and aligned with human preferences across three diverse datasets (MovieLens, Yelp and Amazon Review). (2) A novel robust summarization method that leverages time-hierarchical summarizer and self-critique verifier to produce high-quality summaries while eliminating hallucination. This method serves as a strong baseline for further innovation in summarization techniques.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
