arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.29112cs.AIcs.CL
收藏

GISTBench:通过证据基础的兴趣验证评估大语言模型的用户理解能力

GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification

Meta Recommendation Systems (MRS)(Meta推荐系统(MRS))

Iordanis Fostiropoulos, Muhammad Rafay Azhar, Abdalaziz Sawwan, Boyu Fang, Yuchen Liu, Jiayi Liu, Hanchao Yu, Qi Guo, Jianyu Wang, Fei Liu, Xiangjun Fan

AI总结:

GISTBench通过证据基础的兴趣验证评估大语言模型在推荐系统中理解用户的能力,提出两种新指标并验证了合成数据集的准确性。

摘要:

我们介绍了GISTBench,一个用于评估大语言模型(LLMs)从交互历史中理解用户能力的基准。不同于传统推荐系统基准关注物品预测准确率,我们的基准评估LLMs从参与数据中提取和验证用户兴趣的能力。我们提出了两种新的度量家族:兴趣 groundedness(IG),分解为精度和召回组件以分别惩罚幻觉兴趣类别并奖励覆盖范围,以及兴趣 specificity(IS),评估已验证的LLM预测用户档案的差异性。我们发布了一个基于全球短视频平台真实用户交互构建的合成数据集。该数据集包含隐式和显式参与信号以及丰富的文本描述。我们通过用户调查验证了数据集的准确性,并评估了八个开源权重LLMs,参数范围从7B到120B。我们的发现揭示了当前LLMs的性能瓶颈,特别是其在异构交互类型中准确计数和归因参与信号的能力有限。

英文摘要:

We introduce GISTBench, a benchmark for evaluating Large Language Models' (LLMs) ability to understand users from their interaction histories in recommendation systems. Unlike traditional RecSys benchmarks that focus on item prediction accuracy, our benchmark evaluates how well LLMs can extract and verify user interests from engagement data. We propose two novel metric families: Interest Groundedness (IG), decomposed into precision and recall components to separately penalize hallucinated interest categories and reward coverage, and Interest Specificity (IS), which assesses the distinctiveness of verified LLM-predicted user profiles. We release a synthetic dataset constructed on real user interactions on a global short-form video platform. Our dataset contains both implicit and explicit engagement signals and rich textual descriptions. We validate our dataset fidelity against user surveys, and evaluate eight open-weight LLMs spanning 7B to 235B parameters, together with three proprietary frontier models (GPT-5, Claude 4.6, and Gemini 3.5 Flash). Our findings reveal performance bottlenecks in current LLMs, particularly their limited ability to accurately count and attribute engagement signals across heterogeneous interaction types.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑