用户行为预测：一种通用、稳健、可扩展且低成本的评估策略，用于估计LLM的泛化能力
User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs
AI总结：
针对数据污染导致LLM泛化评估困难的问题，提出将用户行为预测作为通用、稳健、可扩展且低成本的评估策略，并在推荐数据集上验证了其有效性。
摘要：
由于数据污染，衡量大型语言模型（LLM）的泛化能力具有挑战性。随着模型规模的增长和计算成本的降低，确保任务和测试用例在训练阶段未被见过将变得几乎不可能。我们认为，知识检索和推理任务并不适合衡量泛化能力，因为LLM并非针对特定任务进行训练。相反，我们提出用户行为预测（这也是个性化的一个关键方面）作为一种理论上合理、可扩展且稳健的替代方案。我们为此方法引入了一个新颖的框架，并在电影和音乐推荐数据集上对GPT-4o、GPT-4o-mini和Llama-3.1-8B-Instruct进行了测试。结果与我们的框架预测一致，显示GPT-4o优于GPT-4o-mini和Llama，尽管所有模型都有很大的改进空间，尤其是Llama。
英文摘要：
Measuring the generalization ability of Large Language Models (LLMs) is challenging due to data contamination. As models grow and computation becomes cheaper, ensuring tasks and test cases are unseen during training phases will become nearly impossible. We argue that knowledge-retrieval and reasoning tasks are not ideal for measuring generalization, as LLMs are not trained for specific tasks. Instead, we propose user behavior prediction, also a key aspect of personalization, as a theoretically sound, scalable, and robust alternative. We introduce a novel framework for this approach and test it on movie and music recommendation datasets for GPT-4o, GPT-4o-mini, and Llama-3.1-8B-Instruct. Results align with our framework's predictions, showing GPT-4o outperforms GPT-4o-mini and Llama, though all models have much room for improvement, especially Llama.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
