HelloFresh:基于X社区笔记和维基百科编辑的真实世界人类编辑动作流的LLM评估
HelloFresh: LLM Evaluations on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits
AI总结:
本文提出HelloFresh基准,基于X社区笔记和维基百科编辑的真实人类编辑动作流评估LLM,通过持续更新的真实世界数据缓解测试污染与过拟合,发现配备网络搜索的LLM能产生时间一致的排名。
摘要:
基准测试对于推动机器学习的进步至关重要。更好地理解LLM在现实世界任务中的能力对于安全开发至关重要。设计合适的LLM基准测试具有挑战性:现实世界任务中的数据难以收集,静态评估数据的公开可用性会导致测试数据污染和基准过拟合,而定期生成新的评估数据不仅繁琐,还可能导致时间不一致的结果。我们引入了HelloFresh,它基于由内在动机驱动的人类标注者生成的真实世界数据流。它涵盖了来自X(前身为Twitter)社区笔记和维基百科页面编辑的近期事件,降低了测试数据污染和基准过拟合的风险。任何X用户都可以提议添加X笔记,为具有误导性的帖子(前身为推文)提供额外背景;如果社区将其分类为有帮助的,它将与帖子一起显示。类似地,维基百科依赖于基于社区的共识,允许用户编辑文章或撤销其他用户所做的编辑。验证X笔记是否有帮助或维基百科编辑是否应被接受是困难的任务,需要通过查询网络进行事实依据验证。我们对配备了简单网络搜索访问权限的最先进的LLM进行了回测,发现HelloFresh产生了时间一致的排名。为了在HelloFresh上实现持续评估,我们在https://tinyurl.com/hello-fresh-LLM托管了公开排行榜和定期更新的评估数据。
英文摘要:
Benchmarks have been essential for driving progress in machine learning. A better understanding of LLM capabilities on real world tasks is vital for safe development. Designing adequate LLM benchmarks is challenging: Data from real-world tasks is hard to collect, public availability of static evaluation data results in test data contamination and benchmark overfitting, and periodically generating new evaluation data is tedious and may result in temporally inconsistent results. We introduce HelloFresh, based on continuous streams of real-world data generated by intrinsically motivated human labelers. It covers recent events from X (formerly Twitter) community notes and edits of Wikipedia pages, mitigating the risk of test data contamination and benchmark overfitting. Any X user can propose an X note to add additional context to a misleading post (formerly tweet); if the community classifies it as helpful, it is shown with the post. Similarly, Wikipedia relies on community-based consensus, allowing users to edit articles or revert edits made by other users. Verifying whether an X note is helpful or whether a Wikipedia edit should be accepted are hard tasks that require grounding by querying the web. We backtest state-of-the-art LLMs supplemented with simple web search access and find that HelloFresh yields a temporally consistent ranking. To enable continuous evaluation on HelloFresh, we host a public leaderboard and periodically updated evaluation data at https://tinyurl.com/hello-fresh-LLM.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。