arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.18563cs.LGcs.AIcs.CLcs.IR
收藏

利用语言模型逼近人类水平的预测

Approaching Human-Level Forecasting with Language Models

UC Berkeley(加州大学伯克利分校)

Danny Halawi, Fred Zhang, Chen Yueh-Han, Jacob Steinhardt

AI总结:

本文提出检索增强的语言模型预测系统,通过自动检索、预测生成与结果聚合,在知识截止后测试集上接近并在部分场景超过竞争性人类预测者的群体聚合水平。

摘要:

预测未来事件对于政策制定和决策十分重要。在本研究中,我们考察语言模型(LM)能否达到具有竞争力的人类预测者的预测水平。为实现这一目标,我们开发了一个检索增强型 LM 系统,旨在自动搜索相关信息、生成预测并聚合预测结果。为便于研究,我们从竞争性预测平台收集了一个大规模问题数据集。在一个于所用 LM 知识截止日期之后发布的测试集上,我们将该系统的端到端性能与人类预测聚合结果进行比较。平均而言,该系统接近竞争性预测者的群体聚合水平,并在某些场景下超过它。我们的工作表明,利用 LM 预测未来可大规模提供准确预测,并有助于为机构决策提供信息。

英文摘要:

Forecasting future events is important for policy and decision making. In this work, we study whether language models (LMs) can forecast at the level of competitive human forecasters. Towards this goal, we develop a retrieval-augmented LM system designed to automatically search for relevant information, generate forecasts, and aggregate predictions. To facilitate our study, we collect a large dataset of questions from competitive forecasting platforms. Under a test set published after the knowledge cut-offs of our LMs, we evaluate the end-to-end performance of our system against the aggregates of human forecasts. On average, the system nears the crowd aggregate of competitive forecasters, and in some settings surpasses it. Our work suggests that using LMs to forecast the future could provide accurate predictions at scale and help to inform institutional decision making.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑