arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.09385cs.AI
收藏

AI预测AGI:利用AGI预测与同行评审探索LLMs的复杂推理能力

AI Predicts AGI: Leveraging AGI Forecasting and Peer Review to Explore LLMs' Complex Reasoning Capabilities

ISTAT(意大利国家统计研究所);Maastricht University(马斯特里赫特大学);Università Cattolica del Sacro Cuore(圣心天主教大学);IRCCS Istituto Auxologico Italiano(意大利Auxologico国立卫生研究院)

Fabrizio Davide, Pietro Torre, Leonardo Ercolani, Andrea Gaggioli

AI总结:

本研究让16个LLM预测2030年AGI出现概率,通过自动同行评审评估预测质量,并开发了新的AGI基准以突出性能差异,揭示LLM在推测性预测中的能力。

摘要:

我们让16个最先进的大语言模型(LLMs)估计到2030年人工通用智能(AGI)出现的可能性。为评估这些预测的质量,我们实施了一个自动化的同行评审过程(LLM-PR)。LLMs的估计差异很大,从3%(Reka-Core)到47.6%(GPT-4o)不等,中位数为12.5%。这些估计与最近一项专家调查(预测到2027年AGI出现的可能性为10%)紧密吻合,凸显了LLMs在预测复杂、推测性场景中的相关性。LLM-PR过程表现出很强的可靠性,通过较高的组内相关系数(ICC = 0.79)得以证明,反映了各模型评分的一致性显著。在这些模型中,Pplx-70b-online表现最佳,而Gemini-1.5-pro-api排名最低。与外部基准(如LMSYS Chatbot Arena)的交叉比较显示,LLM排名在不同评估方法下保持一致,表明现有基准可能无法涵盖与AGI预测相关的一些技能。我们进一步探索了基于外部基准的加权方案,以优化LLMs预测与人类专家预测的一致性。这一分析促成了一个新型“AGI基准”的开发,旨在突出AGI相关任务中的性能差异。我们的研究结果提供了对LLMs在推测性、跨学科预测任务中能力的见解,并强调了在复杂、不确定的现实场景中评估AI性能的创新评估框架的日益增长的需求。

英文摘要:

We tasked 16 state-of-the-art large language models (LLMs) with estimating the likelihood of Artificial General Intelligence (AGI) emerging by 2030. To assess the quality of these forecasts, we implemented an automated peer review process (LLM-PR). The LLMs' estimates varied widely, ranging from 3% (Reka- Core) to 47.6% (GPT-4o), with a median of 12.5%. These estimates closely align with a recent expert survey that projected a 10% likelihood of AGI by 2027, underscoring the relevance of LLMs in forecasting complex, speculative scenarios. The LLM-PR process demonstrated strong reliability, evidenced by a high Intraclass Correlation Coefficient (ICC = 0.79), reflecting notable consistency in scoring across the models. Among the models, Pplx-70b-online emerged as the top performer, while Gemini-1.5-pro-api ranked the lowest. A cross-comparison with external benchmarks, such as LMSYS Chatbot Arena, revealed that LLM rankings remained consistent across different evaluation methods, suggesting that existing benchmarks may not encapsulate some of the skills relevant for AGI prediction. We further explored the use of weighting schemes based on external benchmarks, optimizing the alignment of LLMs' predictions with human expert forecasts. This analysis led to the development of a new, 'AGI benchmark' designed to highlight performance differences in AGI-related tasks. Our findings offer insights into LLMs' capabilities in speculative, interdisciplinary forecasting tasks and emphasize the growing need for innovative evaluation frameworks for assessing AI performance in complex, uncertain real-world scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑