arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.21397cs.AI
收藏

如果是瓦特?AI数据中心建设的反事实推理基准

Watts If? A Benchmark for Counterfactual Reasoning About the AI Data Center Buildout

AlphaInfra(阿尔法基础设施公司);New York University(纽约大学);Envana Software Solutions(恩瓦纳软件解决方案公司)

Pankaj Chouhan, Rajarshi Das, Aditya Dubey, Manminder Tomar, Sreekar Reddy, Samta Sharma, Satyapragnya Kar, PR Panigrahi

AI总结:

本研究推出DCBench-440基准,测试LLMs对AI数据中心建设的反事实推理能力,发现其在决策节点动作列举任务上表现极差,生成式与多项选择题准确率分别为12.5%-71.5%、58.3%-92.5%。

摘要:

AI数据中心在消耗电力前数年就已确定:少数离散选择(电网或现场燃气、蒸发式或干式冷却、在哪个县备案)决定了其资本成本、用水量和碳排放路径,而现在要求大型语言模型(LLMs)协助筛选这些选择,却未测试它们是否追踪每项承诺所排除的内容。我们推出DCBench-440,分为两个独立部分报告:针对5种美国交易原型的STRIPS世界生成式问题共200个,带有经搜索验证的标准答案,且任何问题类型上都未测试出超过12%的捷径;基于引用的站点记录、排放方程和既定场景的多项选择题共240个。在开放权重部署中,生成式问题的准确率为12.5%-71.5%,多项选择题的准确率为58.3%-92.5%。有一项技能在所有场景下都表现不佳:要求列出决策节点处的所有可用动作时,没有模型的精确匹配率超过4.0%;最强模型能列出72%-85%会改变项目状态的动作,但遗漏了大部分不改变状态的允许动作,若排除这些动作,最佳精确匹配率仍为24%。代码、数据及其他材料可应要求提供。

英文摘要:

An AI data center is committed years before it draws a watt: a few discrete choices (grid or on-site gas, evaporative or dry cooling, which county to file in) fix its capital, water draw, and carbon path, and LLMs are now asked to help screen them with no test of whether they track what each commitment rules out. We introduce DCBench-440, with two arms reported separately: 200 generative questions over STRIPS worlds for five US deal archetypes, with search-verified gold answers and no tested shortcut above 12% on any question type, and 240 multiple-choice questions grounded in quoted site records, emission equations, and stated scenarios. Across open-weight deployments, accuracy spans 12.5-71.5% (generative) and 58.3-92.5% (multiple choice). One skill fails everywhere: asked to list every action available at a decision gate, no model exceeds 4.0% exact match. The strongest list 72-85% of the actions that would change the project's state but omit most permitted actions that change nothing; with those excused, the best exact match is still 24%. Code, data, and other materials are available upon request.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑