arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.10840cs.LG
收藏

SimuScene: 通过训练和基准测试代码生成来模拟物理场景

Training and Benchmarking Code Generation for Physics-Inspired Animations

Yanan Wang, Renxi Wang, Yongxin Wang, Xuezhi Liang, Fajri Koto, Timothy Baldwin, Xiaodan Liang, Haonan Li

AI总结:

SimuScene通过训练和基准测试代码生成模型,旨在提高模拟物理场景的能力,实验显示即使最强模型也仅达21.5%的通过率,表明该任务具有挑战性。

摘要:

大型语言模型(LLMs)已广泛研究用于数学竞赛、复杂编码和科学推理等任务,但它们通过代码准确表示和模拟物理场景的能力仍处于探索阶段。我们提出了SimuScene,这是首个系统研究,旨在在五个物理领域和52个物理概念上训练和评估LLMs的模拟物理场景能力。我们构建了一个自动数据收集管道,并通过人工验证确保数据质量。最终的数据集包含7,659个物理场景,其中334个经人工验证的例子作为测试集。我们评估了10个当前最先进的LLMs,发现即使最强的模型也只能达到21.5%的通过率,表明该任务的难度。最后,我们引入了一种强化学习管道,利用视觉语言模型作为裁判来训练文本模型。实验表明,使用我们的数据训练可以提高通过代码进行物理模拟的能力,同时显著提升通用代码生成性能。

英文摘要:

Large language models (LLMs) have been widely studied in areas such as mathematical reasoning, complex coding, and scientific problem solving. However, their ability to generate executable code that visually depicts physical scenarios and their qualitative dynamics remains underexplored. We propose SimuScene, the first systematic study that trains and evaluates LLMs on code generation for physics-inspired animations across 52 concepts spanning five physics domains. We build an automated data collection pipeline with human verification to ensure data quality. The resulting dataset contains 7,659 scenarios, including a 334-example human-verified test set. We evaluate 10 contemporary LLMs and find that even the strongest model achieves only a 21.5\% Avg@8 accuracy, demonstrating the difficulty of generating animations that are both executable and visually aligned with physical scenario descriptions. Finally, we introduce a reinforcement learning pipeline that uses visual rewards from code-generated videos to train text-only LLMs, with a vision-language model evaluating videos through verification questions. Experiments show that training with our data and video-based rewards improves LLM performance on physics-inspired animation generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑