各有所尺：发现用于物理视频评估的每VLM分类体系
Each Judge Its Own Yardstick: Discovering Per-VLM Taxonomies for Physical Video Evaluation
AI总结：
提出JudgeFit方法，通过迭代优化为每个视觉语言模型发现其专属的物理视频评估分类体系，在16个VLM上平均相对提升约32%，并揭示模型特定盲点。
摘要：
在视频生成器和世界模型中保持物理一致性越来越依赖视觉语言模型（VLM）作为自动评判者，提供奖励信号、排序决策和数据过滤标准。然而，VLM在训练数据和架构上差异显著，通过不同的内部表示编码物理现象。因此，单一的全局评估模式给每个VLM相同的胜任维度，而不考虑每个模型实际能感知什么。我们提出JudgeFit，一种迭代优化过程，发现每个VLM的评估分类体系。初始分类体系通过提示目标VLM枚举一小批视频中的物理错误并对结果描述进行聚类构建。然后通过诊断步骤优化分类体系：将VLM的每维度分数校准到人类物理常识评分，诊断哪些维度评分不可靠或冗余，并提示LLM修复它们，迭代直至收敛。我们进一步将该过程实例化为一个基准，并应用于跨越8个模型家族的16个VLM。优化后的分类体系在保留视频上对所有测试的VLM均优于全局模式基线，平均相对提升约32%。除了总体准确性，每个VLM的配置文件暴露了整体排名无法预料的模型特定盲点，不同模型家族的可靠性模式差异显著。
英文摘要：
Maintaining physical consistency in video generators and world models increasingly relies on vision-language models (VLMs) as automated judges that provide reward signals, ranking decisions, and data-filtering criteria. Yet VLMs differ substantially in training data and architecture, encoding physical phenomena through distinct internal representations. A single global evaluation schema therefore gives every VLM the same axes of competence, regardless of what each can actually perceive. We propose JudgeFit, an iterative refinement procedure that discovers a per-VLM evaluation taxonomy. An initial taxonomy is constructed by prompting the target VLM to enumerate physics errors on a small set of videos and clustering the resulting descriptions. The taxonomy is then refined through a diagnostic step: we calibrate the VLM's per-dimension scores to human physical-commonsense ratings, diagnose which dimensions it scores unreliably or redundantly, and prompt an LLM to repair them, iterating until convergence. We further instantiate this procedure as a benchmark and apply it to 16 VLMs spanning eight model families. The refined taxonomy outperforms the global-schema baseline on held-out videos for every VLM tested, with a mean relative improvement of approximately 32%. Beyond aggregate accuracy, the per-VLM profiles expose model-specific blind spots that overall rankings cannot anticipate, with reliability patterns differing markedly across model families.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
