读书虽好，但开车时不宜！关于可废止常识规范的可视化接地推理
Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms
AI总结：
针对视觉情境下可废止常识规范推理的挑战，构建NORMLENS基准并发现现有模型与人类判断对齐不足，提出从大语言模型提炼社会常识知识以改进对齐。
摘要：
常识规范可被情境废止：读书通常很好，但开车时则不然。虽然情境可以用语言明确描述，但在具身场景中，情境往往以视觉方式呈现。这种关于可废止常识规范的可视化接地推理对人类来说通常很容易，但（如我们所展示的）对机器却构成挑战，因为它既需要视觉理解，也需要对常识规范的推理。我们构建了一个新的多模态基准来研究视觉接地的常识规范：NORMLENS。NORMLENS包含10K条人类判断，并附有覆盖2K种多模态情境的自由形式解释，作为探针来回答两个问题：（1）模型能在多大程度上与人类平均判断对齐？（2）模型能多好地解释其预测的判断？我们发现，最先进的模型判断和解释与人类标注的对齐程度不佳。此外，我们提出了一种新方法，通过从大型语言模型中提炼社会常识知识来更好地使模型与人类对齐。数据和代码发布于https://seungjuhan.me/normlens。
英文摘要：
Commonsense norms are defeasible by context: reading books is usually great, but not when driving a car. While contexts can be explicitly described in language, in embodied scenarios, contexts are often provided visually. This type of visually grounded reasoning about defeasible commonsense norms is generally easy for humans, but (as we show) poses a challenge for machines, as it necessitates both visual understanding and reasoning about commonsense norms. We construct a new multimodal benchmark for studying visual-grounded commonsense norms: NORMLENS. NORMLENS consists of 10K human judgments accompanied by free-form explanations covering 2K multimodal situations, and serves as a probe to address two questions: (1) to what extent can models align with average human judgment? and (2) how well can models explain their predicted judgments? We find that state-of-the-art model judgments and explanations are not well-aligned with human annotation. Additionally, we present a new approach to better align models with humans by distilling social commonsense knowledge from large language models. The data and code are released at https://seungjuhan.me/normlens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
