预测过程性视频指令中的隐式论元
Predicting Implicit Arguments in Procedural Video Instructions
AI总结：
针对过程性视频指令中隐式论元预测问题，提出Implicit-VidSRL数据集和iSRL-Qwen2-VL模型，显著提升多模态上下文推理和隐式语义角色标注性能。
摘要：
过程性文本帮助AI增强对上下文和动作序列的推理。将其转化为语义角色标注（SRL）通过识别如{动词，什么，哪里/用什么}的谓词-论元结构来改善对各个步骤的理解。过程性指令高度省略，例如，向碗中加入黄瓜和加入切片番茄，第二步的where论元是从上下文推断出的，指代黄瓜被放置的地方。先前的SRL基准常遗漏隐式论元，导致理解不完整。为解决此问题，我们引入Implicit-VidSRL，一个需要从多模态烹饪过程的上下文信息中推断隐式和显式论元的数据集。我们提出的数据集基准测试多模态模型的上下文推理，要求通过食谱中的视觉变化进行实体追踪。我们研究近期的多模态LLM，揭示它们在给定动词时难以从多模态过程数据中预测what和where/with的隐式论元。最后，我们提出iSRL-Qwen2-VL，在what-隐式的F1分数上实现17%的相对提升，在where/with-隐式语义角色上实现14.7%的相对提升，优于GPT-4o。
英文摘要：
Procedural texts help AI enhance reasoning about context and action sequences. Transforming these into Semantic Role Labeling (SRL) improves understanding of individual steps by identifying predicate-argument structure like {verb,what,where/with}. Procedural instructions are highly elliptic, for instance, (i) add cucumber to the bowl and (ii) add sliced tomatoes, the second step's where argument is inferred from the context, referring to where the cucumber was placed. Prior SRL benchmarks often miss implicit arguments, leading to incomplete understanding. To address this, we introduce Implicit-VidSRL, a dataset that necessitates inferring implicit and explicit arguments from contextual information in multimodal cooking procedures. Our proposed dataset benchmarks multimodal models' contextual reasoning, requiring entity tracking through visual changes in recipes. We study recent multimodal LLMs and reveal that they struggle to predict implicit arguments of what and where/with from multi-modal procedural data given the verb. Lastly, we propose iSRL-Qwen2-VL, which achieves a 17% relative improvement in F1-score for what-implicit and a 14.7% for where/with-implicit semantic roles over GPT-4o.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
