耦合场景图预测:带约束的召回率何时隐藏了结构不一致性
Coupled Scene Graph Anticipation: When With-Constraint Recall Hides Structural Inconsistency
AI总结:
针对场景图预测中连续时间模型的互斥预测问题,提出CoupledSGA模型,结合图注意力与结构一致性损失,在Action Genome上大幅降低矛盾率,同时指出现有带约束评估指标的局限性并给出报告建议。
摘要:
场景图预测(Scene Graph Anticipation, SGA)用于预测未来视频帧中的物体关系。连续时间SGA模型为每个主体-客体对使用独立的神经ODE(Neural ODE)演化,因此共享同一实体的关系在积分过程中无法相互影响,且无法阻止同一对出现互斥预测,例如“坐在”和“站在”针对同一主体-客体对。我们提出CoupledSGA,该模型通过关系交互图(Relationship Interaction Graph)将此类关系关联起来,在ODE向量场内部使用图注意力机制,并添加基于互斥谓词矩阵的结构一致性损失($\boldsymbol{\textit{L}}_{SC}$)。开发CoupledSGA时,我们发现了评估协议的一个局限性:带约束的评估为每个主体-客体对保留每组中的一个谓词,因此其R@K和mR@K无法捕捉$\boldsymbol{\textit{L}}_{SC}$所针对的组内违规情况——我们证明,只要每组的获胜谓词及其得分保持固定,这些指标就不会改变。我们进一步表明,$\boldsymbol{\textit{L}}_{SC}$的梯度依赖于谓词得分而非真实标签:对于双谓词规则,它会放大模型已有的排序;当组内所有得分相等时,它会更强地抑制属于更多互斥规则的谓词,这解释了为何该惩罚会降低召回率同时提升一致性。在Action Genome数据集上,经过2个预热轮次后激活$\boldsymbol{\textit{L}}_{SC}$,R@20从42.21%降至35.18%,同时无约束矛盾率从45.92%降至4.07%(匹配预算的R@20:CoupledSGA为40.48%,SceneSayerODE为44.78%)。CoupledSGA是一个测试平台,而非声称达到当前最优水平:带约束的召回率捕捉了强制结构一致性的代价,但未捕捉其带来的提升。我们建议,对于约束谓词共同激活的方法,应同时报告无约束结果和矛盾率。
英文摘要:
Scene Graph Anticipation (SGA) predicts object relationships in future video frames. Continuous-time SGA models evolve each subject-object pair with an independent Neural ODE, so relationships sharing an entity cannot influence one another during integration, and nothing prevents mutually exclusive predictions, such as "sitting on" and "standing on", for the same pair. We introduce CoupledSGA, which links such relationships through a Relationship Interaction Graph with graph attention inside the ODE vector field, and adds a Structural Consistency Loss ($\mathcal{L}_{SC}$) based on a matrix of mutually exclusive predicates. Developing CoupledSGA revealed a limitation of the evaluation protocol. With-Constraint evaluation keeps one predicate per group for each subject-object pair, so its R@K and mR@K cannot capture the within-group violations targeted by $\mathcal{L}_{SC}$: we prove they are unchanged whenever each group's winning predicate and its score are held fixed. We further show that the gradient of $\mathcal{L}_{SC}$ depends on predicate scores rather than ground-truth labels: for two-predicate rules it amplifies the ordering the model already holds, and when all scores in a group are equal it suppresses predicates in more exclusion rules more strongly, explaining why the penalty can lower recall while improving consistency. On Action Genome, activating $\mathcal{L}_{SC}$ after two warm-up epochs reduces R@20 from 42.21% to 35.18% while lowering the No-Constraint contradiction rate from 45.92% to 4.07% (matched-budget R@20: 40.48% CoupledSGA vs. 44.78% SceneSayerODE). CoupledSGA is a testbed, not a state-of-the-art claim: With-Constraint recall captures the cost of enforcing structural consistency but not the improvement. We recommend reporting No-Constraint results alongside a contradiction rate for methods that constrain predicate co-activation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。