Learning representations of spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manipulation. Recent work has investigated various neural architectures for learning multi-modal representations for spatial concepts. However, the lack of explicit reasoning over entities makes such approaches vulnerable to noise in input text or state observations. In this paper, we develop effective models for understanding spatial references in text that are robust and interpretable, without sacrificing performance. We design a text-conditioned \textit{relation network} whose parameters are dynamically computed with a cross-modal attention module to capture fine-grained spatial relations between entities. This design choice provides interpretability of learned intermediate outputs. Experiments across three tasks demonstrate that our model achieves superior performance, with a 17\% improvement in predicting goal locations and a 15\% improvement in robustness compared to state-of-the-art systems.
Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structures
Nara Institute of Science and Technology(奈良科学技術研究所);Guardian Robot Project(守護機器人計劃);RIKEN(理化学研究所);Kyoto University(京都大學);Institute of Science Tokyo(東京科學研究院)
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)
事件接地的目的是链接文本语料库中的提及引用从知识库(KB)的事件。此任务以前的工作主要集中在链接到单个KB事件,从而忽略了事件的层次方面。文档中的事件通常以不同的时空粒度级别进行描述(Glavas et al. 2014)。这些层次关系被用于叙事理解和图式构建的下游任务。在这项工作中,我们提出了一个扩展的事件接地任务,需要解决层次事件结构的知识库。我们建议的任务涉及到链接一个提到引用的一组事件标签从一个子事件层次结构中的知识库。我们提出了一种检索方法,通过辅助分层损失来利用事件分层(Murty et al. 2018)。在维基百科和维基数据自动创建的多语言数据集上,我们的实验证明了分层损失对检索和重新排名基线的有效性(Wu et al. 2020; Pratapa,Gupta和Mitamura 2022)。此外,我们证明了系统的能力,以帮助层次发现之间看不见的事件。
Pragmatically Informative Color Generation by Grounding Contextual Modifiers
Stanford University(斯坦福大学);Institute of High Performance Computing, Agency for Science, Technology and Research(科学技术研究局高性能计算研究所);National University of Singapore(新加坡国立大学)
CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction
Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学);ByteDance Inc.(字节跳动公司);Peng Cheng Laboratory(鹏城实验室);Sun-Yat Sen University(孙中山大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(SZ));Seoul National University(首尔国立大学);Tencent(腾讯)
Graph Alignment Topology as an Inductive Bias for Grounding Detection
Department of Pediatrics, University of Illinois College of Medicine Peoria(伊利诺伊大学皮奥里亚医学院儿科部);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机与数据科学学院);Carle Illinois College of Medicine, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校卡莱医学院)