从视觉、语音和文本中进行语法归纳
Grammar Induction from Visual, Speech and Text
AI总结：
本文提出无监督视觉-音频-文本语法归纳任务 VAT-GI 及无文本设置，构建 VaTiora 框架融合多模态互补信息解析成分语法树，并在两个基准上取得新的最先进性能。
摘要：
语法归纳可以受益于丰富的异构信号，例如文本、视觉和声学信息。在此过程中，来自不同模态的特征本质上发挥着彼此互补的作用。基于这一直觉，本工作提出了一个新颖的无监督视觉-音频-文本语法归纳任务（命名为 VAT-GI），旨在从平行的图像、文本和语音输入中归纳出成分语法树。受语言语法天然存在于文本之外这一事实启发，我们认为在语法归纳中文本不必成为主导模态。因此，我们进一步提出 VAT-GI 的无文本设置，在该设置中任务仅依赖视觉和听觉输入。为解决该任务，我们提出了视觉-音频-文本里-外递归自编码器（VaTiora）框架，利用丰富的模态特有特征和互补特征进行有效的语法解析。此外，我们构建了一个更具挑战性的基准数据集，用于评估 VAT-GI 系统的泛化能力。在两个基准数据集上的实验表明，我们提出的 VaTiora 系统在融合各种多模态信号方面更加有效，并展现了 VAT-GI 新的最先进性能。
英文摘要：
Grammar Induction could benefit from rich heterogeneous signals, such as text, vision, and acoustics. In the process, features from distinct modalities essentially serve complementary roles to each other. With such intuition, this work introduces a novel \emph{unsupervised visual-audio-text grammar induction} task (named \textbf{VAT-GI}), to induce the constituent grammar trees from parallel images, text, and speech inputs. Inspired by the fact that language grammar natively exists beyond the texts, we argue that the text has not to be the predominant modality in grammar induction. Thus we further introduce a \emph{textless} setting of VAT-GI, wherein the task solely relies on visual and auditory inputs. To approach the task, we propose a visual-audio-text inside-outside recursive autoencoder (\textbf{VaTiora}) framework, which leverages rich modal-specific and complementary features for effective grammar parsing. Besides, a more challenging benchmark data is constructed to assess the generalization ability of VAT-GI system. Experiments on two benchmark datasets demonstrate that our proposed VaTiora system is more effective in incorporating the various multimodal signals, and also presents new state-of-the-art performance of VAT-GI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
