ZeroNLG：面向零样本多模态与多语言自然语言生成的域对齐与自编码
ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation
AI总结：
提出零样本学习框架ZeroNLG，通过将跨模态与跨语言域投影并对齐到共享潜在空间，并构建无监督多语言自编码器，在无需任何标注下游配对数据的情况下统一处理图像描述、视频描述和机器翻译等多种NLG任务，显著优于现有零样本方法。
摘要：
自然语言生成（NLG）接收图像、视频或文本形式的输入数据，并生成相应的自然语言文本作为输出。现有NLG方法主要采用监督方法，并严重依赖配对的数据-文本对。然而，对于许多目标场景和非英语语言，往往无法获得足够数量的标注数据。为放松对下游任务标注数据的依赖，我们提出了一种直观且有效的零样本学习框架ZeroNLG，它能在统一框架内处理多种NLG任务，包括图像到文本（图像描述）、视频到文本（视频描述）和文本到文本（神经机器翻译），涵盖英语、中文、德语和法语。ZeroNLG在训练时不需要任何标注的下游配对数据。在训练期间，ZeroNLG（i）将不同域（跨模态和跨语言）投影到共享公共潜在空间中的对应坐标；（ii）通过在此空间中对齐不同域的对应坐标来桥接不同域；（iii）构建一个无监督多语言自编码器，通过在给定输入文本于共享潜在空间中的坐标时重建输入文本来学习生成文本。因此，在推理时，基于数据-文本流程，ZeroNLG能在给定输入数据在公共空间中的坐标时跨不同语言生成目标句子。在此统一框架内，以视觉（图像或视频）数据为输入时，ZeroNLG可执行零样本视觉描述；以文本句子为输入时，可执行零样本机器翻译。我们展示了在十二个NLG任务上的大量实验结果，表明在不使用任何标注下游配对数据进行训练的情况下，ZeroNLG能生成高质量且可信的输出，并显著优于现有零样本方法。
英文摘要：
Natural Language Generation (NLG) accepts input data in the form of images, videos, or text and generates corresponding natural language text as output. Existing NLG methods mainly adopt a supervised approach and rely heavily on coupled data-to-text pairs. However, for many targeted scenarios and for non-English languages, sufficient quantities of labeled data are often not available. To relax the dependency on labeled data of downstream tasks, we propose an intuitive and effective zero-shot learning framework, ZeroNLG, which can deal with multiple NLG tasks, including image-to-text (image captioning), video-to-text (video captioning), and text-to-text (neural machine translation), across English, Chinese, German, and French within a unified framework. ZeroNLG does not require any labeled downstream pairs for training. During training, ZeroNLG (i) projects different domains (across modalities and languages) to corresponding coordinates in a shared common latent space; (ii) bridges different domains by aligning their corresponding coordinates in this space; and (iii) builds an unsupervised multilingual auto-encoder to learn to generate text by reconstructing the input text given its coordinate in shared latent space. Consequently, during inference, based on the data-to-text pipeline, ZeroNLG can generate target sentences across different languages given the coordinate of input data in the common space. Within this unified framework, given visual (imaging or video) data as input, ZeroNLG can perform zero-shot visual captioning; given textual sentences as input, ZeroNLG can perform zero-shot machine translation. We present the results of extensive experiments on twelve NLG tasks, showing that, without using any labeled downstream pairs for training, ZeroNLG generates high-quality and believable outputs and significantly outperforms existing zero-shot methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
