X-modaler：一个多功能、高性能的跨模态分析代码库
X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics
摘要：
随着深度学习在过去十年中的兴起和发展，出现了稳定的创新和突破势头，令人信服地推动了多媒体领域中视觉和语言之间的跨模态分析的最新技术。然而，还没有一个开源代码库来支持以统一和模块化的方式训练和部署用于跨模态分析的众多神经网络模型。在这项工作中，我们提出了X-modaler -一个通用的高性能代码库，它将最先进的跨模态分析封装到几个通用阶段（例如，预处理、编码器、跨模态交互、解码器和解码策略）。每个阶段都具有涵盖广泛采用的一系列模块的功能，并允许在其间进行无缝切换。这种方式自然能够灵活地实现用于图像描述、视频描述和视觉语言预训练的最先进算法，旨在促进研究社区的快速发展。同时，由于在几个阶段中的有效模块化设计（例如，跨模态交互）在不同的视觉语言任务中共享，X-modaler可以简单地扩展为跨模态分析中的其他任务的启动原型，包括视觉问答、视觉常识推理和跨模态检索。X-modaler是一个Apache许可的代码库，其源代码、示例项目和预训练模型可在线获取：https://github.com/YehLi/xmodaler。
英文摘要：
With the rise and development of deep learning over the past decade, there has been a steady momentum of innovation and breakthroughs that convincingly push the state-of-the-art of cross-modal analytics between vision and language in multimedia field. Nevertheless, there has not been an open-source codebase in support of training and deploying numerous neural network models for cross-modal analytics in a unified and modular fashion. In this work, we propose X-modaler -- a versatile and high-performance codebase that encapsulates the state-of-the-art cross-modal analytics into several general-purpose stages (e.g., pre-processing, encoder, cross-modal interaction, decoder, and decode strategy). Each stage is empowered with the functionality that covers a series of modules widely adopted in state-of-the-arts and allows seamless switching in between. This way naturally enables a flexible implementation of state-of-the-art algorithms for image captioning, video captioning, and vision-language pre-training, aiming to facilitate the rapid development of research community. Meanwhile, since the effective modular designs in several stages (e.g., cross-modal interaction) are shared across different vision-language tasks, X-modaler can be simply extended to power startup prototypes for other tasks in cross-modal analytics, including visual question answering, visual commonsense reasoning, and cross-modal retrieval. X-modaler is an Apache-licensed codebase, and its source codes, sample projects and pre-trained models are available on-line: https://github.com/YehLi/xmodaler.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
