MarineGPT：向公众解锁海洋的秘密
MarineGPT: Unlocking Secrets of Ocean to the Public
AI总结：
针对通用多模态大模型在海洋领域理解能力不足的问题，提出首个海洋视觉-语言模型MarineGPT，利用包含超500万图文对的Marine-5M数据集注入专业知识，实现视觉语言对齐，为领域专家适应提供标准协议。
摘要：
大型语言模型（LLMs），如ChatGPT/GPT-4，已被证明是作为AI助手提升用户体验的强大工具。持续的研究正在提出多模态大型语言模型（MLLM），通过构建联合语义空间（例如视觉-文本空间）赋予LLMs感知多模态输入的能力。尽管LLMs和MLLMs取得了显著成功，但在需要特定领域知识和专业知识的特定领域应用中探索LLMs和MLLMs的研究开展得较少，尤其是对于海洋领域。与通用MLLMs不同，海洋特定的MLLM需要产生更加敏感、信息丰富且科学的响应。在这项工作中，我们证明了在大量现成的通用训练数据上优化得到的现有MLLMs，在理解特定领域意图并随后生成信息丰富且令人满意的响应方面表现出极小的能力。为了解决这些问题，我们提出了MarineGPT，这是第一个专门为海洋领域设计的视觉-语言模型，向公众解锁海洋的秘密。我们展示了包含超过500万个海洋图像-文本对的Marine-5M数据集，以将特定领域的海洋知识注入我们的模型，并实现更好的海洋视觉和语言对齐。我们的MarineGPT不仅将海洋理解的边界推向普通公众，还为将通用助手适应到下游特定领域专家提供了一个标准协议。我们为广泛的海洋应用铺平了道路，同时为学术界和工业界的未来研究设定了有价值的数据和预训练模型。
英文摘要：
Large language models (LLMs), such as ChatGPT/GPT-4, have proven to be powerful tools in promoting the user experience as an AI assistant. The continuous works are proposing multi-modal large language models (MLLM), empowering LLMs with the ability to sense multiple modality inputs through constructing a joint semantic space (e.g. visual-text space). Though significant success was achieved in LLMs and MLLMs, exploring LLMs and MLLMs in domain-specific applications that required domain-specific knowledge and expertise has been less conducted, especially for \textbf{marine domain}. Different from general-purpose MLLMs, the marine-specific MLLM is required to yield much more \textbf{sensitive}, \textbf{informative}, and \textbf{scientific} responses. In this work, we demonstrate that the existing MLLMs optimized on huge amounts of readily available general-purpose training data show a minimal ability to understand domain-specific intents and then generate informative and satisfactory responses. To address these issues, we propose \textbf{MarineGPT}, the first vision-language model specially designed for the marine domain, unlocking the secrets of the ocean to the public. We present our \textbf{Marine-5M} dataset with more than 5 million marine image-text pairs to inject domain-specific marine knowledge into our model and achieve better marine vision and language alignment. Our MarineGPT not only pushes the boundaries of marine understanding to the general public but also offers a standard protocol for adapting a general-purpose assistant to downstream domain-specific experts. We pave the way for a wide range of marine applications while setting valuable data and pre-trained models for future research in both academic and industrial communities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
