MAviS：一种用于鸟类物种的多模态对话助手
MAviS: A Multimodal Conversational Assistant For Avian Species
AI总结：
本文提出MAviS数据集和MAviS-Chat模型，通过整合图像、音频和文本信息，提升对鸟类物种的细粒度理解与多模态问答能力，并展示了在生态应用中领域适应的多模态大语言模型的重要性。
摘要：
细粒度理解和特定物种的多模态问答对于推进生物多样性保护和生态监测至关重要。然而，现有的多模态大语言模型在处理如鸟类物种等专业领域时面临挑战，难以提供准确且上下文相关的信息。为此，我们引入了MAviS数据集，这是一个大规模的多模态鸟类物种数据集，整合了图像、音频和文本模态，涵盖超过1000种鸟类物种，包含预训练和指令微调子集，并补充了结构化的问答对。基于MAviS数据集，我们引入了MAviS-Chat，一种支持音频、视觉和文本的多模态大语言模型，旨在实现细粒度物种理解、多模态问答和场景特定描述生成。最后，为了定量评估，我们提出了MAviS-Bench，一个包含超过25,000个问答对的基准测试，用于评估跨模态的鸟类物种特定感知和推理能力。实验结果表明，MAviS-Chat在基准MiniCPM-o-2.6上表现显著优于基线，实现了最先进的开源结果，并展示了我们指令微调MAviS数据集的有效性。我们的发现强调了在生态应用中领域适应的多模态大语言模型的必要性。
英文摘要：
Fine-grained understanding and species-specific multimodal question answering are vital for advancing biodiversity conservation and ecological monitoring. However, existing multimodal large language models face challenges when it comes to specialized topics like avian species, making it harder to provide accurate and contextually relevant information in these areas. To address this limitation, we introduce the MAviS-Dataset, a large-scale multimodal avian species dataset that integrates image, audio, and text modalities for over 1,000 bird species, comprising both pretraining and instruction-tuning subsets enriched with structured question-answer pairs. Building on the MAviS-Dataset, we introduce MAviS-Chat, a multimodal LLM that supports audio, vision, and text and is designed for fine-grained species understanding, multimodal question answering, and scene-specific description generation. Finally, for quantitative evaluation, we present MAviS-Bench, a benchmark of over 25,000 QA pairs designed to assess avian species-specific perceptual and reasoning abilities across modalities. Experimental results show that MAviS-Chat outperforms the baseline MiniCPM-o-2.6 by a large margin, achieving state-of-the-art open-source results and demonstrating the effectiveness of our instruction-tuned MAviS-Dataset. Our findings highlight the necessity of domain-adaptive multimodal LLMs for ecological applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
