水产养殖中的多模态鱼类摄食强度评估
Multimodal Fish Feeding Intensity Assessment in Aquaculture
AI总结:
本文针对鱼类摄食强度评估中噪声鲁棒性差、计算成本高和缺乏公开数据的问题,提出AV-FFIA数据集和统一混合模态模型U-FFIA,通过模态丢弃与知识蒸馏实现高效多模态评估。
摘要:
鱼类摄食强度评估(FFIA)旨在评估鱼类在摄食过程中的食欲变化,这在工业化水产养殖应用中至关重要。现有的FFIA方法在噪声鲁棒性、计算复杂度以及缺乏用于模型开发的公开数据集方面存在局限性。为了解决这些问题,我们首先引入了AV-FFIA,这是一个包含27,000个带标签的音频和视频片段的新数据集,这些片段捕捉了不同级别的鱼类摄食强度。然后,我们通过利用在单一模态上预训练的模型并与数据融合方法相结合,引入了用于FFIA的多模态方法。我们在AV-FFIA上对这些方法进行了基准研究,并证明了多模态方法相对于基于单一模态方法的优势,尤其是在噪声环境中。然而,与为单一模态开发的方法相比,多模态方法由于需要为每种模态配备独立的编码器,可能涉及更高的计算成本。为了克服这一问题,我们进一步提出了一种新颖的基于统一混合模态的FFIA方法,称为U-FFIA。U-FFIA是一个单一模型,能够处理音频、视觉或音视频模态,通过在训练期间利用模态丢弃(modality dropout)以及使用由单一模态数据预训练的模型进行知识蒸馏来实现。我们证明,U-FFIA能够达到优于或等同于最先进的模态特定FFIA模型的性能,同时显著降低计算开销,从而实现鲁棒且高效的FFIA,以改进水产养殖管理。
英文摘要:
Fish feeding intensity assessment (FFIA) aims to evaluate fish appetite changes during feeding, which is crucial in industrial aquaculture applications. Existing FFIA methods are limited by their robustness to noise, computational complexity, and the lack of public datasets for developing the models. To address these issues, we first introduce AV-FFIA, a new dataset containing 27,000 labeled audio and video clips that capture different levels of fish feeding intensity. Then, we introduce multi-modal approaches for FFIA by leveraging the models pre-trained on individual modalities and fused with data fusion methods. We perform benchmark studies of these methods on AV-FFIA, and demonstrate the advantages of the multi-modal approach over the single-modality based approach, especially in noisy environments. However, compared to the methods developed for individual modalities, the multimodal approaches may involve higher computational costs due to the need for independent encoders for each modality. To overcome this issue, we further present a novel unified mixed-modality based method for FFIA, termed as U-FFIA. U-FFIA is a single model capable of processing audio, visual, or audio-visual modalities, by leveraging modality dropout during training and knowledge distillation using the models pre-trained with data from single modality. We demonstrate that U-FFIA can achieve performance better than or on par with the state-of-the-art modality-specific FFIA models, with significantly lower computational overhead, enabling robust and efficient FFIA for improved aquaculture management.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。