定向域微调：为特定训练任务定制独立模态
Directed Domain Fine-Tuning: Tailoring Separate Modalities for Specific Training Tasks
AI总结：
本研究提出定向域微调方法，针对烹饪任务用小体量专属指令数据集和LORA微调Video-LLaVA的各模态参数，在YouCook2数据集上以远小于基线的训练数据实现了2%的性能提升。
摘要：
大语言模型（LLM）和大视觉语言模型（LVLM）一直是人工智能领域的前沿方向，尤其适用于文本生成、视频字幕生成和问答等任务。通常而言，在更广泛的知识库或数据集上训练这些模型更具实用性，以此提升泛化能力、学习主题间的关联并识别模式。与之不同，我们提出为不同域内各模态的对应任务提供专属指令数据集，随后使用LORA（低秩适配）微调模型参数。通过我们的方法，能够消除所有与给定任务无关的噪声，同时确保模型生成精度得到提升。在本研究中，我们使用Video-LLaVA模型，在无转录文本的情况下根据烹饪视频生成食谱。Video-LLaVA的多模态架构支持我们向其图像编码器输入烹饪图像、向视频编码器输入烹饪视频、向文本编码器输入通用烹饪问题。由此，我们的目标是移除所有与烹饪无关的噪声，同时增强模型生成具体食材清单和详细操作步骤的能力。实验结果显示，我们对Video-LLaVA的微调方法在YouCook2数据集上较基线Video-LLaVA提升了2%。尽管这一提升看似幅度不大，但我们的模型训练所用的图像指令数据集规模仅为Video-LLaVA的2.5%，视频指令数据集规模仅为Video-LLaVA的23.76%。
英文摘要：
Large language models (LLMs) and large visual language models (LVLMs) have been at the forefront of the artificial intelligence field, particularly for tasks like text generation, video captioning, and question-answering. Typically, it is more applicable to train these models on broader knowledge bases or datasets to increase generalizability, learn relationships between topics, and recognize patterns. Instead, we propose to provide instructional datasets specific to the task of each modality within a distinct domain and then fine-tune the parameters of the model using LORA. With our approach, we can eliminate all noise irrelevant to the given task while also ensuring that the model generates with enhanced precision. For this work, we use Video-LLaVA to generate recipes given cooking videos without transcripts. Video-LLaVA's multimodal architecture allows us to provide cooking images to its image encoder, cooking videos to its video encoder, and general cooking questions to its text encoder. Thus, we aim to remove all noise unrelated to cooking while improving our model's capabilities to generate specific ingredient lists and detailed instructions. As a result, our approach to fine-tuning Video-LLaVA leads to gains over the baseline Video-LLaVA by 2% on the YouCook2 dataset. While this may seem like a marginal increase, our model trains on an image instruction dataset 2.5% the size of Video-LLaVA's and a video instruction dataset 23.76% of Video-LLaVA's.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
