基于多模态LLM的ICT图像描述：弥合通用与行业领域之间的差距
Multi-Modal LLM based Image Captioning in ICT: Bridging the Gap Between General and Industry Domain
AI总结：
本文提出多阶段训练策略，构建ICT领域图像描述模型DICModel，通过合成图像文本对提升模型性能，实验表明其在BLEU指标和准确率上均优于现有模型。
摘要：
在信息与通信技术（ICT）行业中，训练领域专用的大语言模型（LLM）或构建检索增强生成系统需要大量高价值领域知识。然而，这些知识不仅隐藏在文本模态中，也存在于图像模态中。传统方法只能解析文本，但缺乏图像描述能力。多模态LLM（MLLM）能理解图像，但缺乏足够的领域知识。为解决这些问题，本文提出多阶段渐进训练策略，训练ICT领域图像描述模型（DICModel），并构建标准评估系统验证其性能。具体而言，本文首先通过Mermaid工具和LLM合成约7K图像-文本对用于DICModel的第一阶段监督微调（SFT）。然后，ICT领域专家手动标注约2K图像-文本对用于第二阶段SFT。最后，专家和LLM共同合成约1.5K视觉问答数据用于基于指令的SFT。实验结果表明，我们的DICModel仅使用7B参数就优于其他32B参数的最新模型。与7B和32B参数的SOTA模型相比，我们的DICModel在BLEU指标上分别提高了约56.8%和20.8%。在由ICT领域专家构建的客观问题上，我们的DICModel在准确率上优于Qwen2.5-VL 32B模型1%。总之，本文工作能够高效准确地从图像中提取逻辑文本，有望推动ICT领域多模态模型的发展。
英文摘要：
In the information and communications technology (ICT) industry, training a domain-specific large language model (LLM) or constructing a retrieval-augmented generation system requires a substantial amount of high-value domain knowledge. However, the knowledge is not only hidden in the textual modality but also in the image modality. Traditional methods can parse text from domain documents but dont have image captioning ability. Multi-modal LLM (MLLM) can understand images, but they do not have sufficient domain knowledge. To address the above issues, this paper proposes a multi-stage progressive training strategy to train a Domain-specific Image Captioning Model (DICModel) in ICT, and constructs a standard evaluation system to validate the performance of DICModel. Specifically, this work first synthesizes about 7K image-text pairs by combining the Mermaid tool and LLMs, which are used for the first-stage supervised-fine-tuning (SFT) of DICModel. Then, ICT-domain experts manually annotate about 2K image-text pairs for the second-stage SFT of DICModel. Finally, experts and LLMs jointly synthesize about 1.5K visual question answering data for the instruction-based SFT. Experimental results indicate that our DICModel with only 7B parameters performs better than other state-of-the-art models with 32B parameters. Compared to the SOTA models with 7B and 32B parameters, our DICModel increases the BLEU metric by approximately 56.8% and 20.8%, respectively. On the objective questions constructed by ICT domain experts, our DICModel outperforms Qwen2.5-VL 32B by 1% in terms of accuracy rate. In summary, this work can efficiently and accurately extract the logical text from images, which is expected to promote the development of multimodal models in the ICT domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
