部署微型LVLM裁判用于图表模型的真实世界评估:经验教训与最佳实践
Deploying Tiny LVLM Judges for Real-World Evaluation of Chart Models: Lessons Learned and Best Practices
AI总结:
本研究提出多标准提示与领域自适应迁移学习,构建微型LVLM裁判ChartJudge,实现低成本高效评估图表模型,并揭示模型规模与可迁移性的权衡。
摘要:
仅拥有70亿参数的大型视觉语言模型(LVLMs)在图表理解任务中已显示出作为自动化裁判的潜力。然而,微型模型(参数不超过20亿)作为裁判时表现仍然不佳,限制了它们在资源受限环境中的实际应用。为解决这一问题,我们提出了两种确保成本高效评估的方法:(i)多标准提示,将多个独立的评估标准合并为单个查询;(ii)领域自适应迁移学习,我们在图表数据集的合成判断上微调一个20亿参数的LVLM,以创建ChartJudge。实验表明,多标准提示暴露了鲁棒性缺陷,导致70亿模型(包括像LLaVA-Critic这样的专用LVLM裁判)的性能大幅下降。此外,我们发现我们的微型LVLM(ChartJudge)能够有效地将知识从一个数据集迁移到另一个数据集,使其成为更专门的模型。我们跨图表类型和查询复杂度的细粒度分析,为模型大小、提示设计和可迁移性之间的权衡提供了可操作的见解,从而实现了图表推理任务的可扩展、低成本评估。
英文摘要:
Large Vision-Language Models (LVLMs) with only 7B parameters have shown promise as automated judges in chart comprehension tasks. However, tiny models (<=2B parameters) still perform poorly as judges, limiting their real-world use in resource-constrained settings. To address this, we propose two approaches to ensure cost-efficient evaluation: (i) multi-criteria prompting, which combines separate evaluation criteria into a single query, and (ii) domain-adaptive transfer learning, in which we fine-tune a 2B-parameter LVLM on synthetic judgments in a chart dataset to create the ChartJudge. Experiments show that multi-criteria prompting exposes robustness gaps, which led to a huge drop in performance for 7B models, including specialized LVLM judges like LLaVA-Critic. In addition, we find that our tiny LVLM (ChartJudge) can effectively transfer knowledge from one dataset to another to make it a more specialized model. Our fine-grained analysis across chart types and query complexities offers actionable insights into trade-offs between model size, prompt design, and transferability, enabling scalable, low-cost evaluation for chart reasoning tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。