评估多模态搜索相关性判断中的成本-准确性权衡
Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements
AI总结：
本文在多个多模态搜索场景中比较 LLM 与 MLLM 同人类判断的一致性，分析成本与准确性权衡，发现性能随情境显著变化且视觉输入可能损害小模型。
摘要：
大语言模型（LLM）已展现出作为有效搜索相关性评估器的潜力。然而，目前缺乏全面指导来说明哪些模型能够在各种情境下或特定用例中持续表现最佳。本文评估了若干 LLM 和多模态语言模型（MLLM）在多个多模态搜索场景中与人类判断的一致性。我们的分析考察了成本与准确性之间的权衡，强调模型性能会随情境不同而显著变化。有趣的是，在较小模型中，加入视觉组件可能非但不会提升性能，反而会妨碍性能。这些发现凸显了为实际应用选择最合适模型所涉及的复杂性。
英文摘要：
Large Language Models (LLMs) have demonstrated potential as effective search relevance evaluators. However, there is a lack of comprehensive guidance on which models consistently perform optimally across various contexts or within specific use cases. In this paper, we assess several LLMs and Multimodal Language Models (MLLMs) in terms of their alignment with human judgments across multiple multimodal search scenarios. Our analysis investigates the trade-offs between cost and accuracy, highlighting that model performance varies significantly depending on the context. Interestingly, in smaller models, the inclusion of a visual component may hinder performance rather than enhance it. These findings highlight the complexities involved in selecting the most appropriate model for practical applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
