LLMs 作为视觉解释器：通过演化的视觉描述推进图像分类
LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions
AI总结：
本文提出一个整合LLMs和VLMs的免训练框架，利用基于LLM的智能体和进化优化策略迭代优化类描述符，有效提升了图像分类准确率并增强了模型的可解释性与稳健性。
摘要：
视觉语言模型（VLMs）通过比较图像和类别嵌入之间的相似性，为图像分类提供了一个有前景的范式。一个关键挑战在于为类名制作精确的文本表示。虽然先前的研究利用大型语言模型（LLMs）的近期进展来增强这些描述符，但它们的输出常常存在模糊和不准确的问题。我们将此归因于两个主要因素：1）依赖于与LLMs的单轮文本交互，导致生成的文本与VLMs的视觉概念不匹配；2）忽视了类间关系，导致描述符无法有效区分相似的类别。在本文中，我们提出了一个整合LLMs和VLMs的新颖框架，我们的免训练方法开发了一个基于LLM的智能体，采用进化优化策略来迭代优化类描述符。我们证明优化后的描述符具有高质量，能有效提升广泛基准上的分类准确率。此外，，这些描述符提供可解释和稳健的特征，提升各种骨干模型的性能，并补充基于微调的方法。
英文摘要：
Vision-language models (VLMs) offer a promising paradigm for image classification by comparing the similarity between images and class embeddings. A critical challenge lies in crafting precise textual representations for class names. While previous studies have leveraged recent advancements in large language models (LLMs) to enhance these descriptors, their outputs often suffer from ambiguity and inaccuracy. We attribute this to two primary factors: 1) the reliance on single-turn textual interactions with LLMs, leading to a mismatch between generated text and visual concepts for VLMs; 2) the oversight of the inter-class relationships, resulting in descriptors that fail to differentiate similar classes effectively. In this paper, we propose a novel framework that integrates LLMs and VLMs to find the optimal class descriptors. Our training-free approach develops an LLM-based agent with an evolutionary optimization strategy to iteratively refine class descriptors. We demonstrate our optimized descriptors are of high quality which effectively improves classification accuracy on a wide range of benchmarks. Additionally, these descriptors offer explainable and robust features, boosting performance across various backbone models and complementing fine-tuning-based methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
