通过大视觉语言模型辅助对齐任务特定视觉模型
LVLM-Aided Alignment of Task-Specific Vision Models
AI总结：
通过大视觉语言模型辅助对齐任务特定视觉模型，提升模型与人类规范的一致性。
摘要：
在高风险领域中，由于其低计算需求和众多可解释结果的方法，小规模任务特定视觉模型至关重要。然而，这些解释常常显示模型与人类领域知识不匹配，依赖于虚假相关性。一旦部署到现实世界，这可能导致脆弱行为。为了解决这个问题，我们提出了一种新颖且高效的对齐小任务特定视觉模型与人类领域知识的方法，利用大视觉语言模型（LVLM）的泛化能力。我们的LVLM辅助视觉对齐（LVLM-VA）方法提供了一个双向接口，将模型行为翻译成自然语言，并将人类类别级规范映射到图像级批评，使领域专家与模型之间能够有效互动。我们的方法在合成和现实世界数据集上均显示出显著提高对齐模型行为与人类规范的能力。我们证明该方法有效减少了模型对虚假特征和群体特定偏见的依赖，而无需细粒度反馈。
英文摘要：
In high-stakes domains, small task-specific vision models are crucial due to their low computational requirements and the availability of numerous methods to explain their results. However, these explanations often reveal that the models do not align well with human domain knowledge, relying instead on spurious correlations. This might result in brittle behavior once deployed in the real-world. To address this issue, we introduce a novel and efficient method for aligning small task-specific vision models with human domain knowledge by leveraging the generalization capabilities of a Large Vision Language Model (LVLM). Our LVLM-Aided Visual Alignment (LVLM-VA) method provides a bidirectional interface that translates model behavior into natural language and maps human class-level specifications to image-level critiques, enabling effective interaction between domain experts and the model. Our method demonstrates substantial improvement in aligning model behavior with human specifications, as validated on both synthetic and real-world datasets. We show that it effectively reduces the model's dependence on spurious features and on group-specific biases, without requiring fine-grained feedback.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
