MMFactory：面向视觉-语言任务的通用解决方案搜索引擎
MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
AI总结：
本文提出MMFactory，一个通用的视觉-语言任务解决方案搜索引擎，通过模型和指标路由组件，结合多智能体LLM对话生成可执行、多样化的程序化解决方案，并基准测试性能和资源特征，允许用户根据约束选择最优方案。
摘要：
随着基础模型和视觉-语言模型的进步，以及有效的微调技术，针对各种视觉任务已开发出大量通用和专用模型。尽管这些模型具有灵活性和易用性，但没有任何单一模型能够处理潜在用户可能设想的所有任务和/或应用。近期的方法，如视觉编程和集成工具的多模态LLM，旨在通过程序合成来解决复杂视觉任务。然而，这些方法忽略了用户约束（例如，性能/计算需求），产生难以部署的测试时样本特定解决方案，有时还需要超出普通用户能力的低级指令。为解决这些局限性，我们引入了MMFactory，这是一个包含模型和指标路由组件的通用框架，类似于跨各种可用模型的解决方案搜索引擎。基于任务描述、少量样本输入-输出对以及（可选的）资源和/或性能约束，MMFactory通过实例化和组合其模型库中的视觉-语言工具，建议多样化的程序化解决方案池。除了合成这些解决方案外，MMFactory还提出指标并基准测试性能/资源特征，允许用户选择满足其独特设计约束的解决方案。从技术角度来看，我们还引入了一个基于委员会的解决方案提议器，利用多智能体LLM对话为用户生成可执行、多样化、通用且鲁棒的解决方案。实验结果表明，MMFactory通过提供针对用户问题规格定制的最先进解决方案，优于现有方法。项目页面可在https://davidhalladay.github.io/mmfactory_demo获取。
英文摘要：
With advances in foundational and vision-language models, and effective fine-tuning techniques, a large number of both general and special-purpose models have been developed for a variety of visual tasks. Despite the flexibility and accessibility of these models, no single model is able to handle all tasks and/or applications that may be envisioned by potential users. Recent approaches, such as visual programming and multimodal LLMs with integrated tools aim to tackle complex visual tasks, by way of program synthesis. However, such approaches overlook user constraints (e.g., performance / computational needs), produce test-time sample-specific solutions that are difficult to deploy, and, sometimes, require low-level instructions that maybe beyond the abilities of a naive user. To address these limitations, we introduce MMFactory, a universal framework that includes model and metrics routing components, acting like a solution search engine across various available models. Based on a task description and few sample input-output pairs and (optionally) resource and/or performance constraints, MMFactory can suggest a diverse pool of programmatic solutions by instantiating and combining visio-lingual tools from its model repository. In addition to synthesizing these solutions, MMFactory also proposes metrics and benchmarks performance / resource characteristics, allowing users to pick a solution that meets their unique design constraints. From the technical perspective, we also introduced a committee-based solution proposer that leverages multi-agent LLM conversation to generate executable, diverse, universal, and robust solutions for the user. Experimental results show that MMFactory outperforms existing methods by delivering state-of-the-art solutions tailored to user problem specifications. Project page is available at https://davidhalladay.github.io/mmfactory_demo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
