LLM无需任何训练即可看与听
LLMs can see and hear without any training
AI总结：
提出无需训练的MILS框架，利用LLM多步推理迭代生成并评分候选输出，实现零样本图像、视频、音频字幕生成SOTA，并支持媒体生成改进与跨模态嵌入反转。
摘要：
我们提出了MILS：Multimodal Iterative LLM Solver，一种令人惊讶的简单、无需训练的方法，可为你喜爱的LLM赋予多模态能力。利用LLM固有的多步推理能力，MILS提示LLM生成候选输出，每个候选输出都会被评分并迭代反馈，最终生成任务解决方案。这使得各种通常需要在任务特定数据上训练专门模型的应用成为可能。特别是，我们在涌现式零样本图像、视频和音频字幕生成上建立了新的SOTA。MILS也无缝应用于媒体生成，发现提示重写以改进文本到图像生成，甚至编辑提示以进行风格迁移！最后，作为一种无梯度优化方法，MILS可将多模态嵌入反转为文本，实现跨模态算术等应用。
英文摘要：
We present MILS: Multimodal Iterative LLM Solver, a surprisingly simple, training-free approach, to imbue multimodal capabilities into your favorite LLM. Leveraging their innate ability to perform multi-step reasoning, MILS prompts the LLM to generate candidate outputs, each of which are scored and fed back iteratively, eventually generating a solution to the task. This enables various applications that typically require training specialized models on task-specific data. In particular, we establish a new state-of-the-art on emergent zero-shot image, video and audio captioning. MILS seamlessly applies to media generation as well, discovering prompt rewrites to improve text-to-image generation, and even edit prompts for style transfer! Finally, being a gradient-free optimization approach, MILS can invert multimodal embeddings into text, enabling applications like cross-modal arithmetic.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
