Large pre-trained models exhibit distinct and complementary capabilities dependent on the data they are trained on. Language models such as GPT-3 are capable of textual reasoning but cannot understand visual information, while vision models such as DALL-E can generate photorealistic photos but fail to understand complex language descriptions. In this work, we propose a unified framework for composing ensembles of different pre-trained models -- combining the strengths of each individual model to solve various multimodal problems in a zero-shot manner. We use pre-trained models as "generators" or "scorers" and compose them via closed-loop iterative consensus optimization. The generator constructs proposals and the scorers iteratively provide feedback to refine the generated result. Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e.g. improving accuracy on grade school math problems by 7.5%, without requiring any model finetuning. We demonstrate that consensus achieved by an ensemble of scorers outperforms the feedback of a single scorer, by leveraging the strengths of each expert model. Results show that the proposed method can be used as a general purpose framework for a wide range of zero-shot multimodal tasks, such as image generation, video question answering, mathematical reasoning, and robotic manipulation. Project page: https://energy-based-model.github.io/composing-pretrained-models.
Two-Stream Consensus Network for Weakly-Supervised Temporal Action Localization
Xi’an Jiaotong University（西安交通大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；HERE Technologies（HERE科技公司）；State University of New York at Buffalo（纽约州立大学布法罗分校）；Wormpex AI Research（沃姆派克斯人工智能研究院）
MedCoRAG: Interpretable Hepatology Diagnosis via Hybrid Evidence Retrieval and Multispecialty Consensus
School of Cyber Science and Engineering, Nanjing University of Science and Technology（南京理工大学信息科学与工程学院）；School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Department of Neurology, China-Japan Union Hospital of Jilin University（吉林大学中日联谊医院神经内科）；Department of Anesthesiology, China-Japan Union Hospital of Jilin University（吉林大学中日联谊医院麻醉科）；School of Computing, Macquarie University（麦考瑞大学计算机学院）
Agora: Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；School of Information and Telecommunication Engineering, Beijing University of Posts and Telecommunications（北京邮电大学信息与电信工程学院）；Peking University（北京大学）；G Labs（0G实验室）
Morphologically-Aware Consensus Computation via Heuristics-based IterATive Optimization (MACCHIatO)
Université Côte d’Azur（蔚蓝海岸大学）；Inria（法国国家信息与自动化研究所）；Hôpital Pitié-Salpétrière（皮蒂耶-萨尔佩特里埃医院）；Assistance Publique des Hôpitaux de Paris（巴黎公共援助医院）；Sorbonne University（索邦大学）
CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
UMass Amherst（马萨诸塞大学阿默斯特分校）；Wuhan University（武汉大学）；University of California, Los Angeles（加州大学洛杉矶分校）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；South China University of Technology（华南理工大学）
