Large Multimodal Models (LMMs) have ushered in a new era in artificial intelligence, merging capabilities in both language and vision to form highly capable Visual Foundation Agents. These agents are postulated to excel across a myriad of tasks, potentially approaching general artificial intelligence. However, existing benchmarks fail to sufficiently challenge or showcase the full potential of LMMs in complex, real-world environments. To address this gap, we introduce VisualAgentBench (VAB), a comprehensive and pioneering benchmark specifically designed to train and evaluate LMMs as visual foundation agents across diverse scenarios, including Embodied, Graphical User Interface, and Visual Design, with tasks formulated to probe the depth of LMMs' understanding and interaction capabilities. Through rigorous testing across nine proprietary LMM APIs and eight open models, we demonstrate the considerable yet still developing agent capabilities of these models. Additionally, VAB constructs a trajectory training set constructed through hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and Human Demonstrations, promoting substantial performance improvements in LMMs through behavior cloning. Our work not only aims to benchmark existing models but also provides a solid foundation for future development into visual foundation agents. Code, train \& test data, and part of fine-tuned open LMMs are available at \url{https://github.com/THUDM/VisualAgentBench}.
Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models
Technical University of Munich（慕尼黑工业大学）；Hong Kong Baptist University（香港浸会大学）；Aalto University（阿尔托大学）；University of Oulu（奥卢大学）；University of Bonn（波恩大学）；EPFL（洛桑联邦理工学院）
Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine
Harvard Medical School（哈佛医学院）；Stanford University（斯坦福大学）；Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）
Stress-Testing Multimodal Foundation Models for Crystallographic Reasoning
Dept. of Electrical and Computer Engineering, Texas A&M University（电气与计算机工程系，德克萨斯A&M大学）；College of Science and Engineering, Hamad Bin Khalifa University（科学与工程学院，哈马德·本·卡西姆大学）；Dept. of Electrical and Computer Engineering, Texas A&M University at Qatar（电气与计算机工程系，德克萨斯A&M大学（卡塔尔））；Dept. of Prosthetics and Orthotics, Ankara University（假肢与矫形系，安卡拉大学）
Benchmarking Foundation Models with Multimodal Public Electronic Health Records
Centre for Quantitative Medicine and Duke-NUS AI + Medical Science Initiative, Duke-NUS Medical School（定量医学中心和杜克-国立新加坡大学AI+医学科学计划，杜克-国立新加坡大学医学院）；Graduate School of Engineering, The University of Tokyo（东京大学工程研究生院）；Department of Population Health Sciences, Weill Cornell Medicine（流行病学与公共卫生科学系，韦尔·科恩医学中心）；Department of Surgery, Duke University School of Medicine（外科医学系，杜克大学医学学院）；Centre for Quantitative Medicine, Duke-NUS AI + Medical Science Initiative and Programme in Health Services and Systems Research, Duke-NUS Medical School and NUS Artificial Intelligence Institute, National University of Singapore（定量医学中心和杜克-国立新加坡大学AI+医学科学计划及健康服务与系统研究计划，杜克-国立新加坡大学医学院和新加坡国立大学人工智能研究所）
WildTableBench: Benchmarking Multimodal Foundation Models on Table Understanding In the Wild
The University of Queensland（昆士兰大学）；Stanford University（斯坦福大学）；The Australian National University（澳大利亚国立大学）；Zhejiang University（浙江大学）；Murdoch University（莫纳什大学）；The University of Adelaide（阿德莱德大学）
MSEG-VCUQ: Multimodal SEGmentation with Enhanced Vision Foundation Models, Convolutional Neural Networks, and Uncertainty Quantification for High-Speed Video Phase Detection Data
Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation
Chair of Data Science in Earth Observation, Technical University of Munich (TUM)（地球观测数据科学教授职位，慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；AIML Lab, School of Computer Science, University of St. Gallen（人工智能实验室，圣加尔登大学计算机科学学院）；School of Rural, Surveying and Geoinformatics Engineering, National Technical University of Athens（农村、测绘与地理信息工程学院，国家技术大学雅典）；Image Processing Laboratory (IPL), Universitat de València（图像处理实验室（IPL），瓦伦西亚大学）
Source-Free Domain Adaptation with Frozen Multimodal Foundation Model
University of Shanghai for Science and Technology（上海理工大学）；Universität Hamburg（汉堡大学）；ComOriginMat Inc.（康创矩阵公司）；University of Electronic Science and Technology of China（电子科技大学）；University of Surrey（萨里大学）
An analysis of HOI: using a training-free method with multimodal visual foundation models when only the test set is available, without the training set
Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data
Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Translational Data Analytics Institute, The Ohio State University（转化数据分析研究所，俄亥俄州立大学）；Department of Biomedical Informatics, The Ohio State University（生物医学信息学系，俄亥俄州立大学）
Deep learning has been the subject of growing interest in recent years. Specifically, a specific type called Multimodal learning has shown great promise for solving a wide range of problems in domains such as language, vision, audio, etc. One promising research direction to improve this further has been learning rich and robust low-dimensional data representation of the high-dimensional world with the help of large-scale datasets present on the internet. Because of its potential to avoid the cost of annotating large-scale datasets, self-supervised learning has been the de facto standard for th…
Renmin University of China（中国人民大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Hong Kong（香港大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Multimodal learning, especially large-scale multimodal pre-training, has developed rapidly over the past few years and led to the greatest advances in artificial intelligence (AI). Despite its effectiveness, understanding the underlying mechanism of multimodal pre-training models still remains a grand challenge. Revealing the explainability of such models is likely to enable breakthroughs of novel learning paradigms in the AI field. To this end, given the multimodal nature of the human brain, we propose to explore the explainability of multimodal learning models with the aid of non-invasive br…
Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design computer agents with intelligent capabilities such as understanding, reasoning, and learning through integrating multiple communicative modalities, including linguistic, acoustic, visual, tactile, and physiological messages. With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the mac…
EyeFound: A Multimodal Generalist Foundation Model for Ophthalmic Imaging
The Hong Kong Polytechnic University（香港理工大学）；Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University（香港理工大学夏普视觉研究中心）；Swiss Federal Institute of Technology Lausanne (EPFL)（洛桑联邦理工学院）；Clemson University（克莱姆森大学）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Singapore Eye Research Institute（新加坡眼科研究所）；State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University（中山大学中山眼科中心眼科学国家重点实验室）；Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science（广东省眼科与视觉科学重点实验室）；Guangdong Provincial Clinical Research Center for Ocular Diseases（广东省眼部疾病临床医学研究中心）；Centre for Eye and Vision Research (CEVR)（眼科与视觉研究中心）
Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagery
Guanghua School of Management, Peking University（北京大学光华管理学院）；The Bartlett School of Sustainable Construction, University College London（伦敦大学学院巴特lett可持续建筑学院）；School of Architecture, Tsinghua University（清华大学建筑学院）；School of Computing, The University of Georgia（佐治亚大学计算学院）；Department of Geography, University College London（伦敦大学学院地理系）；Department of Geography and the Environment, University of Texas at Austin（德克萨斯大学奥斯汀分校地理与环境系）
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
InertialAI；Department of Electrical and Computer Engineering, Queen’s University（皇后大学电气与计算机工程系）；Department of Mechanical and Materials Engineering, Queen’s University（皇后大学机械与材料工程系）
Multimodal Foundation Models For Echocardiogram Interpretation
Cedars-Sinai Medical Center（西达赛奈医疗中心）；University of California Los Angeles（加州大学洛杉矶分校）；University of California, San Francisco（加州大学旧金山分校）；San Francisco Veterans Affairs Medical Center（旧金山退伍军人事务医疗中心）
CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer
School of Computer Science and Technology, College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部计算机科学与技术学院）；Data Science Program, Columbian College of Arts and Sciences, The George Washington University（乔治华盛顿大学文理学院哥伦比亚艺术与科学学院数据科学项目）
NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；NExT++ Research Center（NExT++研究中心）；National University of Singapore（新加坡国立大学）
A "Wenlu" Brain System for Multimodal Cognition and Embodied Decision-Making: A Secure New Architecture for Deep Integration of Foundation Models and Domain Knowledge
College of Mechanical and Electrical Engineering, Shijiazhuang University（石家庄大学机械与电气工程学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；The Key Laboratory of Agricultural Robotics Intelligent Perception in Shijiazhuang（石家庄农业机器人智能感知重点实验室）
Representation Potentials of Foundation Models for Multimodal Alignment: A Survey
Department of Electrical and Computer Engineering, Northeastern University（东北大学电气与计算机工程系）；Khoury College of Computer Science, Northeastern University（东北大学科赫里计算机科学学院）
GenRL: Multimodal-foundation world models for generalization in embodied agents
IDLab, Ghent University（IDLab，根特大学）；VERSES AI Research Lab（VERSES AI研究实验室）；Mila, University of Montreal（米拉研究所，蒙特利尔大学）；ServiceNow Research（ServiceNow研究院）
Renmin University of China（中国人民大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University of Surrey（萨里大学）
The fundamental goal of artificial intelligence (AI) is to mimic the core cognitive activities of human. Despite tremendous success in the AI research, most of existing methods have only single-cognitive ability. To overcome this limitation and take a solid step towards artificial general intelligence (AGI), we develop a foundation model pre-trained with huge multimodal data, which can be quickly adapted for various downstream cognitive tasks. To achieve this goal, we propose to pre-train our foundation model by self-supervised learning with weak semantic correlation data crawled from the Inte…
CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Department of Computer Science and Engineering（计算机科学与工程系）；Incheon National University（庆尚国立大学）；Department of Artificial Intelligence（人工智能系）；Inha University（Inha大学）；Center for Brain-Machine Interface（脑机接口中心）
Foundations and Recent Trends in Multimodal Mobile Agents: A Survey
Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）；The Pennsylvania State University（宾夕法尼亚州立大学）；Beijing Jiaotong University（北京交通大学）；University of Liverpool（利物浦大学）
