Recent advancements in large language models (LLMs) have expanded their capabilities beyond traditional text-based tasks to multimodal domains, integrating visual, auditory, and textual data. While multimodal LLMs have been extensively explored for high-level planning in domains like robotics and games, their potential as low-level controllers remains largely untapped. In this paper, we introduce a novel benchmark aimed at testing the emergent capabilities of multimodal LLMs as low-level policies in Atari games. Unlike traditional reinforcement learning (RL) methods that require training for each new environment and reward function specification, these LLMs utilize pre-existing multimodal knowledge to directly engage with game environments. Our study assesses the performances of multiple multimodal LLMs against traditional RL agents, human players, and random agents, focusing on their ability to understand and interact with complex visual scenes and formulate strategic responses. Our results show that these multimodal LLMs are not yet capable of being zero-shot low-level policies. Furthermore, we see that this is, in part, due to their visual and spatial reasoning. Additional results and videos are available on our project webpage: https://dev1nw.github.io/atari-gpt/.
Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels
Zhejiang University(浙江大学);School of Engineering, Westlake University(西湖大学工程学院);University College London(伦敦大学学院);University of Cambridge(剑桥大学);LanBridge Group(兰桥集团);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究院)
From Code to Play: Benchmarking Program Search for Games Using Large Language Models
Institute of Applied AI, Stuttgart Media University(应用人工智能研究所,斯图加特媒体大学);School of Electronic Engineering and Computer Science, Queen Mary University of London(电子工程与计算机科学学院,伦敦女王大学);Institute for Information Processing, Leibniz University Hannover(信息处理研究所,汉诺威莱布尼茨大学);Creative Assembly(创意装配);Chair of Learning Technical Systems, Ruhr-University Bochum(学习技术系统教授职位,博德鲁姆鲁尔大学)
RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);The Hong Kong University of Science and Technology(香港科技大学);Beihang University(北京航空航天大学);University of the Chinese Academy of Sciences(中国科学院大学);ETH Zürich(苏黎世联邦理工学院);The Hong Kong Polytechnic University(香港理工大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shanghai AI Lab(上海人工智能实验室)
Radiology's Last Exam (RadLE): Benchmarking Frontier Multimodal AI Against Human Experts and a Taxonomy of Visual Reasoning Errors in Radiology
Centre for Responsible Autonomous Systems in Healthcare (CRASH) Lab, Koita Centre for Digital Health(负责任的自主医疗系统中心(CRASH)实验室、Koita数字健康中心);Ashoka University(阿什oka大学);Independent Researcher(独立研究者);Koita Centre for Digital Health(Koita数字健康中心)
Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks
Center for Language Engineering(语言工程中心);Al-Khawarizmi Institute of Computer Science(阿尔-花剌子密计算机科学研究所);University of Engineering and Technology Lahore(拉合尔工程技术大学)
Benchmarking large language models for biomedical natural language processing applications and recommendations
Yale University(耶鲁大学);National Institutes of Health(美国国立卫生研究院);University of Texas Health Science at Houston(休斯顿德克萨斯大学健康科学中心);University of Minnesota(明尼苏达大学)
Benchmarking Large Multimodal Models for Ophthalmic Visual Question Answering with OphthalWeChat
The Hong Kong Polytechnic University(香港理工大学);Swiss Federal Institute of Technology Lausanne (EPFL)(洛桑联邦理工学院);PolyU-Wuhan Technology and Innovation Research Institute(香港理工大学武汉技术创新研究院);University of Liverpool(利物浦大学);Centre for Eye and Vision Research (CEVR)(眼科视觉研究中心)
GAUSS: Benchmarking Structured Mathematical Skills for Large Language Models
Hyperbolic Labs;California Institute of Technology(加州理工学院);University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);Nvidia(英伟达);University of Washington(华盛顿大学);University of Hong Kong(香港大学)
Validate Your Authority: Benchmarking LLMs on Multi-Label Precedent Treatment Classification
Department of Information Science and Technology(信息科学与技术系);College of Emergency Preparedness, Homeland Security, and Cybersecurity(应急准备、国土安全与网络安全学院);University at Albany, SUNY(萨利纳大学)