Reinforcement learning (RL) has shown impressive results in sequential decision-making tasks. Meanwhile, Large Language Models (LLMs) and Vision-Language Models (VLMs) have emerged, exhibiting impressive capabilities in multimodal understanding and reasoning. These advances have led to a surge of research integrating LLMs and VLMs into RL. In this survey, we review representative works in which LLMs and VLMs are used to overcome key challenges in RL, such as lack of prior knowledge, long-horizon planning, and reward design. We present a taxonomy that categorizes these LLM/VLM-assisted RL approaches into three roles: agent, planner, and reward. We conclude by exploring open problems, including grounding, bias mitigation, improved representations, and action advice. By consolidating existing research and identifying future directions, this survey establishes a framework for integrating LLMs and VLMs into RL, advancing approaches that unify natural language and visual understanding with sequential decision-making.
University of Pennsylvania（宾夕法尼亚大学）；Brown University（布朗大学）；Nanyang Technological University（南洋理工大学）；University of Notre Dame（圣母大学）；George Mason University（乔治梅森大学）
VisPlay: Self-Evolving Vision-Language Models from Images
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Washington University in St. Louis（华盛顿大学圣路易斯分校）；University of Maryland（马里兰大学）；National University of Singapore（新加坡国立大学）
The Evolving Landscape of Generative Large Language Models and Traditional Natural Language Processing in Medicine
Duke–NUS Medical School（杜克-新加坡国立大学医学院）；University of Cambridge（剑桥大学）；Singapore General Hospital（新加坡中央医院）；Nanyang Technological University（南洋理工大学）；The University of Tokyo（东京大学）；University of Geneva（日内瓦大学）；Duke School of Medicine（杜克大学医学院）；Duke Clinical Research Institute（杜克临床研究所）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Center（新加坡国家眼科中心）；Stanford University（斯坦福大学）；Singapore Health Services（新加坡保健服务集团）；National University of Singapore（新加坡国立大学）
Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；NVIDIA（英伟达公司）；Penn State University（宾夕法尼亚州立大学）；Columbia University（哥伦比亚大学）；University of Missouri-Kansas City（密苏里大学堪萨斯分校）；Florida State University（佛罗里达州立大学）；Auburn University（奥本大学）
MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Data
University of Maryland（马里兰大学）；Brown University（布朗大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Adobe；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Southern California（南加州大学）；NVIDIA
Decouple to Generalize: Context-First Self-Evolving Learning for Data-Scarce Vision-Language Reasoning
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai JiaoTong University（上海交通大学）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang University（浙江大学）
From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape
Research Center for Social Computing and Interactive Robotics（社会计算与交互机器人研究中心）；Department of Computer Science and Technology, Institute for AI（计算机科学与技术系，人工智能研究院）
Self-Evolving Spatial Reasoning in Vision Language Models via Geometric Logic Consistency
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The City University of New York（纽约城市大学）；The Chinese University of Hong Kong（香港中文大学）；Data61 CSIRO（Data61澳大利亚国家科学研究院）；University of New South Wales（新南威尔士大学）；Australian National University（澳大利亚国立大学）
AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models
Hong Kong University of Science and Technology（香港科技大学）；Beijing Jiaotong University（北京交通大学）；Fudan University（复旦大学）；Singapore Management University（新加坡管理大学）；Peng Cheng Laboratory（鹏城实验室）
Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction
Department of Computing, University of Turku（图尔库大学计算机系）；Institute of Computer Science, Zurich University of Applied Sciences（应用科学大学计算机科学研究所）；Centre for Artificial Ingelligence, Zurich University of Applied Sciences（应用科学大学人工智能中心）；Agentic Systems Lab, Department of Management, Technology and Economics, ETH Zürich（苏黎世联邦理工学院管理、科技与经济系代理系统实验室）；Faculty of Mathematics and Information Science, Warsaw University of Technology（华沙技术大学数学与信息科学学院）
Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
Department of Computer Science and Engineering, IIT Patna, India（印度帕纳大学计算机科学与工程系）；Stanford University（斯坦福大学）；Amazon AI（亚马逊人工智能）；Indian Institute of Technology Patna, India（印度帕纳大学）
Red-teaming the Multimodal Reasoning: Jailbreaking Vision-Language Models via Cross-modal Entanglement Attacks
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；People’s Public Security University of China（中国人民公安大学）
VICO: Visual Environments Co-Evolving for Vision-Language Model Reasoning
Virginia Tech（弗吉尼亚理工大学）；Cisco（思科公司）；NVIDIA（英伟达公司）；UChicago（芝加哥大学）；Georgia Tech（佐治亚理工学院）；Abaka AI（Abaka人工智能公司）；UC Berkeley（加州大学伯克利分校）；UGA（佐治亚大学）；WashU（华盛顿大学）
Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；WeChat AI, Tencent Inc., China（腾讯公司，中国）；The University of Hong Kong（香港大学）
Recurrent Reasoning with Vision-Language Models for Estimating Long-Horizon Embodied Task Progress
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（教育部下一代智能搜索与推荐工程研究中心）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）
University of Maryland, College Park（马里兰大学帕克分校）；University of California, Los Angeles（加州大学洛杉矶分校）；Peking University（北京大学）；Arena；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Brown University（布朗大学）；University of Michigan Ann Arbor（密歇根大学安娜堡分校）；Hong Kong University of Science and Technology（香港科学与技术大学）；Carnegie Mellon University（卡内基梅隆大学）；Boise State University（博伊西州立大学）
