规划是人类智能的基本属性。异步计划推理具有挑战性,因为它需要顺序和并行规划来优化时间成本。大型语言模型(LLMs)能成功完成此任务吗?在此,我们提出了首个针对此问题的大规模研究。我们发现,在我们的基准测试AsyncHow中,当未提供关于任务解决过程的说明时,包括GPT-4和LLaMA-2在内的一组代表性的闭源和开源LLMs表现不佳。我们提出了一种名为Plan Like a Graph (PLaG)的新技术,该技术将图与自然语言提示结合,并取得了最先进的结果。我们表明,尽管PLaG可以提升模型性能,但当任务复杂性增加时,LLMs仍会出现急剧的性能下降,突显了利用LLMs模拟数字设备的局限性。我们将我们的研究视为使用LLMs作为高效自主智能体的令人兴奋的一步。我们的代码和数据可在https://github.com/fangru-lin/graph-llm-asynchow-plan获取。
英文摘要:
Planning is a fundamental property of human intelligence. Reasoning about asynchronous plans is challenging since it requires sequential and parallel planning to optimize time costs. Can large language models (LLMs) succeed at this task? Here, we present the first large-scale study investigating this question. We find that a representative set of closed and open-source LLMs, including GPT-4 and LLaMA-2, behave poorly when not supplied with illustrations about the task-solving process in our benchmark AsyncHow. We propose a novel technique called Plan Like a Graph (PLaG) that combines graphs with natural language prompts and achieves state-of-the-art results. We show that although PLaG can boost model performance, LLMs still suffer from drastic degradation when task complexity increases, highlighting the limits of utilizing LLMs for simulating digital devices. We see our study as an exciting step towards using LLMs as efficient autonomous agents. Our code and data are available at https://github.com/fangru-lin/graph-llm-asynchow-plan.
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
Mila Quebec AI Institute(魁北克AI研究所);Université de Montréal(蒙特利尔大学);Allen Institute for AI(人工智能研究院);Google Deepmind(谷歌DeepMind);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
Zoom In, Reason Out: Efficient Far-field Anomaly Detection in Expressway Surveillance Videos via Focused VLM Reasoning Guided by Bayesian Inference
School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);Key Laboratory of Big Data & Artificial Intelligence in Transportation, Ministry of Education(教育部交通运输大数据与人工智能重点实验室);Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence (2018)(北京市交通数据挖掘与具身智能重点实验室)
AP-BMM: Approximating Capability-Cost Pareto Sets of LLMs via Asynchronous Prior-Guided Bayesian Model Merging
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies(广东新型安全智能技术重点实验室);Institute of Cyberspace Security(网络空间安全研究院);School of Computer Science and Technology(计算机科学与技术学院)
Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
Case Western Reserve University, Cleveland, OH, USA(凯斯西储大学);Kean University, Union, NJ, USA(凯恩大学);The Ohio State University, Columbus, OH, USA(俄亥俄州立大学);Fudan University, Shanghai, China(复旦大学);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室);The University of Hong Kong, Hong Kong, China(香港大学);North Carolina State University, Raleigh, NC, USA(北卡罗来纳州立大学)
Reactive Knowledge Representation and Asynchronous Reasoning
Honda Research Institute EU Offenbach am Main, Hesse Germany;Uncertainty in Artificial Intelligence Group, TU Eindhoven Eindhoven, North Brabant Netherlands;Konrad Zuse School of Excellence in Learning;Honda Research Institute EU;Hessian Center for Artificial Intelligence (hessian.AI);Centre for Cognitive Science, TU Darmstadt;German Center for Artificial Intelligence (DFKI)
HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worlds
Artyom Sorokin(AXXX);Lomonosov Moscow State University(罗蒙诺索夫莫斯科国立大学);Higher School of Economics(高等经济学院);Kurchatov Institute(库尔斯克研究所);Independent Researcher(独立研究者)
Harmonizing Real-Time Constraints and Long-Horizon Reasoning: An Asynchronous Agentic Framework for Dynamic Scheduling
School of Computer Science and Engineering, Beihang University, Beijing 100191, China(北京航空航天大学计算机科学与工程学院);Shenzhen Loop Area Institute, Shenzhen, China(深圳环形区研究所);Qingdao Research Institute, Beihang University(青岛研究院);Hangzhou Innovation Institute, Beihang University(杭州创新研究院);School of Artificial Intelligence, Xidian University, Xi’an 710071, Shaanxi, China(西安电子科技大学人工智能学院);Guangzhou Institute of Technology, Xidian University, Guangzhou 510555, Guangdong, China(广州技术研究所)