Despite recent progress, learning new tasks through language instructions remains an extremely challenging problem. On the ALFRED benchmark for task learning, the published state-of-the-art system only achieves a task success rate of less than 10% in an unseen environment, compared to the human performance of over 90%. To address this issue, this paper takes a closer look at task learning. In a departure from a widely applied end-to-end architecture, we decomposed task learning into three sub-problems: sub-goal planning, scene navigation, and object manipulation; and developed a model HiTUT (stands for Hierarchical Tasks via Unified Transformers) that addresses each sub-problem in a unified manner to learn a hierarchical task structure. On the ALFRED benchmark, HiTUT has achieved the best performance with a remarkably higher generalization ability. In the unseen environment, HiTUT achieves over 160% performance gain in success rate compared to the previous state of the art. The explicit representation of task structures also enables an in-depth understanding of the nature of the problem and the ability of the agent, which provides insight for future benchmark development and evaluation.
Benchmarking Complex Instruction-Following with Multiple Constraints Composition
Tsinghua University(清华大学);University of Electronic Science and Technology of China(电子科技大学);Zhipu AI(智谱AI);China University of Geosciences(中国地质大学);Central China Normal University(华中师范大学)
PRISM: Recovering Instruction Sets from Language Model Activations
Center for Cybersecurity Systems & Networks, Amrita Vishwa Vidyapeetham(阿姆里塔·维什瓦·维迪亚佩瑟姆网络安全系统与网络中心);Microsoft(微软);Ben-Gurion University of the Negev(内盖夫本-古里安大学)
Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigation
City University of Hong Kong, Dongguan Campus(香港城市大学东莞校区);The University of Melbourne(墨尔本大学);Tsinghua University(清华大学);Baidu Inc.(百度公司);University of Chinese Academy of Science(中国科学院大学);National University of Singapore(新加坡国立大学)
City Editing: Hierarchical Agentic Execution for Dependency-Aware Urban Geospatial Modification
University of Kansas, Lawrence, KS, USA(堪萨斯大学);Seoul National University, Seoul, South Korea(首尔国立大学);University of Florida, Gainesville, FL, USA(佛罗里达大学);NEC Laboratories America, Princeton, NJ, USA(NEC美国实验室);Clemson University, Clemson, SC, USA(克莱姆森大学);Arizona State University, Tempe, AZ, USA(亚利桑那州立大学)
OGAM: Connecting Systematic Testing to Runtime Assurance through Object-Grounded Attention Monitoring for VLA Policies
Carl von Ossietzky University of Oldenburg(卡尔·冯·奥西茨基大学奥尔登堡分校);Huadian Electric Power Research Institute(华电电力科学研究院);Chalmers University of Technology(查尔姆斯理工大学)
NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM
School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);School of Computing, National University of Singapore(新加坡国立大学计算机学院);College of Information Science and Technology, Beijing University of Chemical Technology(北京化工大学信息科学与技术学院)
Bridging Values and Behavior: A Hierarchical Framework for Proactive Embodied Agents
State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);School of Computer Science, Peking University(北京大学计算机学院);Tsinghua University(清华大学);Nat’l Eng. Research Center of Visual Technology, Peking University(北京大学视觉技术国家工程研究中心);Institute for AI, Peking University(北京大学人工智能研究院);State Key Laboratory of General Artificial Intelligence, Peking University(通用人工智能国家重点实验室,北京大学)
Towards Family-Grouped Hierarchical Federated Learning on Sub-5KB Models: A Feasibility Study of Privacy-Preserving ECG Monitoring for Ultra-Resource-Constrained Wearables
LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction
University of California Berkeley(加州大学伯克利分校);Norwegian University of Science and Technology(挪威科学与技术大学);Simon Fraser University(西蒙弗雷泽大学);Carnegie Mellon University(卡内基梅隆大学)