Guiding users through complex procedural plans is an inherently multimodal task in which having visually illustrated plan steps is crucial to deliver an effective plan guidance. However, existing works on plan-following language models (LMs) often are not capable of multimodal input and output. In this work, we present MM-PlanLLM, the first multimodal LLM designed to assist users in executing instructional tasks by leveraging both textual plans and visual information. Specifically, we bring cross-modality through two key tasks: Conversational Video Moment Retrieval, where the model retrieves relevant step-video segments based on user queries, and Visually-Informed Step Generation, where the model generates the next step in a plan, conditioned on an image of the user's current progress. MM-PlanLLM is trained using a novel multitask-multistage approach, designed to gradually expose the model to multimodal instructional-plans semantic layers, achieving strong performance on both multimodal and textual dialogue in a plan-grounded setting. Furthermore, we show that the model delivers cross-modal temporal and plan-structure representations aligned between textual plan steps and instructional video moments.
Retrieval-Augmented and Knowledge-Grounded Language Models for Faithful Clinical Medicine
University of Oxford（牛津大学）；Peking University（北京大学）；Yale University（耶鲁大学）；Tencent（腾讯）；Oxford-Suzhou Centre for Advanced Research（牛津-苏州高等研究院）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）
Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAs
Texas A&M University（德克萨斯农工大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Maryland（马里兰大学）；University of California, Riverside（加利福尼亚大学河滨分校）；University of Pittsburgh（匹兹堡大学）
