多模态大模型

Mediating Modes of Thought: LLM's for design scripting

Moritz Rietschel, Fang Guo, Kyle Steinfeld

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Published at ACADIA 2024

BIP3D: Bridging 2D Images and 3D Perception for Embodied Intelligence

Xuewu Lin, Tianwei Lin, Lichao Huang, Hongyu Xie, Zhizhong Su

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

Neurosymbolic Graph Enrichment for Grounded World Models

Stefano De Giorgis, Aldo Gangemi, Alessandro Russo

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

A Review of Large Language Models and Autonomous Agents in Chemistry

Mayk Caldas Ramos, Christopher J. Collison, Andrew D. White

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CL、cs.AI

To Ask or Not to Ask? Detecting Absence of Information in Vision and Language Navigation

Savitha Sam Abraham, Sourav Garg, Feras Dayoub

专题命中 多模态Agent ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted at WACV 2025

Unlocking the Archives: Using Large Language Models to Transcribe Handwritten Historical Documents

Mark Humphries, Lianne C. Leddy, Quinn Downton, Meredith Legace, John McConnell, Isabella Murray, Elizabeth Spence

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments 29 Pages, 11 Tables, 2 Figures

MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making

Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, Hae Won Park

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CL、cs.AI

Reliable, Routable, and Reproducible: Collection of Pedestrian Pathways at Statewide Scale

Yuxiang Zhang, Bill Howe, Anat Caspi

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments arXiv admin note: text overlap with arXiv:2303.02323

Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding

Yue Fan, Lei Ding, Ching-Chen Kuo, Shan Jiang, Yang Zhao, Xinze Guan, Jie Yang, Yi Zhang, Xin Eric Wang

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL

OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning

Xiaoqiang Wang, Bang Liu

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Work in progress

Details Make a Difference: Object State-Sensitive Neurorobotic Task Planning

Xiaowen Sun, Xufeng Zhao, Jae Hee Lee, Wenhao Lu, Matthias Kerzel, Stefan Wermter

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments ICANN24, Switzerland

Instruction-Guided Visual Masking

Jinliang Zheng, Jianxiong Li, Sijie Cheng, Yinan Zheng, Jiaming Li, Jihao Liu, Yu Liu, Jingjing Liu, Xianyuan Zhan

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024

TOP-Nav: Legged Navigation Integrating Terrain, Obstacle and Proprioception Estimation

Junli Ren, Yikai Liu, Yingru Dai, Junfeng Long, Guijin Wang

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Published on CoRL 2024

Language Agents as Optimizable Graphs

Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, Jürgen Schmidhuber

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Project Website: https://gptswarm.org ; Github Repo: https://github.com/metauto-ai/gptswarm . In Forty-first International Conference on Machine Learning (2024)

Learning to Model the World with Language

Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, Anca Dragan

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments ICML 2024. Website: https://dynalang.github.io/

UMass-BioNLP at MEDIQA-M3G 2024: DermPrompt -- A Systematic Exploration of Prompt Engineering with GPT-4V for Dermatological Diagnosis

Parth Vashisht, Abhilasha Lodha, Mukta Maddipatla, Zonghai Yao, Avijit Mitra, Zhichao Yang, Junda Wang, Sunjae Kwon, Hong Yu

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Accepted at NAACL-ClinicalNLP workshop 2024

Evaluating Tool-Augmented Agents in Remote Sensing Platforms

Simranjit Singh, Michael Fore, Dimitrios Stamoulis

专题命中 多模态Agent ：image-text(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2024 Machine Learning for Remote Sensing (ML4RS) Workshop

The Essential Role of Causality in Foundation World Models for Embodied AI

Tarun Gupta, Wenbo Gong, Chao Ma, Nick Pawlowski, Agrin Hilmkil, Meyer Scetbon, Marc Rigter, Ade Famoti, Ashley Juan Llorens, Jianfeng Gao, Stefan Bauer, Danica Kragic, Bernhard Schölkopf, Cheng Zhang

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CL、cs.AI

GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilots

Simranjit Singh, Michael Fore, Dimitrios Stamoulis

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Earthvision 2024, CVPR Workshop

Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model

Zhonghan Zhao, Ke Ma, Wenhao Chai, Xuan Wang, Kewei Chen, Dongxu Guo, Yanting Zhang, Hongwei Wang, Gaoang Wang

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments arXiv admin note: text overlap with arXiv:2403.08282

Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning

Jaewoo Jeong, Daehee Park, Kuk-Jin Yoon

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 2024 CVPR Highlight

Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs

Yusuke Mikami, Andrew Melnik, Jun Miura, Ville Hautamäki

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments 8 pages, 2 figures

Surfer: Progressive Reasoning with World Models for Robotic Manipulation

Pengzhen Ren, Kaidong Zhang, Hetao Zheng, Zixuan Li, Yuhang Wen, Fengda Zhu, Mas Ma, Xiaodan Liang

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

ByteComposer: a Human-like Melody Composition Method based on Language Model Agent

Xia Liang, Xingjian Du, Jiaju Lin, Pei Zou, Yuan Wan, Bilei Zhu

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.AI、eess.AS

Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning

Tan Zhi-Xuan, Lance Ying, Vikash Mansinghka, Joshua B. Tenenbaum

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Accepted to AAMAS 2024. 8 pages (excl. references), 5 figures/tables. (Appendix: 8 pages, 8 figures/tables). Code available at: https://github.com/probcomp/CLIPS.jl

ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments

Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, Liang Wang

专题命中 多模态Agent ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Project page: https://github.com/MarSaKi/ETPNav

Morphological Profiling for Drug Discovery in the Era of Deep Learning

Qiaosi Tang, Ranjala Ratnayake, Gustavo Seabra, Zhe Jiang, Ruogu Fang, Lina Cui, Yousong Ding, Tamer Kahveci, Jiang Bian, Chenglong Li, Hendrik Luesch, Yanjun Li

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 44 pages, 5 figure, 5 tables

LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following

Cheng-Fu Yang, Yen-Chun Chen, Jianwei Yang, Xiyang Dai, Lu Yuan, Yu-Chiang Frank Wang, Kai-Wei Chang

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2023

MotionLM: Multi-Agent Motion Forecasting as Language Modeling

Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S. Refaat, Rami Al-Rfou, Benjamin Sapp

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments To appear at the International Conference on Computer Vision (ICCV) 2023

Rating Prediction in Conversational Task Assistants with Behavioral and Conversational-Flow Features

Rafael Ferreira, David Semedo, João Magalhães

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI