EAA: Automating materials characterization with vision language model agents
Argonne National Laboratory(阿贡国家实验室);Advanced Photon Source(先进光子源);Data Science and Learning Division(数据科学与学习 division);Department of Radiation Oncology(放射肿瘤学系);Northwestern University(西北大学)
Ming Du, Yanqi Luo, Srutarshi Banerjee, Michael Wojcik, Jelena Popovic, Mathew J. Cherukara
We present Experiment Automation Agents (EAA), a vision-language-model-driven agentic system designed to automate complex experimental microscopy workflows. EAA integrates multimodal reasoning, tool-augmented action, and optional long-term memory to support both autonomous procedures and interactive user-guided measurements. Built on a flexible task-manager architecture, the system enables workflows ranging from fully agent-driven automation to logic-defined routines that embed localized LLM queries. EAA further provides a modern tool ecosystem with two-way compatibility for Model Context Protocol (MCP), allowing instrument-control tools to be consumed or served across applications. We demonstrate EAA at an imaging beamline at the Advanced Photon Source, including automated zone plate focusing, natural language-described feature search, and interactive data acquisition. These results illustrate how vision-capable agents can enhance beamline efficiency, reduce operational burden, and lower the expertise barrier for users.
ZeroGUI: Automating Online GUI Learning at Zero Human Cost
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong(香港中文大学)
Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs
ICG, TU Graz(格拉茨工业大学ICG研究所);CDL-EML(CDL-EML中心);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);JKU(约翰内斯开普勒大学);IBM Research(IBM研究院);Weizmann Institute of Science(魏茨曼科学研究所);University of Bonn(波恩大学)
Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning
Bioengineering Department and Imperial-X(生物工程系和Imperial-X);Imperial College London(帝国理工学院伦敦分校);College of physics and information engineering, Fuzhou University(福州大学物理与信息工程学院);Department of Surgery and Cancer, Imperial College London(外科与癌症部门,帝国理工学院伦敦分校);National Heart and Lung Institute, Imperial College London(国家心脏和肺研究所,帝国理工学院伦敦分校);Cardiovascular Research Centre, Royal Brompton Hospital(心血管研究中心,皇家布里顿医院);School of Biomedical Engineering & Imaging Sciences, King’s College London(生物医学工程与成像科学学院,国王学院伦敦分校)
Towards Reliable LLM-Driven Fuzz Testing: Vision and Road Ahead
Institute of Information Engineering, CAS(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);University of Sydney(悉尼大学);Singapore Management University(新加坡管理大学)
SKT: Integrating State-Aware Keypoint Trajectories with Vision-Language Models for Robotic Garment Manipulation
Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology(香港科技大学);National University of Singapore(新加坡国立大学);Yuandao AI(元岛人工智能);Shanghai AI Lab(上海人工智能实验室)
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
Shanghai AI Laboratory(上海人工智能实验室);The University of Hong Kong(香港大学);Johns Hopkins University(约翰霍普金斯大学);Shanghai Jiao Tong University(上海交通大学);University of Oxford(牛津大学);Hong Kong University of Science and Technology(香港科技大学)
Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
The University of Hong Kong(香港大学);Shanghai Jiao Tong University(上海交通大学);Google Cloud AI Research(谷歌云人工智能研究院);Google DeepMind(谷歌DeepMind);Salesforce Research(salesforce研究院);Yale University(耶鲁大学);Sea AI Lab(Sea人工智能实验室);University of Waterloo(滑铁卢大学)
PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting
University of Wisconsin–Madison Department of Computer Sciences(威斯康星大学麦迪逊分校计算机科学系);University of Wisconsin–Madison Department Radiology(威斯康星大学麦迪逊分校放射学系);Microsoft(微软公司)