Learning To Guide Human Decision Makers With Vision-Language Models
DI, University of Pisa(比萨大学DI);DISI, University of Trento(特伦托大学DISI);CIMEC, University of Trento(特伦托大学CIMEC);Rajib Pal Bankura Sammilani Medical College(拉吉布·帕尔班克尔医学学院)
Debodeep Banerjee, Stefano Teso, Burcu Sayin, Andrea Passerini
There is growing interest in AI systems that support human decision-making in high-stakes domains (e.g., medical diagnosis) to improve decision quality and reduce cognitive load. Mainstream approaches pair human experts with a machine-learning model, offloading low-risk decisions to the model so that experts can focus on cases that require their judgment.
This separation of responsibilities setup, however, is inadequate for high-stakes scenarios. The expert may end up over-relying on the machine's decisions due to anchoring bias, thus losing the human oversight that is increasingly being required by regulatory agencies to ensure trustworthy AI. On the other hand, the expert is left entirely unassisted on the (typically hardest) decisions on which the model abstained.
As a remedy, we introduce learning to guide (LTG), an alternative framework in which -- rather than taking control from the human expert -- the machine provides guidance useful for decision making, and the human is entirely responsible for coming up with a decision.
In order to ensure guidance is interpretable and task-specific, we develop SLOG, an approach for turning any vision-language model into a capable generator of textual guidance by leveraging a modicum of human feedback.
Our empirical evaluation highlights the promise of SLOG on both on a synthetic dataset and a challenging, real-world medical diagnosis task.
Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)
iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understanding
School of Computer Science(计算机科学学院);Technology, Huazhong University of Science(科技,华中科技大学);Li Auto Inc.(Li汽车公司);Institute of AI for Industries, Chinese Academy of Sciences(产业人工智能研究所,中国科学院)
A Pilot Study on Curator-Guided Multilingual Art Description for Blind and Low-Vision Audiences with Small Vision-Language Models
Technical University of Munich(慕尼黑技术大学);Foundation for Research and Technology -- Hellas(希腊研究与技术基金会);National University of Science and Technology Politehnica Bucharest(布加勒斯特政治技术科学与技术国家大学)
Landmark-Guided Knowledge for Vision-and-Language Navigation
School of Computer Science and Technology, Xinjiang University, Urumqi, China(计算机科学与技术学院,新疆大学,乌鲁木齐,中国);No. 59 Middle School of Urumqi, Urumqi, China(乌鲁木齐第五十九中学,乌鲁木齐,中国)
On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstruction
Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital(麻省总医院Athinoula A. Martinos生物医学成像中心);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院)