From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation
Department of Engineering Science, University of Oxford（工程科学系，牛津大学）；Department of Automation, Tsinghua University（自动化系，清华大学）；School of Information Science and Technology, Northwest University（信息科学与技术学院，西北大学）
Medical image segmentation remains challenging due to the high cost of pixel-level annotations for training. In the context of weak supervision, clinician gaze data captures regions of diagnostic interest; however, its sparsity limits its use for segmentation. In contrast, vision-language models (VLMs) provide semantic context through textual descriptions but lack the explanation precision required. Recognizing that neither source alone suffices, we propose a teacher-student framework that integrates both gaze and language supervision, leveraging their complementary strengths. Our key insight is that gaze data indicates where clinicians focus during diagnosis, while VLMs explain why those regions are significant. To implement this, the teacher model first learns from gaze points enhanced by VLM-generated descriptions of lesion morphology, establishing a foundation for guiding the student model. The teacher then directs the student through three strategies: (1) Multi-scale feature alignment to fuse visual cues with textual semantics; (2) Confidence-weighted consistency constraints to focus on reliable predictions; (3) Adaptive masking to limit error propagation in uncertain areas. Experiments on the Kvasir-SEG, NCI-ISBI, and ISIC datasets show that our method achieves Dice scores of 80.78%, 80.53%, and 84.22%, respectively-improving 3-5% over gaze baselines without increasing the annotation burden. By preserving correlations among predictions, gaze data, and lesion descriptions, our framework also maintains clinical interpretability. This work illustrates how integrating human visual attention with AI-generated semantic context can effectively overcome the limitations of individual weak supervision signals, thereby advancing the development of deployable, annotation-efficient medical AI systems. Code is available at: https://github.com/jingkunchen/FGI.
Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong（香港大学穆斯克特基金会数据科学研究所，香港大学）；Department of Electrical and Electronic Engineering, The University of Hong Kong（电气电子工程系，香港大学）
Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs
School of Computing, University of Georgia（佐治亚大学计算机学院）；Department of Computer Science and Engineering, University of Texas, Arlington（德克萨斯大学阿灵顿分校计算机科学与工程系）；Massachusetts General Hospital, Harvard Medical School（麻省总医院哈佛医学院）；Department of Biomedical Engineering, New Jersey Institute of Technology（新泽西理工学院生物医学工程系）
MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Data
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；DAMO Academy, Alibaba Group（阿里巴巴集团 DAMO 院）；Hupan Lab, Zhejiang Province（浙江省 Hupan 实验室）
Enjoying Information Dividend: Gaze Track-based Medical Weakly Supervised Segmentation
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（集成空天地海大数据应用技术国家工程实验室）；School of Computer Science and Engineering, Northwestern Polytechnical University（计算机科学与工程学院，西北工业大学）；Ningbo Institute of Northwestern Polytechnical University（西北工业大学宁波研究院）
Mining Gaze for Contrastive Learning toward Computer-Assisted Diagnosis
School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）
Eyes Tell the Truth: GazeVal Highlights Shortcomings of Generative AI in Medical Imaging
Northwestern University（西北大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Stanford University（斯坦福大学）；Université Sorbonne Paris Nord（巴黎北索邦大学）；Loyola University Chicago（芝加哥洛约拉大学）；DuPage Medical Group（杜佩奇医疗集团）；University of Pennsylvania（宾夕法尼亚大学）；Emory University（埃默里大学）
针对医学影像中合成数据评估与临床真实性脱节的问题，提出 GazeVal 框架，结合专家眼动追踪与放射学评估，实验发现 96.6% 的生成图像被识破，凸显生成式 AI 的不足。
Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；City University of Hong Kong（香港城市大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）；University of Georgia（佐治亚大学）；Massachusetts General Hospital（麻省总医院）
Gaze following aims to predict where a person is looking in a scene, by predicting the target location, or indicating that the target is located outside the image. Recent works detect the gaze target by training a heatmap regression task with a pixel-wise mean-square error (MSE) loss, while formulating the in/out prediction task as a binary classification task. This training formulation puts a strict, pixel-level constraint in higher resolution on the single annotation available in training, and does not consider annotation variance and the correlation between the two subtasks. To address thes…
University of Central Florida（中佛罗里达大学）；National Institutes of Health（美国国立卫生研究院）；Soochow University（苏州大学）
This study introduced a novel system, called Gaze2Segment, integrating biological and computer vision techniques to support radiologists' reading experience with an automatic image segmentation task. During diagnostic assessment of lung CT scans, the radiologists' gaze information were used to create a visual attention map. This map was then combined with a computer-derived saliency map, extracted from the gray-scale CT images. The visual attention map was used as an input for indicating roughly the location of a object of interest. With computer-derived saliency information, on the other hand…
Many recent machine learning approaches used in medical imaging are highly reliant on large amounts of image and ground truth data. In the context of object segmentation, pixel-wise annotations are extremely expensive to collect, especially in video and 3D volumes. To reduce this annotation burden, we propose a novel framework to allow annotators to simply observe the object to segment and record where they have looked at with a \$200 eye gaze tracker. Our method then estimates pixel-wise probabilities for the presence of the object throughout the sequence from which we train a classifier in s…
GEM: Context-Aware Gaze EstiMation with Visual Search Behavior Matching for Chest Radiograph
AI Research Center for Medical Image Analysis and Diagnosis, Shenzhen University（深圳大学医学影像分析与诊断AI研究中心）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；Computer Vision Institute, College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院计算机视觉研究所）；Department of Electrical Engineering, City University of Hong Kong（香港城市大学电气工程系）
Supporting Mitosis Detection AI Training with Inter-Observer Eye-Gaze Consistencies
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；UCLA David Geffen School of Medicine（加州大学洛杉矶分校大卫·格芬医学院）；Kansas University Medical Center（堪萨斯大学医学中心）
MedCLM: Learning to Localize and Reason via a CoT-Curriculum in Medical Vision-Language Models
A.I.MATICS Inc（A.I.MATICS公司）；Boston University（波士顿大学）；University of Southern California（南加州大学）；Heuron（Heuron公司）；MODULABS, Open Neural Networks Research Lab（MODULABS，开放神经网络研究实验室）
The University of Tokyo（东京大学）；The University of British Columbia（不列颠哥伦比亚大学）；RIKEN, AIP（理化学研究所革新智能综合研究中心）；University of Texas Arlington（德克萨斯大学阿灵顿分校）
The human gaze is a cost-efficient physiological data that reveals human underlying attentional patterns. The selective attention mechanism helps the cognition system focus on task-relevant visual clues by ignoring the presence of distractors. Thanks to this ability, human beings can efficiently learn from a very limited number of training samples. Inspired by this mechanism, we aim to leverage gaze for medical image analysis tasks with small training data. Our proposed framework includes a backbone encoder and a Selective Attention Network (SAN) that simulates the underlying attention. The SA…
EG-SpikeFormer: Eye-Gaze Guided Transformer on Spiking Neural Networks for Medical Image Analysis
School of Computing, University of Georgia（佐治亚大学计算学院）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Georgia（佐治亚大学）；Department of Physics and Astronomy（物理与天文学系）
Shanghai Jiao Tong University（上海交通大学）；University of Georgia（佐治亚大学）；ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）
When deep neural network (DNN) was first introduced to the medical image analysis community, researchers were impressed by its performance. However, it is evident now that a large number of manually labeled data is often a must to train a properly functioning DNN. This demand for supervision data and labels is a major bottleneck in current medical image analysis, since collecting a large number of annotations from experienced experts can be time-consuming and expensive. In this paper, we demonstrate that the eye movement of radiologists reading medical images can be a new form of supervision t…
We propose a novel method on refining cross-person gaze prediction task with eye/face images only by explicitly modelling the person-specific differences. Specifically, we first assume that we can obtain some initial gaze prediction results with existing method, which we refer to as InitNet, and then introduce three modules, the Validity Module (VM), Self-Calibration (SC) and Person-specific Transform (PT)) Module. By predicting the reliability of current eye/face images, our VM is able to identify invalid samples, e.g. eye blinking images, and reduce their effects in our modelling process. Ou…
The University of Adelaide（阿德莱德大学）；Northwestern Polytechnical University（西北工业大学）
Convolutional neural networks (CNNs) have been the de facto standard for nowadays 3D medical image segmentation. The convolutional operations used in these networks, however, inevitably have limitations in modeling the long-range dependency due to their inductive bias of locality and weight sharing. Although Transformer was born to address this issue, it suffers from extreme computational and spatial complexities in processing high-resolution 3D feature maps. In this paper, we propose a novel framework that efficiently bridges a {\bf Co}nvolutional neural network and a {\bf Tr}ansformer {\bf (…
When Eye-Tracking Meets Machine Learning: A Systematic Review on Applications in Medical Image Analysis
Macquarie University（麦考瑞大学）；Macquarie Medical School（麦考瑞医学院）；Faculty of Medicine, Health and Human Sciences（医学与健康科学学院）；School of Computing（计算机学院）；Faculty of Science and Engineering（科学与工程学院）；Centre for Health Informatics（健康信息学中心）；Australian Institute of Health Innovation（澳大利亚健康创新研究院）
The way humans attend to, process and classify a given image has the potential to vastly benefit the performance of deep learning models. Exploiting where humans are focusing can rectify models when they are deviating from essential features for correct decisions. To validate that human attention contains valuable information for decision-making processes such as fine-grained classification, we compare human attention and model explanations in discovering important features. Towards this goal, we collect human gaze data for the fine-grained classification dataset CUB and build a dataset named …
Politecnico di Torino（都灵理工大学）；Istituto Italiano di Tecnologia（意大利理工学院）
Deep learning has thrived by training on large-scale datasets. However, in many applications, as for medical image diagnosis, getting massive amount of data is still prohibitive due to privacy, lack of acquisition homogeneity and annotation cost. In this scenario, transfer learning from natural image collections is a standard practice that attempts to tackle shape, texture and color discrepancies all at once through pretrained model fine-tuning. In this work, we propose to disentangle those challenges and design a dedicated network module that focuses on color adaptation. We combine learning f…
Northwestern Polytechnical University（西北工业大学）；University of Georgia（佐治亚大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Athens Academy（雅典学院）；Shanghai Jiao Tong University（上海交通大学）；ShanghaiTech University（上海科技大学）
Learning harmful shortcuts such as spurious correlations and biases prevents deep neural networks from learning the meaningful and useful representations, thus jeopardizing the generalizability and interpretability of the learned representation. The situation becomes even more serious in medical imaging, where the clinical data (e.g., MR images with pathology) are limited and scarce while the reliability, generalizability and transparency of the learned model are highly required. To address this problem, we propose to infuse human experts' intelligence and domain knowledge into the training of…
Anatomical landmarks are a crucial prerequisite for many medical imaging tasks. Usually, the set of landmarks for a given task is predefined by experts. The landmark locations for a given image are then annotated manually or via machine learning methods trained on manual annotations. In this paper, in contrast, we present a method to automatically discover and localize anatomical landmarks in medical images. Specifically, we consider landmarks that attract the visual attention of humans, which we term visually salient landmarks. We illustrate the method for fetal neurosonographic images. First…
Johns Hopkins University（约翰斯·霍普金斯大学）；NVIDIA（英伟达）；Arizona State University（亚利桑那州立大学）
Semantic segmentation of 3D medical images is a challenging task due to the high variability of the shape and pattern of objects (such as organs or tumors). Given the recent success of deep learning in medical image segmentation, Neural Architecture Search (NAS) has been introduced to find high-performance 3D segmentation network architectures. However, because of the massive computational requirements of 3D data and the discrete optimization nature of architecture search, previous NAS methods require a long search time or necessary continuous relaxation, and commonly lead to sub-optimal netwo…
