Evaluating Vision Foundation Models for Pixel and Object Classification in Microscopy
Georg-August-University Göttingen, Institute of Computer Science Department of Tissue Dynamics;Regeneration, Max Planck Institute for Multidisciplinary Sciences, Göttingen Georg-August-University Göttingen, Faculty of Biology;Psychology CAIMed - Lower Saxony Center for AI \& Causal Methods in Medicine, Göttingen Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen
Carolin Teuber, Anwai Archit, Tobias Boothe, Peter Ditte, Jochen Rink, Constantin Pape
Deep learning underlies most modern approaches and tools in computer vision, including biomedical imaging. However, for interactive semantic segmentation (often called pixel classification in this context) and interactive object-level classification (object classification), feature-based shallow learning remains widely used. This is due to the diversity of data in this domain, the lack of large pretraining datasets, and the need for computational and label efficiency. In contrast, state-of-the-art tools for many other vision tasks in microscopy - most notably cellular instance segmentation - already rely on deep learning and have recently benefited substantially from vision foundation models (VFMs), particularly SAM. Here, we investigate whether VFMs can also improve pixel and object classification compared to current approaches. To this end, we evaluate several VFMs, including general-purpose models (SAM, SAM2, SAM3, DINOv3) and domain-specific ones ($μ$SAM, PathoSAM, KRONOS), in combination with shallow learning and attentive probing on five diverse and challenging datasets. Our results demonstrate consistent improvements over hand-crafted features and provide a clear pathway toward practical improvements. Furthermore, our study establishes a benchmark for VFMs in microscopy and informs future developments in this area.
Segmentation in large-scale cellular electron microscopy with deep learning: A literature survey
Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University Groningen(格罗宁根大学伯努利数学、计算机科学与人工智能研究所);University Groningen(格罗宁根大学);University Medical Center Groningen(格罗宁根大学医学中心)
High-speed real-time single-pixel microscopy based on Fourier sampling
Tsinghua National Laboratory for Information Science and Technology (TNList), Department of Electronic Engineering, Tsinghua University(清华大学信息科学与技术国家实验室,电子工程系,清华大学);Department of Biomedical Engineering, School of Medicine, Collaborative Innovation Center for Diagnosis and Treatment of Infectious Diseases, Tsinghua University(清华大学生物医学工程系,医学院,传染病诊治协同创新中心,清华大学)
Max Planck Institute of Molecular Cell Biology and Genetics(马克斯·普朗克分子细胞生物学与遗传学研究所);Center for Systems Biology Dresden(德累斯顿系统生物学中心);Technical University Dresden(德累斯顿工业大学)
Evaluating the visualization of what a Deep Neural Network has learned
Fraunhofer Heinrich Hertz Institute(弗劳恩霍夫海因里希·赫兹研究所);Singapore University of Technology and Design (SUTD)(新加坡科技设计大学);Berlin Institute of Technology (TU Berlin)(柏林工业大学);Korea University(高丽大学)
Combining nonparametric spatial context priors with nonparametric shape priors for dendritic spine segmentation in 2-photon microscopy images
Sabanci University(萨班哲大学);ARM Ltd.(ARM有限公司);University of Zurich(苏黎世大学);University of Utah(犹他大学);Izmir University of Economics(伊兹密尔经济大学);University of Rochester(罗切斯特大学)
From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models
University of South Florida Tampa USA;University of Alabama Tuscaloosa USA;University of Michigan Ann Arbor USA;Texas Tech University Lubbock USA;Texas A \& M University College Station USA;University of Pittsburgh Pittsburgh USA;University of South Florida;University of Alabama;University of Michigan;Texas Tech University;Texas A \& M University;University of Pittsburgh
Beyond the Pixel-Wise Loss for Topology-Aware Delineation
Computer Vision Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院计算机视觉实验室);ARTORG Center for Biomedical Engineering Research, University of Bern(伯尔尼大学ARTORG生物医学工程研究中心)
From Pixel to Coding: Evaluating the Figure Reproduction Capabilities of MLLMs
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University(上海交通大学图像通信与信息处理研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Macao Polytechnic University(澳门理工大学)
Learned, uncertainty-driven adaptive acquisition for photon-efficient scanning microscopy
Fu Foundation School of Engineering and Applied Science, Columbia University in the City of New York(纽约市哥伦比亚大学傅氏工程与应用科学学院);University of California, Berkeley(加利福尼亚大学伯克利分校);Department of Biological Engineering, Massachusetts Institute of Technology(麻省理工学院生物工程系)
Atomic Depth Estimation From Noisy Electron Microscopy Data Via Deep Learning
Courant Institute of Mathematical Sciences, New York University(纽约大学数学科学学院);School for Engineering of Matter, Transport, and Energy, Arizona State University(亚利桑那州立大学工程科学学院);Center for Data Science, New York University(纽约大学数据科学中心)
Deep image prior for undersampling high-speed photoacoustic microscopy
Duke University(杜克大学);Tsinghua University(清华大学);Brigham and Women’s Hospital(布列根和妇女医院);Harvard Medical School(哈佛医学院);University of Southern California(南加州大学)