Between 2021 and 2025, the SciCap project grew from a small seed-funded idea at The Pennsylvania State University (Penn State) into one of the central efforts shaping the scientific figure-captioning landscape. Supported by a Penn State seed grant, Adobe, and the Alfred P. Sloan Foundation, what began as our attempt to test whether domain-specific training, which was successful in text models like SciBERT, could also work for figure captions expanded into a multi-institution collaboration. Over these five years, we curated, released, and continually updated a large collection of figure-caption pairs from arXiv papers, conducted extensive automatic and human evaluations on both generated and author-written captions, navigated the rapid rise of large language models (LLMs), launched annual challenges, and built interactive systems that help scientists write better captions. In this piece, we look back at the first five years of SciCap and summarize the key technical and methodological lessons we learned. We then outline five major unsolved challenges and propose directions for the next phase of research in scientific figure captioning.
Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023
Pennsylvania State University(宾夕法尼亚州立大学);National Tsing Hua University(国立清华大学);AllSci(奥尔赛公司);MetaMetrics Inc.(美塔麦特里克公司);Adobe Research(奥多比研究院);Institute of Information Science, Academia Sinica(中央研究院资讯科学研究所)
Department of Biomedical Engineering, Faculty of Engineering, Mahidol University(生物医学工程系,工程学院,玛希多大学);Faculty of Medicine Ramathibodi Hospital, Mahidol University(拉玛蒂波迪医院医学院,玛希多大学)
MSEarth: A Multimodal Scientific Dataset and Benchmark for Phenomena Uncovering in Earth Science
The Hong Kong Polytechnic University(香港理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学)