Time roots in applying language models for biomedical applications: models are trained on historical data and will be deployed for new or future data, which may vary from training data. While increasing biomedical tasks have employed state-of-the-art language models, there are very few studies have examined temporal effects on biomedical models when data usually shifts across development and deployment. This study fills the gap by statistically probing relations between language model performance and data shifts across three biomedical tasks. We deploy diverse metrics to evaluate model performance, distance methods to measure data drifts, and statistical methods to quantify temporal effects on biomedical language models. Our study shows that time matters for deploying biomedical language models, while the degree of performance degradation varies by biomedical tasks and statistical quantification approaches. We believe this study can establish a solid benchmark to evaluate and assess temporal effects on deploying biomedical language models.
Mapping Biomedical Ontology Terms to IDs: Effect of Domain Prevalence on Prediction Accuracy
Computer Science Department(计算机科学系);Missouri State University(密苏里州立大学);Kummer Institute Center for AI and AS(Kummer研究所人工智能与自动系统中心);Missouri University of Science & Technology(密苏里科技大学);Cooperative Engineering Program(合作工程计划)
Developing a general-purpose clinical language inference model from a large corpus of clinical notes
Bakar Computational Health Sciences Institute, University of California, San Francisco(加州大学旧金山分校巴卡尔计算健康科学研究所);Center for Data-driven Insights and Innovation, University of California, Office of the President(加州大学校长办公室数据驱动洞察与创新中心);University of California, San Francisco(加州大学旧金山分校)
Benchmarking large language models for biomedical natural language processing applications and recommendations
Yale University(耶鲁大学);National Institutes of Health(美国国立卫生研究院);University of Texas Health Science at Houston(休斯顿德克萨斯大学健康科学中心);University of Minnesota(明尼苏达大学)
BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing
Manning College of Information and Computer Sciences, University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校曼宁信息与计算机科学学院);University of Massachusetts Medical School(马萨诸塞大学医学院);Miner School of Computer and Information Sciences, University of Massachusetts Lowell(马萨诸塞大学洛厄尔分校迈纳计算机与信息科学学院);VA Bedford Health Care(贝德福德退伍军人保健中心)