Closing the gap between open-source and commercial large language models for medical evidence summarization
Columbia University(哥伦比亚大学);National Library of Medicine(美国国家医学图书馆);National Institutes of Health(美国国立卫生研究院);Weill Cornell Medicine(威尔康奈尔医学院);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Icahn School of Medicine at Mount Sinai(西奈山伊坎医学院)
Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina R. Idnay, Yiming Luo, Elizabeth Park, Jordan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas Campion… 展开作者
Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina R. Idnay, Yiming Luo, Elizabeth Park, Jordan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas Campion, Zhiyong Lu, Chunhua Weng, Yifan Peng
Large language models (LLMs) hold great promise in summarizing medical evidence. Most recent studies focus on the application of proprietary LLMs. Using proprietary LLMs introduces multiple risk factors, including a lack of transparency and vendor dependency. While open-source LLMs allow better transparency and customization, their performance falls short compared to proprietary ones. In this study, we investigated to what extent fine-tuning open-source LLMs can further improve their performance in summarizing medical evidence. Utilizing a benchmark dataset, MedReview, consisting of 8,161 pairs of systematic reviews and summaries, we fine-tuned three broadly-used, open-sourced LLMs, namely PRIMERA, LongT5, and Llama-2. Overall, the fine-tuned LLMs obtained an increase of 9.89 in ROUGE-L (95% confidence interval: 8.94-10.81), 13.21 in METEOR score (95% confidence interval: 12.05-14.37), and 15.82 in CHRF score (95% confidence interval: 13.89-16.44). The performance of fine-tuned LongT5 is close to GPT-3.5 with zero-shot settings. Furthermore, smaller fine-tuned models sometimes even demonstrated superior performance compared to larger zero-shot models. The above trends of improvement were also manifested in both human and GPT4-simulated evaluations. Our results can be applied to guide model selection for tasks demanding particular domain knowledge, such as medical evidence summarization.
Context Matters: Comparison of commercial large language tools in veterinary medicine
ANI.ML Research, ANI.ML Health Inc.;University of Guelph(圭尔夫大学);Toronto Animal Cancer Centre(多伦多动物癌症中心);The Hospital for Sick Children(病童医院);Anderson University(安德森大学)
Context-Aware Pseudo-Label Scoring for Zero-Shot Video Summarization
Nanyang Technological University(南洋理工大学);Guilin University of Electronic Technology(桂林电子科技大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院)
TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction
Meetyou AI Lab(美柚人工智能实验室);Xiamen Key Laboratory of Women’s Internet Health Management(厦门市女性互联网健康管理重点实验室);Osaka University(大阪大学);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局)
Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluation
School of Computing, Informatics Institute of Technology(计算学院,信息学院技术研究所);School of Mathematics and Computer Science, Swansea University(数学与计算机科学学院,萨默塞特大学);R&D, Zame AI(Zame AI研发部)
LitSumm: Large language models for literature summarisation of non-coding RNAs
European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI)(欧洲分子生物学实验室欧洲生物信息研究所);The University of Manchester(曼彻斯特大学);Riboscope Ltd(Riboscope有限公司)
Prompt-tuning for Clickbait Detection via Text Summarization
School of Information Engineering, Yangzhou University(扬州大学信息工程学院);School of Electronics and Information Engineering, Anhui Jianzhu University(安徽建筑大学电子与信息工程学院)