Scientific literature understanding is crucial for extracting targeted information and garnering insights, thereby significantly advancing scientific discovery. Despite the remarkable success of Large Language Models (LLMs), they face challenges in scientific literature understanding, primarily due to (1) a lack of scientific knowledge and (2) unfamiliarity with specialized scientific tasks.
To develop an LLM specialized in scientific literature understanding, we propose a hybrid strategy that integrates continual pre-training (CPT) and supervised fine-tuning (SFT), to simultaneously infuse scientific domain knowledge and enhance instruction-following capabilities for domain-specific tasks.cIn this process, we identify two key challenges: (1) constructing high-quality CPT corpora, and (2) generating diverse SFT instructions. We address these challenges through a meticulous pipeline, including PDF text extraction, parsing content error correction, quality filtering, and synthetic instruction creation. Applying this strategy, we present a suite of LLMs: SciLitLLM, specialized in scientific literature understanding. These models demonstrate promising performance on scientific literature understanding benchmarks.
Our contributions are threefold: (1) We present an effective framework that integrates CPT and SFT to adapt LLMs to scientific literature understanding, which can also be easily adapted to other domains. (2) We propose an LLM-based synthesis method to generate diverse and high-quality scientific instructions, resulting in a new instruction set -- SciLitIns -- for supervised fine-tuning in less-represented scientific domains. (3) SciLitLLM achieves promising performance improvements on scientific literature understanding benchmarks.
WisPerMed at BioLaySumm: Adapting Autoregressive Large Language Models for Lay Summarization of Scientific Articles
University Hospital Essen（埃森大学医院）；University of Applied Sciences and Arts Dortmund（多特蒙德应用科学与艺术大学）；Institute for Medical Informatics, Biometry and Epidemiology (IMIBE)（医学信息学、生物统计学与流行病学研究所（IMIBE））
Adapt Semantics, Not Structure: Few-Instance Schema Calibration for Scientific PDF Extraction
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of Cognitive Intelligence（认知智能全国重点实验室）；CCCC Second Highway Consultants Co., Ltd.（中交第二公路勘察设计研究院有限公司）
STEP: Scientific Time-Series Encoder Pretraining via Cross-Domain Distillation
Shanghai Aritificial Intelligence Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；Harbin Institute of Technology（哈尔滨工业大学）；Sichuan University（四川大学）；University of Cambridge（剑桥大学）
AHAM: Adapt, Help, Ask, Model -- Harvesting LLMs for literature mining
Jožef Stefan Institute（约瑟夫·斯特凡研究所）；International Postgraduate School Jožef Stefan（约瑟夫·斯特凡国际研究生院）；University of Nova Gorica（新戈里察大学）；Temida d.o.o（Temida有限责任公司）；University of Ljubljana（卢布尔雅那大学）
LLM-to-Flow Distillation: Teaching Natural Language Goals to Scientific Generators
ETH Zurich（苏黎世联邦理工学院）；ETH AI Center（苏黎世联邦理工学院人工智能中心）；Technische Universität Wien（维也纳技术大学）；AITHYRA；University of Pennsylvania（宾夕法尼亚大学）；University of Oxford（牛津大学）
PINN Balls: Scaling Second-Order Methods for PINNs with Domain Decomposition and Adaptive Sampling
BMW AG, Digital Campus Munich（宝马集团慕尼黑数字园区）；University of Göttingen（哥廷根大学）；Basque Center for Applied Mathematics（巴斯克应用数学中心）；Campus Institute Data Science（数据科学校园研究所）；University of the Basque Country（巴斯克大学）；Intelligent Systems Group（智能系统研究组）；IKERBASQUE, Basque Foundation for Science（巴斯克科学基金会）；Swansea University（斯旺西大学）
