The task of scientific Natural Language Inference (NLI) involves predicting the semantic relation between two sentences extracted from research articles. This task was recently proposed along with a new dataset called SciNLI derived from papers published in the computational linguistics domain. In this paper, we aim to introduce diversity in the scientific NLI task and present MSciNLI, a dataset containing 132,320 sentence pairs extracted from five new scientific domains. The availability of multiple domains makes it possible to study domain shift for scientific NLI. We establish strong baselines on MSciNLI by fine-tuning Pre-trained Language Models (PLMs) and prompting Large Language Models (LLMs). The highest Macro F1 scores of PLM and LLM baselines are 77.21% and 51.77%, respectively, illustrating that MSciNLI is challenging for both types of models. Furthermore, we show that domain shift degrades the performance of scientific NLI models which demonstrates the diverse characteristics of different domains in our dataset. Finally, we use both scientific NLI datasets in an intermediate task transfer learning setting and show that they can improve the performance of downstream tasks in the scientific domain. We make our dataset and code available on Github.
SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models
ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University（浙江大学杭州国际科创中心）；The Polytechnic Institute, Zhejiang University（浙江大学理工学院）；ZJU-UIUC, Zhejiang University（浙江大学伊利诺伊大学厄巴纳-香槟联合学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；AntGroup（蚂蚁集团）
Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI) Technical University of Darmstadt（普遍知识处理实验室（UKP实验室）计算机科学系海德堡人工智能中心（hessian.AI）达姆施塔特技术大学）；Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)（康拉德·祖斯卓越学习与智能系统学校（ELIZA））
Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning
Department of Computer Science, Yale University；Department of Molecular \& Cellular Biology, Harvard University；Allen Institute for AI；Department of Computer Science, Northwestern University
STEP: Scientific Time-Series Encoder Pretraining via Cross-Domain Distillation
Shanghai Aritificial Intelligence Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；Harbin Institute of Technology（哈尔滨工业大学）；Sichuan University（四川大学）；University of Cambridge（剑桥大学）
LSR-Ben: A Logical and Scientific Reasoning Benchmark for Evaluating Process Reward Models
Research Center for Social Computing（社会计算研究中心）；Interactive Robotics, Harbin Institute of Technology, China（交互机器人学，哈尔滨工业大学，中国）；Beijing Academy of Artificial Intelligence, Beijing, China（北京人工智能研究院，北京，中国）
xVal: A Continuous Numerical Tokenization for Scientific Language Models
Flatiron Institute（弗莱蒂伦研究所）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；University of Cambridge（剑桥大学）；Université Paris-Saclay（巴黎萨克雷大学）；Université Paris Cité（巴黎西岱大学）；CEA（法国原子能和替代能源委员会）；CNRS（法国国家科学研究中心）；University of Colorado Boulder（科罗拉多大学博尔德分校）；New York University（纽约大学）；Prescient Design；Genentech（基因泰克公司）；Princeton University（普林斯顿大学）
