As large language models become increasingly capable of generating, and assisting with, scientific writing, there is a need for greater transparency and disclosure around the use of AI in research. Consequently, a veritable subfield of research and a number of commercial services aim to flag AI-generated content automatically. However, existing detectors which rely only on text extracted from PDFs fail to benefit from the rich visual information contained in the formatting of math-heavy scientific content. To study this, we select human-written proof and prose passages from pre-2022 arXiv LaTeX sources and re-generate them using 15 modern LLMs within the context of the same paper. This leaves us with coupled human and AI outputs in three different representations: the LaTeX source, a rendered image of the PDF, and the text extracted from the PDF. We find that the accuracy of the state-of-the-art Pangram 4 drops significantly as the mathematical content in a passage increases, especially when evaluated on text as opposed to LaTeX. However, this is not an inherent limitation of detecting AI-generated math: finetuning Qwen-3.5-27B based detectors on this domain recovers the gap, with LaTeX inputs performing significantly better than text since they preserve more formatting information. Since LaTeX sources are rarely available in practice, we also train SnapJudge: an image-based detector operating on screenshots of both prose and proofs from papers. At 0.1% false positive rate, SnapJudge achieves a detection rate of 92% on proofs and 93% on prose generated by frontier models, substantially outperforming both Pangram 4 and our text-based detector. We release our data, models and code at https://nlplab.cs.duke.edu/snapjudge.
SciAgent: Tool-augmented Language Models for Scientific Reasoning
Microsoft(微软公司);University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);Tsinghua University(清华大学);Singapore Management University(新加坡管理大学);Nanyang Technological University(南洋理工大学)
LSR-Ben: A Logical and Scientific Reasoning Benchmark for Evaluating Process Reward Models
Research Center for Social Computing(社会计算研究中心);Interactive Robotics, Harbin Institute of Technology, China(交互机器人学,哈尔滨工业大学,中国);Beijing Academy of Artificial Intelligence, Beijing, China(北京人工智能研究院,北京,中国)
On LLM-Based Scientific Inductive Reasoning Beyond Equations
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, Tsinghua University(计算机科学与技术系,人工智能研究院,BNRist中心,清华大学);Jiangsu Collaborative Innovation Center for Language Ability, Jiangsu Normal University(江苏语言能力协同创新中心,江苏师范大学);Beijing Language and Culture University(北京语言文化大学);Xiamen University(厦门大学);Harbin Institute of Technology(哈尔滨工业大学)
Learning Beyond Pattern Matching? Assaying Mathematical Understanding in LLMs
University of Cambridge(剑桥大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Mila, University of Montreal(蒙特利尔大学Mila研究所);Google DeepMind(谷歌DeepMind)
MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses
Nanyang Technological University(南洋理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Wuhan University(武汉大学);University of New South Wales(新南威尔士大学);GreenDynamics;Singapore University of Technology and Design(新加坡科技设计大学)
Investigating Disagreement in the Scientific Literature
Centre for Science and Technology Studies, Leiden University(莱顿大学科学技术研究中心);SciTech Strategies, Inc.(SciTech Strategies 公司);School of Public Policy, Georgia Institute of Technology(佐治亚理工学院公共政策学院);School of Informatics, Computing, and Engineering, Indiana University Bloomington(印第安纳大学伯明顿分校信息学、计算与工程学院)
MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems
Department of Computer Science, Yale University(耶鲁大学计算机科学系);Department of Computer Science, Brown University(布朗大学计算机科学系);School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Key Laboratory of Data Science(上海数据科学关键实验室);Datawiz LLC