arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.21062cs.CL
收藏

面向数学密集型科学文本的多模态AI检测

SnapJudge: Multimodal AI Detection for Math-Heavy Scientific Text

Duke University(杜克大学);Indian Institute of Science(印度科学学院)

Yixuan Huang, Danish Pruthi, Bhuwan Dhingra

AI总结:

本研究针对数学密集型科学文本,构建了基于Qwen-3.5-27B的检测器与图像检测器SnapJudge,解决现有AI文本检测器对数学内容检测准确率低的问题,其中SnapJudge在0.1%误报率下对前沿模型生成内容检测率超92%。

摘要:

随着大语言模型在生成和辅助科学写作方面的能力不断提升,研究领域对科研中使用AI的透明度和披露需求日益增长。因此,一个名副其实的研究子领域及多项商业服务旨在自动标记AI生成内容。然而,仅依赖从PDF中提取文本的现有检测器,无法利用数学密集型科学内容格式中包含的丰富视觉信息。为研究这一问题,我们从2022年之前的arXiv LaTeX资源中选取人类撰写的证明文本和散文段落,并在同一论文的语境下使用15种现代大语言模型(LLM)重新生成这些内容,最终得到三种不同表示形式的配对人类与AI输出:LaTeX源文件、PDF渲染图像以及从PDF中提取的文本。我们发现,最先进的Pangram 4检测器的准确率会随着段落中数学内容的增加而显著下降,尤其是在基于文本而非LaTeX进行评估时。不过,这并非检测AI生成数学内容的固有局限:在该领域对基于Qwen-3.5-27B的检测器进行微调可弥补这一差距,且LaTeX输入的表现明显优于文本,因为其保留了更多格式信息。由于实际场景中很少能获取LaTeX源文件,我们还训练了SnapJudge:一种基于图像的检测器,可对论文中散文和证明的截图进行操作。在0.1%的误报率下,SnapJudge对前沿模型生成的证明的检测率达92%,对散文的检测率达93%,显著优于Pangram 4和我们的基于文本的检测器。我们在该httpsURL发布了数据、模型和代码。

英文摘要:

As large language models become increasingly capable of generating, and assisting with, scientific writing, there is a need for greater transparency and disclosure around the use of AI in research. Consequently, a veritable subfield of research and a number of commercial services aim to flag AI-generated content automatically. However, existing detectors which rely only on text extracted from PDFs fail to benefit from the rich visual information contained in the formatting of math-heavy scientific content. To study this, we select human-written proof and prose passages from pre-2022 arXiv LaTeX sources and re-generate them using 15 modern LLMs within the context of the same paper. This leaves us with coupled human and AI outputs in three different representations: the LaTeX source, a rendered image of the PDF, and the text extracted from the PDF. We find that the accuracy of the state-of-the-art Pangram 4 drops significantly as the mathematical content in a passage increases, especially when evaluated on text as opposed to LaTeX. However, this is not an inherent limitation of detecting AI-generated math: finetuning Qwen-3.5-27B based detectors on this domain recovers the gap, with LaTeX inputs performing significantly better than text since they preserve more formatting information. Since LaTeX sources are rarely available in practice, we also train SnapJudge: an image-based detector operating on screenshots of both prose and proofs from papers. At 0.1% false positive rate, SnapJudge achieves a detection rate of 92% on proofs and 93% on prose generated by frontier models, substantially outperforming both Pangram 4 and our text-based detector. We release our data, models and code at https://nlplab.cs.duke.edu/snapjudge.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑