大语言模型真的比人类更聪明吗？
Are Large Language Models Truly Smarter Than Humans?
AI总结：
本文通过三个实验评估六种前沿大语言模型的评估数据污染问题，发现STEM领域污染率最高，且部分模型存在记忆偏差。
摘要：
公共排行榜表明大语言模型在学术知识、法律和编程等领域超越了人类专家，但大多数基准测试问题广泛存在于互联网上，导致模型可能在训练数据和评估数据之间存在系统性风险。本文通过三个互补实验对六个前沿大语言模型进行了严格的多方法污染审计：GPT-4o、GPT-4o-mini、DeepSeek-R1、DeepSeek-V3、Llama-3.3-70B和Qwen3-235B。实验1应用词汇污染检测管道对513个MMLU问题进行分析，发现整体污染率为13.8%（STEM领域为18.1%，哲学领域高达66.7%），并估计各领域准确率提升0.030至0.054个百分点。实验2对100个MMLU问题应用改写和间接引用诊断，发现间接引用下准确率平均下降7.0个百分点，法律和伦理领域下降至19.8个百分点。实验3对所有513个问题和六个模型应用TS-Guessing行为探测，发现72.5%的问题触发记忆信号远高于随机水平，DeepSeek-R1表现出分布式的记忆签名（76.6%部分重构，0%原文回忆），解释了其异常的实验2表现。所有三个实验得出相同的污染排名：STEM > 专业 > 社会科学 > 人文学科。
英文摘要：
Public leaderboards increasingly suggest that large language models (LLMs) surpass human experts on benchmarks spanning academic knowledge, law, and programming. Yet most benchmarks are fully public, their questions widely mirrored across the internet, creating systematic risk that models were trained on the very data used to evaluate them. This paper presents three complementary experiments forming a rigorous multi-method contamination audit of six frontier LLMs: GPT-4o, GPT-4o-mini, DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, and Qwen3-235B. Experiment 1 applies a lexical contamination detection pipeline to 513 MMLU questions across all 57 subjects, finding an overall contamination rate of 13.8% (18.1% in STEM, up to 66.7% in Philosophy) and estimated performance gains of +0.030 to +0.054 accuracy points by category. Experiment 2 applies a paraphrase and indirect-reference diagnostic to 100 MMLU questions, finding accuracy drops by an average of 7.0 percentage points under indirect reference, rising to 19.8 pp in both Law and Ethics. Experiment 3 applies TS-Guessing behavioral probes to all 513 questions and all six models, finding that 72.5% trigger memorization signals far above chance, with DeepSeek-R1 displaying a distributed memorization signature (76.6% partial reconstruction, 0% verbatim recall) that explains its anomalous Experiment 2 profile. All three experiments converge on the same contamination ranking: STEM > Professional > Social Sciences > Humanities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
