大语言模型能取代人类程序员吗？引入ContentBench
Can Large Language Models Replace Human Coders? Introducing ContentBench
AI总结：
本文通过ContentBench基准测试，评估低成本大语言模型在解释性编程任务中的表现，揭示其在高精度任务中的优势及在讽刺处理上的局限性。
摘要：
低成本的大语言模型（LLMs）能否取代仍然主导经验内容分析的解释性编程工作？本文介绍了ContentBench，一个公开的基准测试套件，通过跟踪低成本LLMs在相同解释性编程任务上达成的共识程度和成本，来回答这一替代问题。该套件使用版本化的赛道，邀请研究人员贡献新的基准数据集。我报告了第一个赛道ContentBench-ResearchTalk v1.0的结果：1000个合成的社交媒体风格帖子，关于学术研究，被标记为五个类别，涵盖赞扬、批评、讽刺、问题和程序性评论。参考标签仅在三种最先进的推理模型（GPT-5、Gemini 2.5 Pro和Claude Opus 4.1）一致同意时才会分配，并且所有最终标签都由作者作为质量控制审计检查。在评估的59个模型中，最好的低成本LLMs与这些陪审团标签的共识率约为97-99%，远高于早期ChatGPT背后模型GPT-3.5 Turbo以及LLM基础文本标注的初始浪潮。几个顶级模型可以仅用几美元对50,000个帖子进行编码，将大规模解释性编码从劳动力瓶颈推向验证、报告和治理的问题。同时，小型轻量模型在本地运行时仍难以处理讽刺密集的项目（例如，Llama 3.2 3B在处理硬讽刺时仅达到4%的共识）。ContentBench随数据、文档和一个交互式测验发布，网址为contentbench.github.io，以支持随时间推移的可比评估并邀请社区扩展。
英文摘要：
Can low-cost large language models (LLMs) take over the interpretive coding work that still anchors much of empirical content analysis? This paper introduces ContentBench, a public benchmark suite that helps answer this replacement question by tracking how much agreement low-cost LLMs achieve and what they cost on the same interpretive coding tasks. The suite uses versioned tracks that invite researchers to contribute new benchmark datasets. I report results from the first track, ContentBench-ResearchTalk v1.0: 1,000 synthetic, social-media-style posts about academic research labeled into five categories spanning praise, critique, sarcasm, questions, and procedural remarks. Reference labels are assigned only when three state-of-the-art reasoning models (GPT-5, Gemini 2.5 Pro, and Claude Opus 4.1) agree unanimously, and all final labels are checked by the author as a quality-control audit. Among the 59 evaluated models, the best low-cost LLMs reach roughly 97-99% agreement with these jury labels, far above GPT-3.5 Turbo, the model behind early ChatGPT and the initial wave of LLM-based text annotation. Several top models can code 50,000 posts for only a few dollars, pushing large-scale interpretive coding from a labor bottleneck toward questions of validation, reporting, and governance. At the same time, small open-weight models that run locally still struggle on sarcasm-heavy items (for example, Llama 3.2 3B reaches only 4% agreement on hard-sarcasm). ContentBench is released with data, documentation, and an interactive quiz at contentbench.github.io to support comparable evaluations over time and to invite community extensions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
