AI 中文总结
针对MMLongBench-Doc基准的标注错误等问题，推出MMLongBench-Doc-V2，修正106个标注、调整评估方式，移除错误文件名和重复问题，形成不可与V1分数对比的新基准。
AI 中文摘要
MMLongBench-Doc是一个长文档问答基准，包含135份PDF文档对应的1082个问题。该基准的两个特性导致测量分数偏离其应捕捉的实际值：一是参考指标对比提取的答案时，1,358,000会输给1358000；二是相当一部分真实标注存在错误、模糊或不完整的问题——由于这些标注的发现方式，它们恰好集中在能力强的系统能正确回答的问题上。MMLongBench-Doc-V2修正了106个标注，每个标注都附带确定其正确性的页码和算术依据，并用固定的LLM评判员取代字符串指标，该评判员需判断响应是否与参考内容语义一致。此外，移除了10个因文件名错误的问题，而非将其计为错误，还移除了1个重复问题，最终保留134份文档对应的1071个问题。最具复用性的贡献是一套决策程序，用于确定何时可扩展空集键，以及何时扩展会破坏刻意设置的负样本；将该程序应用于全部208条数据，共扩展了14条。V2的分数与已发布的V1分数不可比。修正后的语料库、每个条目的修正记录及评估工具均在此https URL处提供。
英文摘要
MMLongBench-Doc is a long-document QA benchmark of 1,082 questions over 135 PDFs. Two properties of it push measured scores away from the quantity they are meant to capture: the reference metric compares extracted answers, so 1,358,000 loses to 1358000; and a non-trivial share of ground-truth annotations are wrong, ambiguous, or incomplete --- concentrated, because of how they were found, in exactly the questions capable systems answer correctly. MMLongBench-Doc-V2 corrects 106 annotations, each published with the page and arithmetic that settle it, and replaces the string metric with a pinned LLM judge asked whether a response means the reference. Ten questions whose document ships under the wrong filename are removed rather than counted wrong, along with one duplicated question, leaving 1,071 questions over 134 documents. The most reusable contribution is a decision procedure for when an empty set key may be widened and when widening would destroy a deliberate negative sample; applied to all 208 rows, it widened 14. V2 scores are not comparable with published V1 numbers. The corrected corpus, the per-entry correction record and the evaluation harness are available at https://github.com/VectifyAI/MMLongBench-Doc-V2.
