NEXT-EVAL：传统与LLM网页数据记录提取的下一步评估
NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction
AI总结：
针对网页数据记录提取评估的静态与不透明问题，提出NEXT-EVAL框架，通过生成数据集、结构感知指标及多种输入预处理，基准测试表明扁平化JSON输入使LLM获得最佳F1分数（0.9567）并减少幻觉。
摘要：
对网页数据记录提取方法进行有效评估至关重要，然而静态的、特定领域的基准和不透明的评分实践阻碍了这一进程。这使得在依赖结构启发式的传统算法技术与基于大语言模型（LLM）的方法（后者能在不同布局中提供零样本提取）之间进行公平比较尤为困难。为克服这些局限，我们引入了一个具体的评估框架。该框架从任意MHTML快照中系统性地生成评估数据集，标注基于XPath的监督标签，并采用结构感知的指标进行一致性评分，具体而言，该指标可防止文本幻觉，仅允许评估位置幻觉。框架还整合了预处理策略，以优化LLM的输入，同时保留DOM语义：HTML精简、层次化JSON和扁平化JSON。此外，我们通过转换DOM结构和修改内容，创建了一个公开可用的合成数据集。我们在多种输入格式下对确定性启发式算法和现成的LLM进行了基准测试。基准测试表明，与其他输入格式（如精简HTML和层次化JSON）相比，扁平化JSON输入使LLM能够实现更高的提取准确率（F1分数为0.9567）和更少的幻觉。我们为严格的基准测试建立了标准化基础，为网页数据记录提取领域下一步原则性的进展铺平了道路。
英文摘要：
Effective evaluation of web data record extraction methods is crucial, yet hampered by static, domain-specific benchmarks and opaque scoring practices. This makes fair comparison between traditional algorithmic techniques, which rely on structural heuristics, and Large Language Model (LLM)-based approaches, offering zero-shot extraction across diverse layouts, particularly challenging. To overcome these limitations, we introduce a concrete evaluation framework. Our framework systematically generates evaluation datasets from arbitrary MHTML snapshots, annotates XPath-based supervision labels, and employs structure-aware metrics for consistent scoring, specifically preventing text hallucination and allowing only for the assessment of positional hallucination. It also incorporates preprocessing strategies to optimize input for LLMs while preserving DOM semantics: HTML slimming, Hierarchical JSON, and Flat JSON. Additionally, we created a publicly available synthetic dataset by transforming DOM structures and modifying content. We benchmark deterministic heuristic algorithms and off-the-shelf LLMs across these multiple input formats. Our benchmarking shows that Flat JSON input enables LLMs to achieve superior extraction accuracy (F1 score of 0.9567) and minimal hallucination compared to other input formats like Slimmed HTML and Hierarchical JSON. We establish a standardized foundation for rigorous benchmarking, paving the way for the next principled advancements in web data record extraction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
