arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.06641cs.IRcs.CL
收藏

基于索引的方法用于高效有效的网络内容提取

An Index-based Approach for Efficient and Effective Web Content Extraction

University of Science and Technology of China（中国科学技术大学）；Metastone Technology（Metastone技术）

Yihan Chen, Benfeng Xu, Xiaorui Wang, Zhendong Mao

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于索引的方法，用于高效提取网络相关内容，提升RAG系统查询准确性与处理速度。

摘要：

随着网络代理（例如Deep Research）经常性地消耗大量网页内容以收集和分析信息，LLM上下文管理——在大token预算和低信号密度下——成为代理和RAG管道中的基础、重要且技术上具有挑战性的问题。现有提取相关内容的解决方案不足：生成式提取模型存在高延迟，基于规则的启发式方法缺乏适应性，分块和重排序方法对网页结构盲。为克服这些问题，我们引入了基于索引的网络内容提取，将提取过程从缓慢的逐token生成转变为高效的、具有判别性的索引预测任务，实现了有效性和效率。我们将HTML分成结构感知、可寻址的片段，并仅提取与给定查询相关的内容位置索引。该方法将提取延迟与内容长度解耦，实现了快速、与查询相关的提取。我们首先评估了该方法作为RAG QA系统中的后检索处理组件，并发现它提高了QA准确性。然后我们直接测量其在两个场景中的匹配率：主要内容提取（ME）和查询相关提取（QE）。实验结果表明，我们的方法在准确性和速度上均优于现有方法，有效弥合了LLM与庞大网页之间的差距。

英文摘要：

As web agents (e.g., Deep Research) routinely consume massive volumes of web pages to gather and analyze information, LLM context management -- under large token budgets and low signal density -- emerges as a foundational, high-importance, and technically challenging problem for agentic and RAG pipelines. Existing solutions for extracting relevant content are inadequate: generative extraction models suffer from high latency, rule-based heuristics lack adaptability, and chunk-and-rerank methods are blind to webpage structure. To overcome these issues, we introduce Index-based Web Content Extraction to reframe the extraction process from slow, token-by-token generation into a highly efficient, discriminative task of index prediction, achieving both effectiveness and efficiency. We partition HTML into structure-aware, addressable segments, and extract only the positional indices of content relevant to a given query. This method decouples extraction latency from content length, enabling rapid, query-relevant extraction. We first evaluate our method as a post-retrieval processing component within an RAG QA system and find that it improves QA accuracy. Then we directly measure its match rate with the target content in two scenarios: main content extraction (ME) and query-relevant extraction (QE). Experimental results show that our method outperforms existing works in both accuracy and speed, effectively bridging the gap between LLMs and the vast webpages.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑