ChunkRank：面向LLM管道的模型感知文本分块与弃权（不执行）感知答案选择
ChunkRank: Model-Aware Text Chunking and Abstention-Aware Answer Selection for LLM Pipelines
浏览论文内容
中文总结 AI 辅助
ChunkRank是一个开源Python库，通过模型感知分块避免上下文溢出，并在答案选择中发现弃权（不执行）导致内容排序器无效，故采用首个非空答案，适用于小窗口和超窗口场景。
中文摘要 AI 辅助
我们提出ChunkRank，一个开源的Python库，它从目标模型的tokenizer和上下文窗口中推导分块边界，并在每个块独立生成的候选答案中选择一个答案。该库附带一个经过验证的注册表，涵盖15个提供商的90个模型和六种答案选择方法，且仅需三个核心依赖。在分块方面，ChunkRank根据模型名称自动避免上下文窗口溢出，而基于字符的分割器会溢出或浪费预算，一项跨11种语言的保真度研究显示了在英语之外，精确的token预算为何重要。在答案选择方面，我们报告了一个负面结果：在NaturalQuestions、TriviaQA和HotpotQA上，使用抽取式和生成式阅读器，没有任何基于内容的排序器能可靠地优于取第一个非空答案。原因在于阅读器对缺乏答案的块进行弃权（不执行），而非答案位置。一个长上下文基线显示，在单跳问题上，分块与单次调用阅读相匹配，因此ChunkRank针对小窗口和超出窗口的设置。代码、注册表和评估工具已发布。
英文摘要
We present ChunkRank, an open-source Python library that derives chunk boundaries from a target model's tokenizer and context window, and selects an answer among candidates produced independently per chunk. It ships a validated registry of 90 models across 15 providers and six answer-selection methods, and needs only three core dependencies. For chunking, ChunkRank avoids context-window overflow automatically from the model name, whereas character-based splitters overflow or waste the budget, and a fidelity study across 11 languages shows why token-exact budgets matter beyond English. For answer selection we report a negative result: on NaturalQuestions, TriviaQA and HotpotQA, with extractive and generative readers, no content-based ranker reliably beats taking the first non-empty answer. The reason is reader abstention on chunks that lack the answer, not answer position. A long-context baseline shows that chunking matches single-call reading on single-hop questions, so ChunkRank targets small-window and beyond-window settings. Code, registry and evaluation harness are released.
