arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.14969cs.CL
收藏

Lost in Space: 为结构化输出找到正确的Token

Lost in Space: Finding the Right Tokens for Structured Output

Department of Information Science(信息科学系);Cornell University(康奈尔大学)

Sil Hamilton, David Mimno

AI总结:

本研究针对大语言模型结构化输出会不可预测降低下游性能的问题,测试不同模型、输出格式与基准,发现符合惯例的格式及带前导空格的token可提升性能,并给出零样本分类的最佳实践。

摘要:

通用语言模型被训练为生成多样化的自然语言输出,但对于标注或分类等某些任务,我们需要更特定的输出格式。大语言模型(LLM)系统越来越多地支持结构化输出,这类输出通过根据语法采样token来强制格式规范,但这也会不可预测地降低下游性能。对于人类而言在语义上(通常在视觉上)相似的语法之间,是否存在系统性差异?为回答这一问题,我们在四个常见的NLP基准上,用五种不同的输出格式测试了四个主流模型系列。我们发现,当引导模型使用符合惯例的格式(例如选择题用字母、数值预测用实数)时,所有模型的准确率都最高。当引导模型返回包含前导空格的token时,性能也会提升5%-10%,其中较小的模型受益最大。我们发现前导空格有助于模型避免子词token表示中的结构缺陷。最后,我们为使用语言模型作为带结构化输出的零样本分类器的研究人员提供了最佳实践。

英文摘要:

General-purpose language models are trained to produce varied natural language outputs, but for some tasks, like annotation or classification, we need more specific output formats. LLM systems increasingly support structured output, which enforces formats by sampling tokens according to a grammar -- but also unpredictably reduces downstream performance. Are there systematic differences between grammars that appear semantically (and often visually) similar to humans? To answer this, we test four popular model families with five varying output formats on four common NLP benchmarks. We find all models perform most accurately when guided to use formats respecting convention, such as letters for multiple choice and real numbers for numerical prediction. Performance also improves by 5%-10% when guiding models to return tokens incorporating leading whitespace, with smaller models benefiting the most. We find leading whitespace helps models avoid structural deficiencies in subword token representations. We finally present best practices for researchers using language models as zero-shot classifiers with structured output.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑