BenchCLAMP：一个用于评估语言模型在句法和语义解析上的基准
BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing
AI总结：
本文提出 BenchCLAMP 基准，通过提供上下文无关文法和受约束解码接口，评估语言模型在句法和语义解析任务上的性能，发现受约束的编码器-解码器模型能超越现有最先进方法。
摘要：
最近的工作表明，当输出被约束为有效的语义表示时，通过提示或微调语言模型进行生成可以在语义解析中表现良好。我们引入了 BenchCLAMP，一个用于评估受约束语言模型解析的基准，它包括七个语义解析数据集和两个句法解析数据集的上下文无关文法，这些数据集具有不同的输出表示，以及一个受约束的解码接口，用于仅生成这些文法所覆盖的有效输出。我们为每个数据集提供低、中、高资源划分，允许在不同数据机制下准确比较各种语言模型。我们的基准支持使用基于提示的学习以及微调来评估语言模型。我们对八个语言模型进行了基准测试，包括两个仅可通过 API 获取的 GPT-3 变体。我们的实验表明，当模型输出被约束为有效时，编码器-解码器预训练语言模型在句法和语义解析上可以达到相似的性能或超越最先进的方法。
英文摘要：
Recent work has shown that generation from a prompted or fine-tuned language model can perform well at semantic parsing when the output is constrained to be a valid semantic representation. We introduce BenchCLAMP, a Benchmark to evaluate Constrained LAnguage Model Parsing, that includes context-free grammars for seven semantic parsing datasets and two syntactic parsing datasets with varied output representations, as well as a constrained decoding interface to generate only valid outputs covered by these grammars. We provide low, medium, and high resource splits for each dataset, allowing accurate comparison of various language models under different data regimes. Our benchmark supports evaluation of language models using prompt-based learning as well as fine-tuning. We benchmark eight language models, including two GPT-3 variants available only through an API. Our experiments show that encoder-decoder pretrained language models can achieve similar performance or surpass state-of-the-art methods for syntactic and semantic parsing when the model output is constrained to be valid.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
