SLOG：面向语义解析的结构泛化基准
SLOG: A Structural Generalization Benchmark for Semantic Parsing
AI总结：
本文提出了SLOG，一个扩展COGS的语义解析数据集，包含17个结构泛化案例，揭示了现有模型在结构泛化上的不足，凸显了词汇泛化与结构泛化能力的巨大差异。
摘要：
组合泛化基准的目标是评估模型对新的复杂语言表达的泛化能力。现有基准通常关注词汇泛化，即在训练中熟悉的句法结构中对新词汇项的解释；而结构泛化任务，即模型需要解释训练中本身不熟悉的句法结构，往往代表性不足，导致对模型泛化能力的过度乐观看法。我们引入了SLOG，这是一个语义解析数据集，通过17个结构泛化案例扩展了COGS（Kim和Linzen，2020）。在我们的实验中，包括预训练模型在内的Transformer模型的泛化准确率仅达到40.6%，而结构感知解析器也仅达到70.8%。这些结果远未达到现有模型在COGS上取得的近乎完美的准确率，证明了SLOG在凸显模型词汇泛化与结构泛化能力之间巨大差异方面的作用。
英文摘要：
The goal of compositional generalization benchmarks is to evaluate how well models generalize to new complex linguistic expressions. Existing benchmarks often focus on lexical generalization, the interpretation of novel lexical items in syntactic structures familiar from training; structural generalization tasks, where a model needs to interpret syntactic structures that are themselves unfamiliar from training, are often underrepresented, resulting in overly optimistic perceptions of how well models can generalize. We introduce SLOG, a semantic parsing dataset that extends COGS (Kim and Linzen, 2020) with 17 structural generalization cases. In our experiments, the generalization accuracy of Transformer models, including pretrained ones, only reaches 40.6%, while a structure-aware parser only achieves 70.8%. These results are far from the near-perfect accuracy existing models achieve on COGS, demonstrating the role of SLOG in foregrounding the large discrepancy between models' lexical and structural generalization capacities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
