State-of-the-art NLP benchmarks require interpretation of natural language that specifies conditions, procedures, and exceptions, often relying on implicit assumptions and external knowledge. Constructing complete semantic representations with proof-theoretic guarantees is frequently impractical at scale, and purely text-based reasoning offers limited means of inspection. This paper asks how much understanding of benchmark language can be achieved when formal semantic guarantees are weakened. We investigate this question by extracting computables: executable representations whose runtime behavior provides operational evidence of semantic adequacy, including executability, execution traces, and runtime failures. We induce and iteratively refine computables for benchmark instances using retrieval from external knowledge. Across mathematical reasoning, multi-step reasoning, causal inference, and rule- and exception-heavy legal and biomedical benchmarks, we find that the proposed approach consistently exceeds text-only reasoning and one-shot code execution. Beyond accuracy, our analyses show that these computables provide scalable, inspectable semantic evidence: they expose conditions and exceptions benchmark language forces into executable form, offering a practical bridge between proof-oriented semantics and purely textual reasoning.
A Survey on Semantic Parsing from the perspective of Compositionality
Indian Institute of Technology Delhi（印度理工学院德里分校）
与之前在语义解析（Kamath和Das，2018）和知识库问答（KBQA）（Chakraborty等人，2019; Zhu等人，2019; Hoffner等人，2017）我们试图从不同的角度来研究语义解析。具体来说，我们将集中在（a）从句法结构（Partee，1975年）的意义组合，（b）语义分析器处理给定的知识库（KB）上下文的词汇变化的能力。在接下来的部分中，在介绍了语义解析领域及其在KBQA中的应用之后，我们将描述使用语法形式主义CCG（Steedman，1996）的意义表示。我们将在第2节讨论使用形式语言的语义组合。在第3节中，我们将考虑使用形式语言的系统，例如$λ$ -演算（Steedman，1996），$λ$ -DCS（Liang，2013）。第4节和第5节考虑使用结构化语言的逻辑形式的语义分析器。第6节是关于不同的基准数据集ComplexQuestions（Bao et al.，2016）和GraphQuestions（Su et al.，2016），可用于评估语义解析器回答本质上高度组合的复杂问题的能力。
