首页/论文/2001.10929

arXiv 2001.109292020-01cs.CLcs.AI

AMR Similarity Metrics from Principles

Heidelberg University（海德堡大学）

Juri Opitz, Letitia Parcalabescu, Anette Frank

备注：: TACL 2020 https://doi.org/10.1162/tacl_a_00329
DOI：: 10.1162/tacl_a_00329

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

Different metrics have been proposed to compare Abstract Meaning Representation (AMR) graphs. The canonical Smatch metric (Cai and Knight, 2013) aligns the variables of two graphs and assesses triple matches. The recent SemBleu metric (Song and Gildea, 2019) is based on the machine-translation metric Bleu (Papineni et al., 2002) and increases computational efficiency by ablating the variable-alignment. In this paper, i) we establish criteria that enable researchers to perform a principled assessment of metrics comparing meaning representations like AMR; ii) we undertake a thorough analysis of Smatch and SemBleu where we show that the latter exhibits some undesirable properties. For example, it does not conform to the identity of indiscernibles rule and introduces biases that are hard to control; iii) we propose a novel metric S$^2$match that is more benevolent to only very slight meaning deviations and targets the fulfilment of all established criteria. We assess its suitability and show its advantages over Smatch and SemBleu.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Weisfeiler-Leman in the BAMBOO: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity

Heidelberg University（海德堡大学）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）

Several metrics have been proposed for assessing the similarity of (abstract) meaning representations (AMRs), but little is known about how they relate to human similarity ratings. Moreover, the current metrics have complementary strengths and weaknesses: some emphasize speed, while others make the alignment of graph structures explicit, at the price of a costly alignment step. In this work we propose new Weisfeiler-Leman AMR similarity metrics that unify the strengths of previous metrics, while mitigating their weaknesses. Specifically, our new metrics are able to match contextualized substru…

2021-08-26cs.AIcs.CL

A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation -- through the Lens of Semantic Similarity Rating

Heidelberg University（海德堡大学）

Evaluating the quality of generated text is difficult, since traditional NLG evaluation metrics, focusing more on surface form than meaning, often fail to assign appropriate scores. This is especially problematic for AMR-to-text evaluation, given the abstract nature of AMR. Our work aims to support the development and improvement of NLG evaluation metrics that focus on meaning, by developing a dynamic CheckList for NLG metrics that is interpreted by being organized around meaning-relevant linguistic phenomena. Each test instance consists of a pair of sentences with their AMR graphs and a human…

2022-05-24cs.CL

Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation

The Chinese University of Hong Kong（香港中文大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）

We introduce a new method to improve existing multilingual sentence embeddings with Abstract Meaning Representation (AMR). Compared with the original textual input, AMR is a structured semantic representation that presents the core concepts and relations in a sentence explicitly and unambiguously. It also helps reduce surface variations across different expressions and languages. Unlike most prior work that only evaluates the ability to measure semantic similarity, we present a thorough evaluation of existing multilingual sentence embeddings and our improved versions, which include a collectio…

2022-10-18cs.AIcs.CL

Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation

Georgetown University（乔治城大学）

Identifying semantically equivalent sentences is important for many cross-lingual and mono-lingual NLP tasks. Current approaches to semantic equivalence take a loose, sentence-level approach to "equivalence," despite previous evidence that fine-grained differences and implicit content have an effect on human understanding (Roth and Anthonio, 2021) and system performance (Briakou and Carpuat, 2021). In this work, we introduce a novel, more sensitive method of characterizing semantic equivalence that leverages Abstract Meaning Representation graph structures. We develop an approach, which can be…

2022-10-06cs.CL

Rematch：局部知识图谱的鲁棒高效匹配以提升结构与语义相似度

Rematch: Robust and Efficient Matching of Local Knowledge Graphs to Improve Structural and Semantic Similarity

Observatory on Social Media（社会媒体观测站）；Indiana University（印第安纳大学）

本文提出AMR相似度指标rematch和结构相似度评估基准RARE，解决了现有方法效率低且难以捕捉语义相似度的问题，在语义相似度上取得领先并显著提升计算效率。

2024-04-02cs.CLcs.IR

AMR-RE：用于关系抽取中基于检索的上下文学习的抽象意义表示

AMR-RE: Abstract Meaning Representations for Retrieval-Based In-Context Learning in Relation Extraction

Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；National Institute of Information and Communications Technology (NICT)（信息通信研究机构）；Kyoto University（京都大学）；Kyoto Institute of Technology（京都工艺纤维大学）

针对关系抽取中上下文学习重语言相似性轻结构相似性导致忽略实体关系的问题，提出AMR增强的基于检索的ICL方法，通过语义结构相似性检索示例，在无监督设置下全面超越基线，并在监督设置下于多数数据集达到最优。

2025-04-25cs.CL

更多相关论文收起更多论文

Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

Heidelberg University（海德堡大学）

Systems that generate natural language text from abstract meaning representations such as AMR are typically evaluated using automatic surface matching metrics that compare the generated texts to reference texts from which the input meaning representations were constructed. We show that besides well-known issues from which such metrics suffer, an additional problem arises when applying these metrics for AMR-to-text evaluation, since an abstract meaning representation allows for numerous surface realizations. In this work we aim to alleviate these issues by proposing $\mathcal{M}\mathcal{F}_β$, …

2021-01-26cs.CL

Core Semantic First: A Top-down Approach for AMR Parsing

The Chinese University of Hong Kong（香港中文大学）

We introduce a novel scheme for parsing a piece of text into its Abstract Meaning Representation (AMR): Graph Spanning based Parsing (GSP). One novel characteristic of GSP is that it constructs a parse graph incrementally in a top-down fashion. Starting from the root, at each step, a new node and its connections to existing nodes will be jointly predicted. The output graph spans the nodes by the distance to the root, following the intuition of first grasping the main ideas then digging into more details. The \textit{core semantic first} principle emphasizes capturing the main ideas of a senten…

2019-09-12cs.AIcs.CL

大型语言模型能否解释并利用结构化的语言表示？一项基于AMR的案例研究

Can LLMs Interpret and Leverage Structured Linguistic Representations? A Case Study with AMRs

University of Colorado Boulder（科罗拉多大学博尔德分校）；Northwestern University（西北大学）

本文研究了大型语言模型在结构化语言表示下的表现，发现短文任务中添加AMR会降低性能，但长文任务如对话摘要能提升模型表现，尤其在大模型中效果更显著。

2025-06-27cs.CL

AMREx：用于可解释事实验证的 AMR

AMREx: AMR for Explainable Fact Verification

University of Florida（佛罗里达大学）

本文提出 AMREx，一种结合 Smatch 与 AMR 节点映射的可解释事实验证系统，在 FEVER 和 AVeriTeC 上验证有效，并可引导 LLM 生成低幻觉自然语言解释。

2024-11-02cs.CL

基于平行语料库的原则性复述生成

Principled Paraphrase Generation with Parallel Corpora

HiTZ Center, University of the Basque Country (UPV/EHU)（巴斯克大学HiTZ中心）；Meta AI

本文针对往返机器翻译复述生成易受歧义翻译影响的问题，提出基于信息瓶颈的对抗方法，通过匹配翻译分布学习表示，无需枢轴翻译即可生成复述，实现可控的保真度-多样性权衡且效果更优。

2023-05-23cs.CL

AMR Dependency Parsing with a Typed Semantic Algebra

Saarland University（萨尔兰大学）；Macquarie University（麦考瑞大学）

We present a semantic parser for Abstract Meaning Representations which learns to parse strings into tree representations of the compositional structure of an AMR graph. This allows us to use standard neural techniques for supertagging and dependency tree parsing, constrained by a linguistically principled type system. We present two approximative decoding algorithms, which achieve state-of-the-art accuracy and outperform strong baselines.

2018-05-29cs.CL

从统一意义表示生成文本

Generating Text from Uniform Meaning Representation

Amherst College（阿默斯特学院）

本文提出从多语言UMR图生成文本的方法，通过基线模型、管道转换和微调策略，实现了多语言文本生成的高效效果。

2026-01-13cs.CL

SBERT研究意义表示：将句子嵌入分解为可解释的语义特征

SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features

Heidelberg University（海德堡大学）

本研究提出S$^3$BERT，通过逼近AMR图度量将SBERT句子嵌入分解为可解释的语义子嵌入，同时保持原嵌入的相似度性能，兼顾可解释性与效率。

2022-10-28cs.AIcs.CL

强调结构化信息：将抽象意义表示集成到 LLMs 中以增强开放域对话评估

Emphasising Structured Information: Integrating Abstract Meaning Representation into LLMs for Enhanced Open-Domain Dialogue Evaluation

针对开放域对话评估中对抗性负样本的挑战，提出一种将抽象意义表示（AMR）增强的特定领域语言模型与大语言模型集成的评估框架，通过门控机制和提示融合显著提升了与人类判断的相关性。

2025-09-16cs.CL

基于词对齐的统一意义表示评估

Word Alignment-Based Evaluation of Uniform Meaning Representations

Charles University Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics (ÚFAL)（查理大学数学物理学院形式与应用语言学研究所（ÚFAL））

本文提出基于词对齐的统一意义表示比较方法，通过分析词对齐关系提高语义表示比较的直观性和可解释性，避免了smatch的NP难搜索问题。

2026-03-27cs.CL

Semantic Similarity-Based Clustering of Findings From Security Testing Tools

Technical University of Munich（慕尼黑工业大学）；Siemens Technology（西门子科技）；LMU Munich（慕尼黑大学）

Over the last years, software development in domains with high security demands transitioned from traditional methodologies to uniting modern approaches from software development and operations (DevOps). Key principles of DevOps gained more importance and are now applied to security aspects of software development, resulting in the automation of security-enhancing activities. In particular, it is common practice to use automated security testing tools that generate reports after inspecting a software artifact from multiple perspectives. However, this raises the challenge of generating duplicat…

2022-11-20cs.CLcs.SE

Lacuna公司在SemEval-2026任务4中的表现：用于解开叙事相似性的结构门控状态空间模型

Lacuna Inc. at SemEval-2026 Task 4: Structurally Gated State-Space Models for Disentangling Narrative Similarity

Lacuna Inc.（拉库纳公司）；HSE University（俄罗斯高等经济学院）

研究叙事相似性评估难题，用混合状态空间模型避免标准Transformer二次瓶颈，引入结构门控对齐头，在不同任务中验证从词汇变体中分离结构不变量可助力深度叙事理解。

2026-07-07cs.CL

L1增强注意力作为一种改进的向量相似性度量

L1 Augmented Attention as an Improved Vector Similarity Metric

研究针对缩放点积注意力在Transformer模型中作为相似性度量的局限，提出L1增强注意力方法，通过减去特定头的L1距离改进相似性计算，经实验在WikiText 2上取得更好效果，揭示了各层几何作用及头级专业化，有效提升语言模型相似性计算。

2026-07-21cs.CLcs.LG