异构依赖图引导的专利表示学习注意力机制
Heterogeneous Dependency Graph-Guided Attentionfor Patent Representation Learning
AI总结:
针对专利权利要求间的依赖层次被忽略的问题,提出专利异构注意力图编码器(PHAGE),通过构建类型图区分法律引用与技术关系,并引入可学习偏置的连通性掩码将权利要求级拓扑投射到令牌级注意力,结合双粒度对比学习,在分类、检索和聚类任务上超越领域自适应和引用感知基线。
摘要:
预训练语言模型通过将权利要求编码为扁平令牌序列来推进专利分类和检索,但忽略了权利要求之间的依赖层次。将层次结构融入自注意力面临两个挑战。首先,权利要求依赖涉及不同可靠性的关系类型:不加区分地对待它们会使有噪声的技术关系污染更清洁的法律引用信号。其次,当依赖图在权利要求级别定义时,Transformer模型会失败,因为它们在令牌级别操作;广播权利要求级别的邻接可能会稀释跨无关令牌对的结构信息。一种新颖的专利异构注意力图编码器(PHAGE)解决了这些挑战。为了处理异构依赖,PHAGE构建了一个类型图,将法律引用与技术关系区分为不同的边类型。为了弥合层次差距,PHAGE引入了一个带有可学习关系感知偏置的连通性掩码,将权利要求级别的拓扑投射到令牌级别的注意力中。PHAGE学习一个双粒度对比目标,以将表示与专利间分类法和专利内拓扑对齐。实验表明,PHAGE在专利分类、检索和聚类上优于领域自适应和引用感知基线。PHAGE揭示,专利内权利要求拓扑比专利间结构捕获了更强的归纳偏置。
英文摘要:
Pre-trained language models advance patent classification and retrieval by encoding claims as flat token sequences, but they overlook the dependency hierarchy among claims. Incorporating this hierarchy into self-attention poses two challenges. First, claim dependencies include relation types with different levels of reliability, so treating them uniformly may allow noisy technical relations to interfere with more reliable legal citations. Second, claim dependencies are defined at the claim level, whereas Transformer attention operates over tokens, making direct structural injection nontrivial. We propose the Patent Heterogeneous Attention-Guided Graph Encoder (PHAGE), which constructs a typed claim graph that distinguishes legal citations from technical relations. PHAGE projects this claim-level topology into token-level attention through a connectivity mask and learnable relation-aware biases, and fine-tunes the encoder using a dual-granularity contrastive objective that combines inter-patent taxonomy with intra-patent topology. At inference, the graph-specific attention components are removed, allowing representations to be generated through a standard encoder forward pass without CDG construction. Experiments on patent classification, retrieval, and clustering show that PHAGE consistently outperforms domain-adapted and citation-aware baselines, demonstrating the value of claim-level structural guidance for graph-free patent representation learning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。