首页/论文/1701.02481

arXiv 1701.024812017-01cs.CLcs.LG

Implicitly Incorporating Morphological Information into Word Embedding

University of Science and Technology of China（中国科学技术大学）

Yang Xu, Jiawei Liu

备注：: 7 pages, 7 figures

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

In this paper, we propose three novel models to enhance word embedding by implicitly using morphological information. Experiments on word similarity and syntactic analogy show that the implicit models are superior to traditional explicit ones. Our models outperform all state-of-the-art baselines and significantly improve the performance on both tasks. Moreover, our performance on the smallest corpus is similar to the performance of CBOW on the corpus which is five times the size of ours. Parameter analysis indicates that the implicit models can supplement semantic information during the word embedding training process.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Evaluation of Morphological Embeddings for the Russian Language

Innopolis University（因诺波利斯大学）

A number of morphology-based word embedding models were introduced in recent years. However, their evaluation was mostly limited to English, which is known to be a morphologically simple language. In this paper, we explore whether and to what extent incorporating morphology into word embeddings improves performance on downstream NLP tasks, in the case of morphologically rich Russian language. NLP tasks of our choice are POS tagging, Chunking, and NER -- for Russian language, all can be mostly solved using only morphology without understanding the semantics of words. Our experiments show that m…

2021-03-11cs.CLcs.LG

Rule-based Morphological Inflection Improves Neural Terminology Translation

University of Maryland（马里兰大学）

Current approaches to incorporating terminology constraints in machine translation (MT) typically assume that the constraint terms are provided in their correct morphological forms. This limits their application to real-world scenarios where constraint terms are provided as lemmas. In this paper, we introduce a modular framework for incorporating lemma constraints in neural MT (NMT) in which linguistic knowledge and diverse types of NMT models can be flexibly applied. It is based on a novel cross-lingual inflection module that inflects the target lemma constraints based on the source context. …

2021-10-08cs.CL

Morphological Priors for Probabilistic Neural Word Embeddings

Yik Yak, Inc.（Yik Yak公司）；Georgia Institute of Technology（佐治亚理工学院）

Word embeddings allow natural language processing systems to share statistical information across related words. These embeddings are typically based on distributional statistics, making it difficult for them to generalize to rare or unseen words. We propose to improve word embeddings by incorporating morphological information, capturing shared sub-word features. Unlike previous work that constructs word embeddings directly from morphemes, we combine morphological and distributional information in a unified probabilistic framework, in which the word embedding is a latent variable. The morpholo…

2016-09-24cs.CL

A Joint Model for Word Embedding and Word Morphology

University of Cambridge（剑桥大学）

This paper presents a joint model for performing unsupervised morphological analysis on words, and learning a character-level composition function from morphemes to word embeddings. Our model splits individual words into segments, and weights each segment according to its ability to predict context words. Our morphological analysis is comparable to dedicated morphological analyzers at the task of morpheme boundary recovery, and also performs better than word-based embedding models at the task of syntactic analogy answering. Finally, we show that incorporating morphology explicitly into charact…

2016-06-08cs.CL

形态学信息在上下文词元还原中的作用

On the Role of Morphological Information for Contextual Lemmatization

University of the Basque Country UPV/EHU（巴斯克大学（UPV/EHU））

本文实证研究了形态学信息在六种语言上下文词元还原中的作用，发现细粒度形态特征对训练并非关键，现代上下文词表示已隐式编码足够信息，使用简单UPOS标签或无形态学训练的模型在领域外表现更优。

2023-10-20cs.AIcs.CL

从 Smør-re-brød 到子词：逐个语素训练丹麦语 LLMs

From Smør-re-brød to Subwords: Training LLMs on Danish, One Morpheme at a Time

IT University of Copenhagen（哥本哈根信息技术大学）

本研究利用带标注的丹麦语形态学数据集训练半监督分割模型并开发定制分词器，显著提升了形态学分割准确度及生成式 transformer 模型在丹麦语下游任务中的性能。

2025-04-02cs.CL

更多相关论文收起更多论文

Incorporating Visual Semantics into Sentence Representations within a Grounded Space

Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；LIP6（巴黎第六大学计算机科学实验室）；Criteo AI Lab（Criteo人工智能实验室）

Language grounding is an active field aiming at enriching textual representations with visual information. Generally, textual and visual elements are embedded in the same representation space, which implicitly assumes a one-to-one correspondence between modalities. This hypothesis does not hold when representing words, and becomes problematic when used to learn sentence representations --- the focus of this paper --- as a visual scene can be described by a wide variety of sentences. To overcome this limitation, we propose to transfer visual information to textual representations by learning an…

2020-02-07cs.CL

Hierarchical Multi Task Learning with Subword Contextual Embeddings for Languages with Rich Morphology

University of Tokyo（东京大学）；Bogazici University（海峡大学）

Morphological information is important for many sequence labeling tasks in Natural Language Processing (NLP). Yet, existing approaches rely heavily on manual annotations or external software to capture this information. In this study, we propose using subword contextual embeddings to capture the morphological information for languages with rich morphology. In addition, we incorporate these embeddings in a hierarchical multi-task setting which is not employed before, to the best of our knowledge. Evaluated on Dependency Parsing (DEP) and Named Entity Recognition (NER) tasks, which are shown to …

2020-04-25cs.CLcs.IRcs.LG

Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring

Center for Artificial Intelligence Research (CAiRE)（人工智能研究中心（CAiRE））；The Hong Kong University of Science and Technology（香港科技大学）

This paper describes CAiRE's submission to the unsupervised machine translation track of the WMT'19 news shared task from German to Czech. We leverage a phrase-based statistical machine translation (PBSMT) model and a pre-trained language model to combine word-level neural machine translation (NMT) and subword-level NMT models without using any parallel data. We propose to solve the morphological richness problem of languages by training byte-pair encoding (BPE) embeddings for German and Czech separately, and they are aligned using MUSE (Conneau et al., 2018). To ensure the fluency and consist…

2019-11-18cs.CL

融合语言学与人工智能：西孟加拉邦濒危托托语的形态分析与语料库建设

Integrating Linguistics and AI: Morphological Analysis and Corpus development of Endangered Toto Language of West Bengal

Adamas University（阿达马斯大学）

本研究融合传统语言学与AI，通过田野调查构建托托语语素标注三语语料库，训练小型语言模型和Transformer翻译引擎，并开发三语学习应用，为濒危语言保护提供可持续模型。

2025-10-26cs.AIcs.CL

Do Syntax Trees Help Pre-trained Transformers Extract Information?

Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；School of Computer Science, McGill University（麦吉尔大学计算机学院）；Stanford University（斯坦福大学）

Much recent work suggests that incorporating syntax information from dependency trees can improve task-specific transformer models. However, the effect of incorporating dependency tree information into pre-trained transformer models (e.g., BERT) remains unclear, especially given recent studies highlighting how these models implicitly encode syntax. In this work, we systematically study the utility of incorporating dependency trees into pre-trained transformers on three representative information extraction tasks: semantic role labeling (SRL), named entity recognition, and relation extraction. …

2021-01-27cs.CL

挑选未充分利用的注意力头：面向对话共指信息融合的注意力头选择的网络剪枝视角

Picking the Underused Heads: A Network Pruning Perspective of Attention Head Selection for Fusing Dialogue Coreference Information

Institute for Infocomm Research, A*STAR（资讯通信研究院，新加坡科技研究局）

该研究从网络剪枝视角提出基于层级重要度的注意力头选择策略，通过操控未充分利用的头注入结构感知特征，在对话摘要任务中融入共指信息实现了性能提升，无需额外增加神经组件与参数量。

2023-12-15cs.CL

融合词汇与句法知识的无监督跨语言迁移

Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer

Tsinghua University（清华大学）；Beijing Ultrapower Software Co.,Ltd.（北京神州泰岳软件股份有限公司）

针对无监督跨语言迁移中词汇与句法知识未能结合的问题，提出融合两者的 Lexicon-Syntax Enhanced Multilingual BERT 框架，通过代码切换和句法图注意力网络增强 mBERT，在多项零样本任务上持续优于基线。

2024-04-25cs.CL

Modeling Order in Neural Word Embeddings at Scale

Digital Reasoning Systems, Inc.（数字推理系统公司）

Natural Language Processing (NLP) systems commonly leverage bag-of-words co-occurrence techniques to capture semantic and syntactic word relationships. The resulting word-level distributed representations often ignore morphological information, though character-level embeddings have proven valuable to NLP tasks. We propose a new neural language model incorporating both word order and character order in its embedding. The model produces several vector spaces with meaningful substructure, as evidenced by its performance of 85.8% on a recent word-analogy task, exceeding best published syntactic w…

2015-06-11cs.CL

Relational Word Embeddings

Cardiff University（卡迪夫大学）

While word embeddings have been shown to implicitly encode various forms of attributional knowledge, the extent to which they capture relational information is far more limited. In previous work, this limitation has been addressed by incorporating relational knowledge from external knowledge bases when learning the word embedding. Such strategies may not be optimal, however, as they are limited by the coverage of available resources and conflate similarity with other forms of relatedness. As an alternative, in this paper we propose to encode relational knowledge in a separate word embedding, w…

2019-06-04cs.CL

从文本到预测：通过时间演化的语义空间弥合模态差距

From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space

本文提出TESS方法，通过引入时间演化语义空间弥合文本与预测模型间的模态差距，实验显示在四个真实世界数据集上预测误差降低29%。

2026-03-16cs.AIcs.CL