Preprocessing tools for automated text analysis have become more widely available in major languages, but non-English tools are often still limited in their functionality. When working with Spanish-language text, researchers can easily find tools for tokenization and stemming, but may not have the means to extract more complex word features like verb tense or mood. Yet Spanish is a morphologically rich language in which such features are often identifiable from word form. Conjugation rules are consistent, but many special verbs and nouns take on different rules. While building a complete dictionary of known words and their morphological rules would be labor intensive, resources to do so already exist, in spell checkers designed to generate valid forms of known words. This paper introduces a set of tools for Spanish-language morphological analysis, built using the COES spell checking tools, to label person, mood, tense, gender and number, derive a word's root noun or verb infinitive, and convert verbs to their nominal form.
Insight Centre for Data Analytics（数据分析洞察中心）；National University of Ireland Galway（爱尔兰国立大学戈尔韦分校）
Spell checking and morphological analysis are two fundamental tasks in text and natural language processing and are addressed in the early stages of the development of language technology. Despite the previous efforts, there is no progress in open-source to create such tools for Sorani Kurdish, also known as Central Kurdish, as a less-resourced language. In this paper, we present our efforts in annotating a lexicon with morphosyntactic tags and also, extracting morphological rules of Sorani Kurdish to build a morphological analyzer, a stemmer and a spell-checking system using Hunspell. This im…
Universidad Politécnica de Madrid（马德里理工大学）；Telefónica（西班牙电信公司）；IFEMA（马德里展览中心）；Universidad Oberta De Catalunya（加泰罗尼亚开放大学）
Text classification tasks have improved substantially during the last years by the usage of transformers. However, the majority of researches focus on prose texts, with poetry receiving less attention, specially for Spanish language. In this paper, we propose a semi-supervised learning approach for inferring 21 psychological categories evoked by a corpus of 4572 sonnets, along with 10 affective and lexico-semantic multiclass ones. The subset of poems used for training an evaluation includes 270 sonnets. With our approach, we achieve an AUC beyond 0.7 for 76% of the psychological categories, an…
Semantic neologisms (SN) are defined as words that acquire a new word meaning while maintaining their form. Given the nature of this kind of neologisms, the task of identifying these new word meanings is currently performed manually by specialists at observatories of neology. To detect SN in a semi-automatic way, we developed a system that implements a combination of the following strategies: topic modeling, keyword extraction, and word sense disambiguation. The role of topic modeling is to detect the themes that are treated in the input text. Themes within a text give clues about the particul…
University of Kurdistan（库尔德斯坦大学）；Paris Descartes University（巴黎笛卡尔大学）
The present paper aims at presenting a lemmatization and a word-level error correction system for Sorani Kurdish. We propose a hybrid approach based on the morphological rules and a n-gram language model. We have called our lemmatization and error correction systems Peyv and Rênûs respectively, which are the first tools presented for Sorani Kurdish to the best of our knowledge. The Peyv lemmatizer has shown 86.7% accuracy. As for Rênûs, using a lexicon, we have obtained 96.4% accuracy while without a lexicon, the correction system has 87% accuracy. As two fundamental text processing tools, the…
University of Rochester（罗切斯特大学）；Department of Computer Science（计算机科学系）
Recent embedding-based methods in unsupervised bilingual lexicon induction have shown good results, but generally have not leveraged orthographic (spelling) information, which can be helpful for pairs of related languages. This work augments a state-of-the-art method with orthographic features, and extends prior work in this space by proposing methods that can learn and utilize orthographic correspondences even between languages with different scripts. We demonstrate this by experimenting on three language pairs with different scripts and varying degrees of lexical similarity.
The IMPACT-es diachronic corpus of historical Spanish compiles over one hundred books --containing approximately 8 million words-- in addition to a complementary lexicon which links more than 10 thousand lemmas with attestations of the different variants found in the documents. This textual corpus and the accompanying lexicon have been released under an open license (Creative Commons by-nc-sa) in order to permit their intensive exploitation in linguistic research. Approximately 7% of the words in the corpus (a selection aimed at enhancing the coverage of the most frequent word forms) have been…
Speech recognition systems for irregularly-spelled languages like English normally require hand-written pronunciations. In this paper, we describe a system for automatically obtaining pronunciations of words for which pronunciations are not available, but for which transcribed data exists. Our method integrates information from the letter sequence and from the acoustic evidence. The novel aspect of the problem that we address is the problem of how to prune entries from such a lexicon (since, empirically, lexicons with too many entries do not tend to be good for ASR performance). Experiments on…
Language Resources in Spanish for Automatic Text Simplification across Domains
Universidad Autónoma de Madrid（马德里自治大学）；Consejo Superior de Investigaciones Científicas（西班牙高等科学研究理事会）；Universidad Nacional de Educación a Distancia（国立远程教育大学）
In this work we focus on the task of automatically extracting bilingual lexicon for the language pair Spanish-Nahuatl. This is a low-resource setting where only a small amount of parallel corpus is available. Most of the downstream methods do not work well under low-resources conditions. This is specially true for the approaches that use vectorial representations like Word2Vec. Our proposal is to construct bilingual word vectors from a graph. This graph is generated using translation pairs obtained from an unsupervised word alignment method. We show that, in a low-resource setting, these type …
University of Cambridge（剑桥大学）；National University of Ireland（爱尔兰国立大学）；Facebook AI Research（Facebook AI研究院）
Human translators routinely have to translate rare inflections of words - due to the Zipfian distribution of words in a language. When translating from Spanish, a good translator would have no problem identifying the proper translation of a statistically rare inflection such as habláramos. Note the lexeme itself, hablar, is relatively common. In this work, we investigate whether state-of-the-art bilingual lexicon inducers are capable of learning this kind of generalization. We introduce 40 morphologically complete dictionaries in 10 languages and evaluate three of the state-of-the-art models o…
Nigerian English adaptation, Pidgin, has evolved over the years through multi-language code switching, code mixing and linguistic adaptation. While Pidgin preserves many of the words in the normal English language corpus, both in spelling and pronunciation, the fundamental meaning of these words have changed significantly. For example,'ginger' is not a plant but an expression of motivation and 'tank' is not a container but an expression of gratitude. The implication is that the current approach of using direct English sentiment analysis of social media text from Nigeria is sub-optimal, as it w…
Resource-Lean Lexicon Induction for German Dialects
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany（MaiNLP，信息与语言处理中心，慕尼黑大学，德国）；Munich Center for Machine Learning (MCML), Munich, Germany（慕尼黑机器学习中心（MCML），慕尼黑，德国）
This paper deals with the discovery, representation, and use of lexical rules (LRs) during large-scale semi-automatic computational lexicon acquisition. The analysis is based on a set of LRs implemented and tested on the basis of Spanish and English business- and finance-related corpora. We show that, though the use of LRs is justified, they do not come cost-free. Semi-automatic output checking is required, even with blocking and preemtion procedures built in. Nevertheless, large-scope LRs are justified because they facilitate the unavoidable process of large-scale semi-automatic lexical acqui…
A New NMT Model for Translating Clinical Texts from English to Spanish
College of Information and Computer Science, University of Massachusetts Amherst（马萨诸塞大学阿姆赫斯特信息与计算机科学学院）；Department of Computer Sciences, University of Massachusetts Lowell（马萨诸塞大学洛约拉计算机科学系）；Department of Quantitative Health Sciences, University of Massachusetts Medical School（马萨诸塞大学医学学院定量健康科学系）；Bedford VAMC and CHOIR（贝德福德退伍军人医疗中心和CHOIR）
Sentiment analysis is a widely studied NLP task where the goal is to determine opinions, emotions, and evaluations of users towards a product, an entity or a service that they are reviewing. One of the biggest challenges for sentiment analysis is that it is highly language dependent. Word embeddings, sentiment lexicons, and even annotated data are language specific. Further, optimizing models for each language is very time consuming and labor intensive especially for recurrent neural network models. From a resource perspective, it is very challenging to collect data for different languages. In…
A model for the full treatment of Spanish inflection for verbs, nouns and adjectives is presented. This model is based on feature unification and it relies upon a lexicon of allomorphs both for stems and morphemes. Word forms are built by the concatenation of allomorphs by means of special contextual features. We make use of standard Definite Clause Grammars (DCG) included in most Prolog implementations, instead of the typical finite-state approach. This allows us to take advantage of the declarativity and bidirectionality of Logic Programming for NLP. The most salient feature of this approach…
Lexicon-based methods using syntactic rules for polarity classification rely on parsers that are dependent on the language and on treebank guidelines. Thus, rules are also dependent and require adaptation, especially in multilingual scenarios. We tackle this challenge in the context of the Iberian Peninsula, releasing the first symbolic syntax-based Iberian system with rules shared across five official languages: Basque, Catalan, Galician, Portuguese and Spanish. The model is made available.
Multilingual training of neural machine translation (NMT) systems has led to impressive accuracy improvements on low-resource languages. However, there are still significant challenges in efficiently learning word representations in the face of paucity of data. In this paper, we propose Soft Decoupled Encoding (SDE), a multilingual lexicon encoding framework specifically designed to share lexical-level information intelligently without requiring heuristic preprocessing such as pre-segmenting the data. SDE represents a word by its spelling through a character encoding, and its semantic meaning …
Language in social media is mostly driven by new words and spellings that are constantly entering the lexicon thereby polluting it and resulting in high deviation from the formal written version. The primary entities of such language are the out-of-vocabulary (OOV) words. In this paper, we study various sociolinguistic properties of the OOV words and propose a classification model to categorize them into at least six categories. We achieve 81.26% accuracy with high precision and recall. We observe that the content features are the most discriminative ones followed by lexical and context featur…
University of Buenos Aires（布宜诺斯艾利斯大学）；Physics Institute of Buenos Aires (IFIBA) CONICET（布宜诺斯艾利斯物理研究所（IFIBA）CONICET）；Torcuato Di Tella University（托库阿托·迪·特莱拉大学）
Language change involves the competition between alternative linguistic forms (1). The spontaneous evolution of these forms typically results in monotonic growths or decays (2, 3) like in winner-take-all attractor behaviors. In the case of the Spanish past subjunctive, the spontaneous evolution of its two competing forms (ended in -ra and -se) was perturbed by the appearance of the Royal Spanish Academy in 1713, which enforced the spelling of both forms as perfectly interchangeable variants (4), at a moment in which the -ra form was dominant (5). Time series extracted from a massive corpus of …
Facultad de Ciencias, Universidad Nacional Autónoma de México（墨西哥国立自治大学理学院）；Instituto de Física, Universidad Nacional Autónoma de México（墨西哥国立自治大学物理研究所）；Instituto de Investigaciones en Matemáticas Aplicadas y Sistemas, Universidad Nacional Autónoma de México（墨西哥国立自治大学应用数学与系统研究所）；Centro de Ciencias de la Complejidad, Universidad Nacional Autónoma de México（墨西哥国立自治大学复杂性科学中心）；Santa Fe Institute（圣塔菲研究所）
A recent increase in data availability has allowed the possibility to perform different statistical linguistic studies. Here we use the Google Books Ngram dataset to analyze word flow among English, French, German, Italian, and Spanish. We study what we define as ``migrant words'', a type of loanwords that do not change their spelling. We quantify migrant words from one language to another for different decades, and notice that most migrant words can be aggregated in semantic fields and associated to historic events. We also study the statistical properties of accumulated migrant words and the…
How many words does ChatGPT know? The answer is ChatWords
Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Universidad Politécnica de Madrid（马德里理工大学）；Universidad de Valladolid（巴利亚多利德大学）；Northeastern University（东北大学）
