Parallel text is required for building high-quality machine translation (MT) systems, as well as for other multilingual NLP applications. For many South Asian languages, such data is in short supply. In this paper, we described a new publicly available corpus (PMIndia) consisting of parallel sentences which pair 13 major languages of India with English. The corpus includes up to 56000 sentences for each language pair. We explain how the corpus was constructed, including an assessment of two different automatic sentence alignment methods, and present some initial NMT results on the corpus.
International Institute of Information Technology - Hyderabad（国际信息技术研究所海得拉巴分校）；Indian Institute of Technology - Kanpur（印度理工学院坎普尔分校）
We present sentence aligned parallel corpora across 10 Indian Languages - Hindi, Telugu, Tamil, Malayalam, Gujarati, Urdu, Bengali, Oriya, Marathi, Punjabi, and English - many of which are categorized as low resource. The corpora are compiled from online sources which have content shared across languages. The corpora presented significantly extends present resources that are either not large enough or are restricted to a specific domain (such as health). We also provide a separate test corpus compiled from an independent online source that can be independently used for validating the performan…
IIIT Hyderabad（国际信息技术研究所海得拉巴分校）；University of Bath（巴斯大学）
Indian language machine translation performance is hampered due to the lack of large scale multi-lingual sentence aligned corpora and robust benchmarks. Through this paper, we provide and analyse an automated framework to obtain such a corpus for Indian language neural machine translation (NMT) systems. Our pipeline consists of a baseline NMT system, a retrieval module, and an alignment module that is used to work with publicly available websites such as press releases by the government. The main contribution towards this effort is to obtain an incremental method that uses the above pipeline t…
In this paper, we address the task of improving pair-wise machine translation for specific low resource Indian languages. Multilingual NMT models have demonstrated a reasonable amount of effectiveness on resource-poor languages. In this work, we show that the performance of these models can be significantly improved upon by using back-translation through a filtered back-translation process and subsequent fine-tuning on the limited pair-wise language corpora. The analysis in this paper suggests that this method can significantly improve a multilingual model's performance over its baseline, yiel…
Shallow parsing is an essential task for many NLP applications like machine translation, summarization, sentiment analysis, aspect identification and many more. Quality annotated corpora is critical for building accurate shallow parsers. Many Indian languages are resource poor with respect to the availability of corpora in general. So, this paper is an attempt towards creating quality corpora for shallow parsers. The contribution of this paper is two folds: creation pos and chunk annotated corpora for Odia and development of baseline systems for pos tagging and chunking in Odia.
Google（谷歌）；Indian Institute of Technology, Patna（印度理工学院巴特那分校）；Indian Institute of Technology, Bombay（印度理工学院孟买分校）；Delhi Technological University（德里理工大学）
India is a multilingual society with 1369 rationalized languages and dialects being spoken across the country (INDIA, 2011). Of these, the 22 scheduled languages have a staggering total of 1.17 billion speakers and 121 languages have more than 10,000 speakers (INDIA, 2011). India also has the second largest (and an ever growing) digital footprint (Statista, 2020). Despite this, today's state-of-the-art multilingual systems perform suboptimally on Indian (IN) languages. This can be explained by the fact that multilingual language models (LMs) are often trained on 100+ languages together, leadin…
Microsoft India（微软印度）；Indian Institute of Technology Bombay（印度理工学院孟买分校）
In this work, we present an extensive study of statistical machine translation involving languages of the Indian subcontinent. These languages are related by genetic and contact relationships. We describe the similarities between Indic languages arising from these relationships. We explore how lexical and orthographic similarity among these languages can be utilized to improve translation quality between Indic languages when limited parallel corpora is available. We also explore how the structural correspondence between Indic languages can be utilized to re-use linguistic resources for English…
National Institute of Technology Karnataka（卡纳塔克邦国立理工学院）；Indian Institute of Technology Guwahati（印度理工学院古瓦哈提分校）
Developing Named Entity Recognition (NER) systems for Indian languages has been a long-standing challenge, mainly owing to the requirement of a large amount of annotated clean training instances. This paper proposes an end-to-end framework for NER for Indian languages in a low-resource setting by exploiting parallel corpora of English and Indian languages and an English NER dataset. The proposed framework includes an annotation projection method that combines word alignment score and NER tag prediction confidence score on source language (English) data to generate weakly labeled data in a targ…
CorIL: Towards Enriching Indian Language to Indian Language Parallel Corpora and Machine Translation Systems
Department of Computer Science and Engineering, Indian Institute of Technology Patna（计算机科学与工程系，印度理工学院帕纳布）；SNLP Lab, CDAC Noida（SNLP实验室，CDAC诺伊达）；Department of Computer Science and Engineering, Manipal Institute of Technology（计算机科学与工程系，曼尼普尔理工学院）；Department of Computer Science and Engineering, Dhirubhai Ambani University, Gandhinagar（计算机科学与工程系，迪鲁布希阿米大学，甘地 Nagar）；LTRC, IIIT Hyderabad（LTRC，IIIT海得拉尔）；Department of CSE, IIIT Bhubaneshwar（CSE系，IIIT布巴内什瓦尔）；Department of AI, SVNIT, Surat（人工智能系，SVNIT，萨拉特）；Department of Linguistics, University of Kashmir（语言学系，克什米尔大学）；VLSI Design Group, CDAC Bangalore（VLSI设计组，CDAC班加罗尔）；Department of CSE, Punjabi University（CSE系，旁遮普大学）；Department of CSE, Govt. College for Women Jammu（CSE系，贾姆穆政府女子学院）
Indian Institute of Technology Bombay（印度理工学院孟买分校）；IITB-Monash Research Academy（印度理工学院孟买分校-莫纳什大学联合研究学院）；Monash University（莫纳什大学）
Automatic Cognate Detection (ACD) is a challenging task which has been utilized to help NLP applications like Machine Translation, Information Retrieval and Computational Phylogenetics. Unidentified cognate pairs can pose a challenge to these applications and result in a degradation of performance. In this paper, we detect cognate word pairs among ten Indian languages with Hindi and use deep learning methodologies to predict whether a word pair is cognate or not. We identify IndoWordnet as a potential resource to detect cognate word pairs based on orthographic similarity-based methods and trai…
Developing parallel corpora is an important and a difficult activity for Machine Translation. This requires manual annotation by Human Translators. Translating same text again is a useless activity. There are tools available to implement this for European Languages, but no such tool is available for Indian Languages. In this paper we present a tool for Indian Languages which not only provides automatic translations of the previously available translation but also provides multiple translations, in cases where a sentence has multiple translations, in ranked list of suggestive translations for a…
IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
Nilekani Centre at AI4Bharat（AI4Bharat 尼勒卡尼中心）；Indian Institute of Technology Madras（马德拉斯印度理工学院）；Microsoft（微软）；National Institute of Information and Communications Technology（信息通信研究机构）；Flipkart（弗利普卡特）
Microsoft Research（微软研究院）；Delhi Technological University（德里理工大学）；University of Utah（犹他大学）
While Indic NLP has made rapid advances recently in terms of the availability of corpora and pre-trained models, benchmark datasets on standard NLU tasks are limited. To this end, we introduce IndicXNLI, an NLI dataset for 11 Indic languages. It has been created by high-quality machine translation of the original English XNLI dataset and our analysis attests to the quality of IndicXNLI. By finetuning different pre-trained LMs on this IndicXNLI, we analyze various cross-lingual transfer techniques with respect to the impact of the choice of language models, languages, multi-linguality, mix-lang…
Indian Institute of Technology Bombay（孟买印度理工学院）；Google Research（谷歌研究院）
Recent research in multilingual language models (LM) has demonstrated their ability to effectively handle multiple languages in a single model. This holds promise for low web-resource languages (LRL) as multilingual models can enable transfer of supervision from high resource languages to LRLs. However, incorporating a new language in an LM still remains a challenge, particularly for languages with limited corpora and in unseen scripts. In this paper we argue that relatedness among languages in a language family may be exploited to overcome some of the corpora limitations of LRLs, and propose …
First Attempt at Building Parallel Corpora for Machine Translation of Northeast India's Very Low-Resource Languages
University of Colorado, Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）；Instituto Politécnico Nacional（墨西哥国立理工学院）；Lelapa AI（莱拉帕人工智能公司）；Palmer Ridge High School（帕默里奇高中）
CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation
Nilekani Centre at AI4Bharat（AI4Bharat的Nilekani中心）；Indian Institute of Technology（印度理工学院）；Indian Institute of Technology Bombay（印度理工学院孟买分校）；Indian Institute of Engineering Science and Technology Shibpur（Shibpur的印度工程科学与技术学院）
Improving Multilingual Neural Machine Translation System for Indic Languages
National Institute of Technology (NIT), Rourkela（国家理工学院（鲁尔凯拉））；Pune Institute of Computer Technology (PICT)（浦那计算机技术学院）；Indian Institute of Technology (IIT), Varanasi（印度理工学院（瓦拉纳西））
This paper presents the challenges in creating and managing large parallel corpora of 12 major Indian languages (which is soon to be extended to 23 languages) as part of a major consortium project funded by the Department of Information Technology (DIT), Govt. of India, and running parallel in 10 different universities of India. In order to efficiently manage the process of creation and dissemination of these huge corpora, the web-based (with a reduced stand-alone version also) annotation tool ILCIANN (Indian Languages Corpora Initiative Annotation Tool) has been developed. It was primarily de…
Silo AI；University of Lille（里尔大学）；Institute of Mathematics and Applications（数学与应用研究所）；KIIT University（KIIT大学）；Ravenshaw University（雷文肖大学）
This paper presents the first publicly available treebank of Odia, a morphologically rich low resource Indian language. The treebank contains approx. 1082 tokens (100 sentences) in Odia selected from "Samantar", the largest available parallel corpora collection for Indic languages. All the selected sentences are manually annotated following the ``Universal Dependency (UD)" guidelines. The morphological analysis of the Odia treebank was performed using machine learning techniques. The Odia annotated treebank will enrich the Odia language resource and will help in building language technology to…
Phrase-based Statistical models are more commonly used as they perform optimally in terms of both, translation quality and complexity of the system. Hindi and in general all Indian languages are morphologically richer than English. Hence, even though Phrase-based systems perform very well for the less divergent language pairs, for English to Indian language translation, we need more linguistic information (such as morphology, parse tree, parts of speech tags, etc.) on the source side. Factored models seem to be useful in this case, as Factored models consider word as a vector of factors. These…
BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India（印度理工学院帕纳加邦计算机科学与工程系）；Indian Institute of Technology Patna, India（印度理工学院帕纳加邦）
Dr. Bhim Rao Ambedkar University（比姆·拉奥·安贝德卡尔博士大学）；Jadavpur University（贾达普大学）；Rutgers University（罗格斯大学）；Jawaharlal Nehru University（贾瓦哈拉尔·尼赫鲁大学）
In this paper, we discuss an attempt to develop an automatic language identification system for 5 closely-related Indo-Aryan languages of India, Awadhi, Bhojpuri, Braj, Hindi and Magahi. We have compiled a comparable corpora of varying length for these languages from various resources. We discuss the method of creation of these corpora in detail. Using these corpora, a language identification system was developed, which currently gives state of the art accuracy of 96.48\%. We also used these corpora to study the similarity between the 5 languages at the lexical level, which is the first data-b…
Jules Bloch's work on formation of the Marathi language has to be expanded further to provide for a study of evolution and formation of Indian languages in the Indian language union (sprachbund). The paper analyses the stages in the evolution of early writing systems which began with the evolution of counting in the ancient Near East. A stage anterior to the stage of syllabic representation of sounds of a language, is identified. Unique geometric shapes required for tokens to categorize objects became too large to handle to abstract hundreds of categories of goods and metallurgical processes d…
Microsoft India（微软印度）；IIT Madras（马德拉斯印度理工学院）；AI4Bharat
We present the IndicNLP corpus, a large-scale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. We share pre-trained word embeddings trained on these corpora. We create news article category classification datasets for 9 languages to evaluate the embeddings. We show that the IndicNLP embeddings significantly outperform publicly available pre-trained embedding on multiple evaluation tasks. We hope that the availability of the corpus will accelerate Indic NLP research. The resources are available at https://github.com/ai4bharat-indicnlp/indic…
Building Part-of-Speech (POS) taggers for code-mixed Indian languages is a particularly challenging problem in computational linguistics due to a dearth of accurately annotated training corpora. ICON, as part of its NLP tools contest has organized this challenge as a shared task for the second consecutive year to improve the state-of-the-art. This paper describes the POS tagger built at Surukam to predict the coarse-grained and fine-grained POS tags for three language pairs - Bengali-English, Telugu-English and Hindi-English, with the text spanning three popular social media platforms - Facebo…
Indian Institute of Technology Kharagpur（印度理工学院克勒格布尔分校）；Université de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；LORIA（洛林信息学及其应用实验室）
This work addresses the cross-corpora generalization issue for the low-resourced spoken language identification (LID) problem. We have conducted the experiments in the context of Indian LID and identified strikingly poor cross-corpora generalization due to corpora-dependent non-lingual biases. Our contribution to this work is twofold. First, we propose domain diversification, which diversifies the limited training data using different audio data augmentation methods. We then propose the concept of maximally diversity-aware cascaded augmentations and optimize the augmentation fold-factor for ef…
The text generated on social media platforms is essentially a mixed lingual text. The mixing of language in any form produces considerable amount of difficulty in language processing systems. Moreover, the advancements in language processing research depends upon the availability of standard corpora. The development of mixed lingual Indian Named Entity Recognition (NER) systems are facing obstacles due to unavailability of the standard evaluation corpora. Such corpora may be of mixed lingual nature in which text is written using multiple languages predominantly using a single script only. The …
Recent research has revealed undesirable biases in NLP data and models. However, these efforts largely focus on social disparities in the West, and are not directly portable to other geo-cultural contexts. In this position paper, we outline a holistic research agenda to re-contextualize NLP fairness research for the Indian context, accounting for Indian societal context, bridging technological gaps in capability and resources, and adapting to Indian cultural values. We also summarize findings from an empirical study on various social biases along different axes of disparities relevant to India…
Language Identification is a very important part of several text processing pipelines. Extensive research has been done in this field. This paper proposes a procedure for automatic language identification of poems for poem analysis task, consisting of 10 Devanagari based languages of India i.e. Angika, Awadhi, Braj, Bhojpuri, Chhattisgarhi, Garhwali, Haryanvi, Hindi, Magahi, and Maithili. We collated corpora of poems of varying length and studied the similarity of poems among the 10 languages at the lexical level. Finally, various language identification systems based on supervised machine lea…
Indian Institute of Technology (BHU), India（印度理工学院（巴纳拉斯印度教大学））；Alexa AI, Amazon, India（亚马逊Alexa AI（印度））；Samsung Research Institute, India（三星研究院（印度））
Corpus preparation for low-resource languages and for development of human language technology to analyze or computationally process them is a laborious task, primarily due to the unavailability of expert linguists who are native speakers of these languages and also due to the time and resources required. Bhojpuri, Magahi, and Maithili, languages of the Purvanchal region of India (in the north-eastern parts), are low-resource languages belonging to the Indo-Aryan (or Indic) family. They are closely related to Hindi, which is a relatively high-resource language, which is why we compare with Hin…
Multilingual and Explainable Text Detoxification with Parallel Corpora
Technical University of Munich（慕尼黑工业大学）；Universidade de Santiago de Compostela（圣地亚哥-德孔波斯特拉大学）；Shenkar College（申卡尔学院）；University of Hamburg（汉堡大学）；Bahir Dar University（巴达尔大学）；IIT Kharagpur（印度理工学院克勒格布尔分校）；Skoltech（斯科尔科沃科技学院）；AIRI（人工智能研究所）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Sharjah（沙迦大学）
Government Engineering College（政府工程学院）；A P J Abdul Kalam Technological University（阿卜杜勒·卡拉姆技术大学）；IIIT（印度信息技术学院）；State Project Facilitation Unit（邦项目促进单位）
Modern text-to-speech (TTS) systems use deep learning to synthesize speech increasingly approaching human quality, but they require a database of high quality audio-text sentence pairs for training. Malayalam, the official language of the Indian state of Kerala and spoken by 35+ million people, is a low resource language in terms of available corpora for TTS systems. In this paper, we present IMaSC, a Malayalam text and speech corpora containing approximately 50 hours of recorded speech. With 8 speakers and a total of 34,473 text-audio pairs, IMaSC is larger than every other publicly available…
IKS-Instruct: A 24,000-Example Multilingual Dataset for Teaching Language Models Indian Knowledge Systems
Intrinsic Lab, Centre for Sensors, Instrumentation and Cyber-Physical System Engineering (Centre for SeNSE), Indian Institute of Technology Delhi（印度理工学院德里分校传感器、仪器仪表和网络物理系统工程中心（传感中心）本征实验室）；RSL Quantum, FITT, IIT Delhi（印度理工学院德里分校 FITT 学院 RSL 量子实验室）
University of Electronic Science and Technology of China（电子科技大学）
Representing words and phrases into dense vectors of real numbers which encode semantic and syntactic properties is a vital constituent in natural language processing (NLP). The success of neural network (NN) models in NLP largely rely on such dense word representations learned on the large unlabeled corpus. Sindhi is one of the rich morphological language, spoken by large population in Pakistan and India lacks corpora which plays an essential role of a test-bed for generating word embeddings and developing language independent NLP systems. In this paper, a large corpus of more than 61 million…
