Language models based on the Transformer architecture have achieved state-of-the-art performance on a wide range of NLP tasks such as text classification, question-answering, and token classification. However, this performance is usually tested and reported on high-resource languages, like English, French, Spanish, and German. Indian languages, on the other hand, are underrepresented in such benchmarks. Despite some Indian languages being included in training multilingual Transformer models, they have not been the primary focus of such work. In order to evaluate the performance on Indian languages specifically, we analyze these language models through extensive experiments on multiple downstream tasks in Hindi, Bengali, and Telugu language. Here, we compare the efficacy of fine-tuning model parameters of pre-trained models against that of training a language model from scratch. Moreover, we empirically argue against the strict dependency between the dataset size and model performance, but rather encourage task-specific model and method selection. We achieve state-of-the-art performance on Hindi and Bengali languages for text classification task. Finally, we present effective strategies for handling the modeling of Indian languages and we release our model checkpoints for the community : https://huggingface.co/neuralspace-reverie.
L3Cube-IndicHeadline-ID: A Dataset for Headline Identification and Semantic Evaluation in Low-Resource Indian Languages
Department of Information Technology, PICT, Pune（信息科技系，PICT，普那）；Indian Institute of Technology Madras, Chennai（印度理工学院马德拉斯分校，钦奈）；L3Cube Labs, Pune（L3Cube实验室，普那）
Peoples' Friendship University of Russia (RUDN University)（俄罗斯人民友谊大学（RUDN大学））；Joint Institute for Nuclear Research（联合核研究所）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Skövde（舍夫德大学）；International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术学院）
With the advent of multilingual models like mBART, mT5, IndicBART etc., summarization in low resource Indian languages is getting a lot of attention now a days. But still the number of datasets is low in number. In this work, we (Team HakunaMatata) study how these multilingual models perform on the datasets which have Indian languages as source and target text while performing summarization. We experimented with IndicBART and mT5 models to perform the experiments and report the ROUGE-1, ROUGE-2, ROUGE-3 and ROUGE-4 scores as a performance metric.
Harnessing Pre-Trained Sentence Transformers for Offensive Language Detection in Indian Languages
MKSSS Cummins College of Engineering for Women（MKSSS卡明斯女子工程学院）；Indian Institute of Technology Madras（印度理工学院马德拉斯分校）；Peoples' Friendship University of Russia (RUDN University)（俄罗斯人民友谊大学（RUDN大学））；Joint Institute for Nuclear Research（联合核研究所）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Skövde（舍夫德大学）
IIT Kharagpur（印度理工学院卡哈拉格普尔分校）；University of Maryland, College Park（马里兰大学帕克分校）；Carnegie Mellon University（卡内基梅隆大学）
Scarcity of data and technological limitations for resource-poor languages in developing countries like India poses a threat to the development of sophisticated NLU systems for healthcare. To assess the current status of various state-of-the-art language models in healthcare, this paper studies the problem by initially proposing two different Healthcare datasets, Indian Healthcare Query Intent-WebMD and 1mg (IHQID-WebMD and IHQID-1mg) and one real world Indian hospital query data in English and multiple Indic languages (Hindi, Bengali, Tamil, Telugu, Marathi and Gujarati) which are annotated w…
Transformers have recently become very popular for sequence-to-sequence applications such as machine translation and speech recognition. In this work, we propose a multi-task learning-based transformer model for low-resource multilingual speech recognition for Indian languages. Our proposed model consists of a conformer [1] encoder and two parallel transformer decoders. We use a phoneme decoder (PHN-DEC) for the phoneme recognition task and a grapheme decoder (GRP-DEC) to predict grapheme sequence. We consider the phoneme recognition task as an auxiliary task for our multi-task learning framew…
SCTR's Pune Institute of Computer Technology（SCTR浦那计算机技术学院）；Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
The research on text summarization for low-resource Indian languages has been limited due to the availability of relevant datasets. This paper presents a summary of various deep-learning approaches used for the ILSUM 2022 Indic language summarization datasets. The ISUM 2022 dataset consists of news articles written in Indian English, Hindi, and Gujarati respectively, and their ground-truth summarizations. In our work, we explore different pre-trained seq2seq models and fine-tune those with the ILSUM 2022 datasets. In our case, the fine-tuned SoTA PEGASUS model worked the best for English, the …
Microsoft India（微软印度）；Indian Institute of Technology Bombay（印度理工学院孟买分校）
In this work, we present an extensive study of statistical machine translation involving languages of the Indian subcontinent. These languages are related by genetic and contact relationships. We describe the similarities between Indic languages arising from these relationships. We explore how lexical and orthographic similarity among these languages can be utilized to improve translation quality between Indic languages when limited parallel corpora is available. We also explore how the structural correspondence between Indic languages can be utilized to re-use linguistic resources for English…
There have been very few attempts to benchmark performances of state-of-the-art algorithms for Neural Machine Translation task on Indian Languages. Google, Bing, Facebook and Yandex are some of the very few companies which have built translation systems for few of the Indian Languages. Among them, translation results from Google are supposed to be better, based on general inspection. Bing-Translator do not even support Marathi language which has around 95 million speakers and ranks 15th in the world in terms of combined primary and secondary speakers. In this exercise, we trained and compared …
Microsoft Research（微软研究院）；Delhi Technological University（德里理工大学）；University of Utah（犹他大学）
While Indic NLP has made rapid advances recently in terms of the availability of corpora and pre-trained models, benchmark datasets on standard NLU tasks are limited. To this end, we introduce IndicXNLI, an NLI dataset for 11 Indic languages. It has been created by high-quality machine translation of the original English XNLI dataset and our analysis attests to the quality of IndicXNLI. By finetuning different pre-trained LMs on this IndicXNLI, we analyze various cross-lingual transfer techniques with respect to the impact of the choice of language models, languages, multi-linguality, mix-lang…
Transformer based language models have led to impressive results across all domains in Natural Language Processing. Pretraining these models on language modeling tasks and finetuning them on downstream tasks such as Text Classification, Question Answering and Neural Machine Translation has consistently shown exemplary results. In this work, we propose a Multitask Finetuning methodology which combines the Bilingual Machine Translation task with an auxiliary Causal Language Modeling task to improve performance on the former task on Indian Languages. We conduct an empirical study on three languag…
Document summarization aims to create a precise and coherent summary of a text document. Many deep learning summarization models are developed mainly for English, often requiring a large training corpus and efficient pre-trained language models and tools. However, English summarization models for low-resource Indian languages are often limited by rich morphological variation, syntax, and semantic differences. In this paper, we propose GAE-ISumm, an unsupervised Indic summarization model that extracts summaries from text documents. In particular, our proposed model, GAE-ISumm uses Graph Autoenc…
Indic-TunedLens: Interpreting Multilingual Models in Indian Languages
Dwarkadas Jivanlal Sanghvi College of Engineering（达沃拉斯·吉文拉尔工程学院）；Indian Institute of Technology Jodhpur（印度理工学院乔浦尔分校）；King’s College London（伦敦国王学院）；Indian Institute of Technology Patna（印度理工学院帕特纳分校）
Text Classification is an integral part of many Natural Language Processing tasks such as sarcasm detection, sentiment analysis and many more such applications. Many e-commerce websites, social-media/entertainment platforms use such models to enhance user-experience to generate traffic and thus, revenue on their platforms. In this paper, we are presenting our solution to Multilingual Abusive Comment Identification Problem on Moj, an Indian video-sharing social networking service, powered by ShareChat. The problem dealt with detecting abusive comments, in 13 regional Indic languages such as Hin…
WordAlchemy: A transformer-based Reverse Dictionary
Dept. of Computer Engineering and Dept. of Computer Engineering and Information Technology Information Technology College of Engineering, Pune College of Engineering, Pune Maharashtra, India Maharashtra, India（计算机工程系和计算机工程与信息技术系，工程学院，比拉尔，马哈拉施特拉邦，印度）
IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages
Nilekani Centre at AI4Bharat（尼莱卡尼中心（AI4Bharat））；Indian Institute of Technology, Madras（马德拉斯印度理工学院）；Sant Longowal Institute of Engineering and Technology（桑特隆戈瓦尔工程技术学院）；IIIT D&M Kancheepuram（坎奇普兰印度信息技术学院（设计与制造））；Maharaja Agrasen Institute of Technology（Maharaja Agrasen工程技术学院）；Microsoft（微软公司）；Sarvam AI；National Institute of Information and Communications Technology, Kyoto, Japan（日本京都信息通信技术研究所）
IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
Nilekani Centre at AI4Bharat（AI4Bharat 尼勒卡尼中心）；Indian Institute of Technology Madras（马德拉斯印度理工学院）；Microsoft（微软）；National Institute of Information and Communications Technology（信息通信研究机构）；Flipkart（弗利普卡特）
National Institute of Technology Karnataka（卡纳塔克邦国立理工学院）；Indian Institute of Technology Guwahati（印度理工学院古瓦哈提分校）
Developing Named Entity Recognition (NER) systems for Indian languages has been a long-standing challenge, mainly owing to the requirement of a large amount of annotated clean training instances. This paper proposes an end-to-end framework for NER for Indian languages in a low-resource setting by exploiting parallel corpora of English and Indian languages and an English NER dataset. The proposed framework includes an annotation projection method that combines word alignment score and NER tag prediction confidence score on source language (English) data to generate weakly labeled data in a targ…
Indian Institute of Technology, Madras（马德拉斯印度理工学院）；AI4Bharat；Microsoft（微软）
A cornerstone in AI research has been the creation and adoption of standardized training and test datasets to earmark the progress of state-of-the-art models. A particularly successful example is the GLUE dataset for training and evaluating Natural Language Understanding (NLU) models for English. The large body of research around self-supervised BERT-based language models revolved around performance improvements on NLU tasks in GLUE. To evaluate language models in other languages, several language-specific GLUE datasets were created. The area of speech language understanding (SLU) has followed…
Chandigarh University（昌迪加尔大学）；SRM Institute of Science and Technology（SRM科学技术学院）
Transliteration is a task in the domain of NLP where the output word is a similar-sounding word written using the letters of any foreign language. Today this system has been developed for several language pairs that involve English as either the source or target word and deployed in several places like Google Translate and chatbots. However, there is very little research done in the field of Indic languages transliterated to other Indic languages. This paper demonstrates a multilingual model based on transformers (with some modifications) that can give noticeably higher performance and accurac…
IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
Artificial Intelligence Institute, University of South Carolina（南卡罗来纳大学人工智能学院）；Rochester Institute of Technology（罗切斯特理工学院）；Meta AI；Amazon Gen AI（亚马逊生成式人工智能部门）
