首页/论文/2005.06383

arXiv 2005.063832020-05-13cs.CL

Sanskrit Segmentation Revisited

University of Hyderabad（海得拉巴大学）

Sriram Krishnan, Amba Kulkarni

英文摘要：

Computationally analyzing Sanskrit texts requires proper segmentation in the initial stages. There have been various tools developed for Sanskrit text segmentation. Of these, Gérard Huet's Reader in the Sanskrit Heritage Engine analyzes the input text and segments it based on the word parameters - phases like iic, ifc, Pr, Subst, etc., and sandhi (or transition) that takes place at the end of a word with the initial part of the next word. And it enlists all the possible solutions differentiating them with the help of the phases. The phases and their analyses have their use in the domain of sentential parsers. In segmentation, though, they are not used beyond deciding whether the words formed with the phases are morphologically valid. This paper tries to modify the above segmenter by ignoring the phase details (except for a few cases), and also proposes a probability function to prioritize the list of solutions to bring up the most valid solutions at the top.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Word Segmentation and Morphological Parsing for Sanskrit

University of Tübingen（蒂宾根大学）

We describe our participation in the Word Segmentation and Morphological Parsing (WSMP) for Sanskrit hackathon. We approach the word segmentation task as a sequence labelling task by predicting edit operations from which segmentations are derived. We approach the morphological analysis task by predicting morphological tags and rules that transform inflected words into their corresponding stems. Also, we propose an end-to-end trainable pipeline model for joint segmentation and morphological analysis. Our model performed best in the joint segmentation and analysis subtask (80.018 F1 score) and p…

2022-01-30cs.CL

Free as in Free Word Order: An Energy Based Model for Word Segmentation and Morphological Tagging in Sanskrit

IIT Kharagpur（印度理工学院克勒格布尔分校）；Flipkart（Flipkart公司）；University of Waterloo（滑铁卢大学）；American Express India Pvt Ltd（美国运通印度私人有限公司）；Chinmaya Vishwavidyapeeth（钦玛雅大学）

The configurational information in sentences of a free word order language such as Sanskrit is of limited use. Thus, the context of the entire sentence will be desirable even for basic processing tasks such as word segmentation. We propose a structured prediction framework that jointly solves the word segmentation and morphological tagging tasks in Sanskrit. We build an energy based model where we adopt approaches generally employed in graph based parsing techniques (McDonald et al., 2005a; Carreras, 2007). Our model outperforms the state of the art with an F-Score of 96.92 (percentage improve…

2018-10-25cs.CL

LEVOS：利用梵语词汇重叠生成印度语言技术词典

LEVOS: Leveraging Vocabulary Overlap with Sanskrit to Generate Technical Lexicons in Indian Languages

Indian Institute of Technology Bombay（印度理工学院孟买学院）

本文提出利用梵语段落生成印度语言技术词典的方法，通过子词级相似性和形态对齐提升翻译准确性，实验显示在技术术语翻译中取得显著提升。

2025-06-24cs.CL

面向梵语词汇、句法和语义任务的语言信息驱动神经架构

Linguistically-Informed Neural Architectures for Lexical, Syntactic and Semantic Tasks in Sanskrit

INDIAN INSTITUTE OF TECHNOLOGY KANPUR（印度理工学院坎普尔分校）

该论文针对梵语低资源、形态丰富及连音等挑战，提出融合语言信息的神经架构以解决词分割、依存分析、复合词识别和诗歌分析任务，并发布 SanskritShala 在线工具包。

2023-08-17cs.CL

Validation and Normalization of DCS corpus using Sanskrit Heritage tools to build a tagged Gold Corpus

University of Hyderabad（海得拉巴大学）；Inria Paris Center（法国国家信息与自动化研究所巴黎中心）

The Digital Corpus of Sanskrit records around 650,000 sentences along with their morphological and lexical tagging. But inconsistencies in morphological analysis, and in providing crucial information like the segmented word, urges the need for standardization and validation of this corpus. Automating the validation process requires efficient analyzers which also provide the missing information. The Sanskrit Heritage Engine's Reader produces all possible segmentations with morphological and lexical analyses. Aligning these systems would help us in recording the linguistic differences, which can…

2020-05-13cs.CL

SanskritShala：一个用于教学与标注目的的、具有网页界面的神经梵语自然语言处理工具包

SanskritShala: A Neural Sanskrit NLP Toolkit with Web-Based Interface for Pedagogical and Annotation Purposes

IIT Kanpur（印度理工学院坎普尔分校）；IIT Kharagpur（印度理工学院克勒格布尔分校）；IIT Mandi（印度理工学院曼迪分校）

本文提出首个带网页界面的神经梵语NLP工具包SanskritShala，集成词切分、形态标注、依存解析和复合词识别等模块，在多个基准上达到最先进性能，并公开代码、词嵌入模型及标注数据，以支持梵语教学与标注工作。

2023-05-29cs.CL

更多相关论文收起更多论文

Sanskrit Sandhi Splitting using seq2(seq)^2

IBM Research（IBM研究院）

In Sanskrit, small words (morphemes) are combined to form compound words through a process known as Sandhi. Sandhi splitting is the process of splitting a given compound word into its constituent morphemes. Although rules governing word splitting exists in the language, it is highly challenging to identify the location of the splits in a compound word. Though existing Sandhi splitting systems incorporate these pre-defined splitting rules, they have a low accuracy as the same compound word might be broken down in multiple ways to provide syntactically correct splits. In this research, we propos…

2019-07-15cs.CL

TransLIST: A Transformer-Based Linguistically Informed Sanskrit Tokenizer

IIT Kanpur（印度理工学院坎普尔分校）；UCLA（加州大学洛杉矶分校）；IIT Kharagpur（印度理工学院克勒格布尔分校）；IIT Mandi（印度理工学院曼迪分校）

Sanskrit Word Segmentation (SWS) is essential in making digitized texts available and in deploying downstream tasks. It is, however, non-trivial because of the sandhi phenomenon that modifies the characters at the word boundaries, and needs special treatment. Existing lexicon driven approaches for SWS make use of Sanskrit Heritage Reader, a lexicon-driven shallow parser, to generate the complete candidate solution space, over which various methods are applied to produce the most valid solution. However, these approaches fail while encountering out-of-vocabulary tokens. On the other hand, purel…

2022-10-21cs.CL

Building a Word Segmenter for Sanskrit Overnight

IIT Kharagpur（印度理工学院卡拉格普尔分校）；American Express India Pvt Ltd（美国运通印度私人有限公司）

There is an abundance of digitised texts available in Sanskrit. However, the word segmentation task in such texts are challenging due to the issue of 'Sandhi'. In Sandhi, words in a sentence often fuse together to form a single chunk of text, where the word delimiter vanishes and sounds at the word boundaries undergo transformations, which is also reflected in the written text. Here, we propose an approach that uses a deep sequence to sequence (seq2seq) model that takes only the sandhied string as the input and predicts the unsandhied string. The state of the art models are linguistically invo…

2018-02-17cs.CLcs.IR

段落分割再审视：迈向结构化语音的标准任务

Paragraph Segmentation Revisited: Towards a Standard Task for Structuring Speech

Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Carnegie Mellon University（卡内基梅隆大学）

本文提出TEDPara和YTSegPara基准，通过约束解码方法和紧凑模型MiniSeg，实现语音段落分割的标准化与实用化。

2026-04-09cs.CL