Chinese text segmentation is a well-known and difficult problem. On one side, there is not a simple notion of "word" in Chinese language making really hard to implement rule-based systems to segment written texts, thus lexicons and statistical information are usually employed to achieve such a task. On the other side, any piece of Chinese text usually includes segments present neither in the lexicons nor in the training data. Even worse, such unseen sequences can be segmented into a number of totally unrelated words making later processing phases difficult. For instance, using a lexicon-based system the sequence ???(Baluozuo, Barroso, current president-designate of the European Commission) can be segmented into ?(ba, to hope, to wish) and ??(luozuo, an undefined word) changing completely the meaning of the sentence. A new and extremely simple algorithm specially suited to work over short Chinese documents is introduced. This new algorithm performs text "self-segmentation" producing results comparable to those achieved by native speakers without using either lexicons or any statistical information beyond the obtained from the input text. Furthermore, it is really robust for finding new "words", especially proper nouns, and it is well suited to build lexicons from scratch. Some preliminary results are provided in addition to examples of its employment.
Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter
DAMO Academy, Alibaba Group(阿里巴巴达摩院);College of Computer Science, Nankai University(南开大学计算机学院);School of Engineering, Westlake University(西湖大学工学院);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院前沿技术研究所)
Fudan University(复旦大学);Research Institute of Intelligent and Complex Systems, Fudan University(复旦大学智能复杂系统研究院);School of Data Science, Fudan University(复旦大学数据科学学院)
SLK-NER: Exploiting Second-order Lexicon Knowledge for Chinese NER
National Computer System Engineering Research Institute of China(国家计算机系统工程技术研究中心);University of Chinese Academy of Sciences(中国科学院大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
Lexicon-constrained Copying Network for Chinese Abstractive Summarization
College of Computer Science and Electronic Engineering, Hunan University(湖南大学信息科学与工程学院);National Supercomputing Center in Changsha(国家超级计算长沙中心);Department of Industrial Engineering, International Ataturk Alatoo University(国际阿塔图尔克-阿拉图大学工业工程系)
Detecting Suicidal Ideation in Chinese Microblogs with Psychological Lexicons
Institute of Psychology, Chinese Academy of Sciences(中国科学院心理研究所);Institute of Population Research, Peking University(北京大学人口研究所);University of Puget Sound(普吉特湾大学)
DualCoTs: Dual Chain-of-Thoughts Prompting for Sentiment Lexicon Expansion of Idioms
The College of Big Data and Internet, Shenzhen Technology University(深圳技术大学大数据与互联网学院);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Harbin Institute of Technology(哈尔滨工业大学)
Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratch
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)
A realistic and robust model for Chinese word segmentation
Institute of Linguistics, Academia Sinica(中央研究院语言学研究所);Hong Kong Polytechnic University(香港理工大学);Academia Sinica(中央研究院);National Taiwan Normal University(台湾师范大学)
A Natural Language Processing Pipeline of Chinese Free-text Radiology Reports for Liver Cancer Diagnosis
School of Biomedical Engineering, Capital Medical University(首都医科大学生物医学工程学院);Beijing Friendship Hospital, Capital Medical University(首都医科大学附属北京友谊医院);Tsinghua University(清华大学)
Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis
School of Software Engineering, Beijing University of Technology(北京工业大学软件学院);School of Nursing, Wuhan University(武汉大学护理学院);Sorbonne Université(索邦大学);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);Inserm(法国国家健康与医学研究院);AP-HP(巴黎公共医疗集团);Hôpital de la Pitié-Salpêtrière(皮蒂-萨尔皮特里医院)
Exploring the effectiveness of ChatGPT-based feedback compared with teacher feedback and self-feedback: Evidence from Chinese to English translation
School of Foreign Languages, Southeast University(东南大学外国语学院);The Hong Kong Polytechnic University(香港理工大学);School of Languages, Literature, Cultures and Linguistics, Monash University(莫纳什大学语言、文学、文化与语言学学院)
Towards a resource for multilingual lexicons: an MT assisted and human-in-the-loop multilingual parallel corpus with multi-word expression annotation
Leiden University Medical Center(莱顿大学医学中心);Leiden Institute of Advanced Computer Science (LIACS)(莱顿先进计算机科学研究所);Leiden University(莱顿大学);University of Tours(图尔大学);Arabic Natural Language Processing Research Group(阿拉伯自然语言处理研究组);University of Sfax(索菲亚大学);Institute for Natural Language Processing (IMS)(自然语言处理研究所);University of Stuttgart(斯图加特大学);ADAPT Research Centre(ADAPT研究中心);Dublin City University(都柏林城市大学);Insight Centre for Data Analytics(数据洞察中心);University of Manchester(曼彻斯特大学)
Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text
Stanford University(斯坦福大学);Facebook AI Research(脸书人工智能研究院);UPF Linguistics Department(庞培法布拉大学语言学系)
递归神经网络(RNN)在许多自然语言处理任务中取得了惊人的性能。这重新引起了人们对这些通用序列处理设备是否诱导真正的语言知识的兴趣。然而,几乎所有当前的分析研究都用已知单词的词汇表初始化RNN,并在训练过程中为它们提供标记化的输入。我们提出了一个多语言的语言知识编码的RNN训练字符级语言模型,在输入数据与词边界删除的研究。这些网络面临着一个更艰难、更现实的认知任务,必须根据输入统计数据从头开始发现任何有用的语言单元。结果表明,我们的“near tabula rasa”RNN大多能够解决直观地假设单词级别知识的形态,句法和语义任务,并且确实在某种程度上学会了跟踪单词边界。我们的研究为语言学习和使用中明确的、严格的词汇库的必要性的思考打开了大门。