Recent studies in sequence-to-sequence learning demonstrate that RNN encoder-decoder structure can successfully generate Chinese poetry. However, existing methods can only generate poetry with a given first line or user's intent theme. In this paper, we proposed a three-stage multi-modal Chinese poetry generation approach. Given a picture, the first line, the title and the other lines of the poem are successively generated in three stages. According to the characteristics of Chinese poems, we propose a hierarchy-attention seq2seq model which can effectively capture character, phrase, and sentence information between contexts and improve the symmetry delivered in poems. In addition, the Latent Dirichlet allocation (LDA) model is utilized for title generation and improve the relevance of the whole poem and the title. Compared with strong baseline, the experimental results demonstrate the effectiveness of our approach, using machine evaluations as well as human judgments.
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
Poetry is one of the most important art forms of human languages. Recently many studies have focused on incorporating some linguistic features of poetry, such as style and sentiment, into its understanding or generation system. However, there is no focus on understanding or evaluating the semantics of poetry. Therefore, we propose a novel task to assess a model's semantic understanding of poetry by poem matching. Specifically, this task requires the model to select one line of Chinese classical poetry among four candidates according to the modern Chinese translation of a line of poetry. To con…
How Images Inspire Poems: Generating Classical Chinese Poetry from Images with Memory Networks
Anhui Province Key Laboratory of Big Data Analysis and Application（安徽省大数据分析与应用重点实验室）；School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；AI Department, Ant Financial Services Group（蚂蚁金融服务集团人工智能部门）
Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?
Graduate School of Informatics, Kyoto University（京都大学信息学研究科）；University of Macau（澳门大学）；College of Computer Science, Inner Mongolia University（内蒙古大学计算机学院）
Generating Major Types of Chinese Classical Poetry in a Uniformed Framework
Tsinghua University（清华大学）；Institute for Artificial Intelligence, Tsinghua University（清华大学人工智能研究院）；State Key Lab on Intelligent Technology and Systems, Tsinghua University（清华大学智能技术与系统国家重点实验室）
诗歌生成是文本生成领域中一个有趣的研究课题。中国古典诗歌是中华民族最宝贵的文学文化遗产之一，为一代又一代中国人所熟悉和喜爱。它的语言结构从形、音到义都有其独特的特点，是文本生成的理想测试任务。在本文中，我们提出了一个基于GPT-2的统一框架，用于生成中国古典诗歌的主要类型。我们定义了一个统一的格式来制定所有类型的训练样本，通过整合详细的形式信息，然后提出了一个简单的形式强调加权方法在GPT-2中，以加强对形式的控制，特别是那些较长的身体长度的形式。初步实验结果表明，该模型能够生成形式和内容都具有较高质量的主要类型的中国古典诗歌，验证了该策略的有效性。该模型已被纳入清华大学开发的最具影响力的中国古典诗歌生成系统九歌（Guo et al.，2019年）。
Central Conservatory of Music（中央音乐学院）；Tsinghua University（清华大学）
Poetry generation has been a difficult task in natural language processing. Unlike plain neural text generation tasks, poetry has a high requirement for novelty, since an easily-understood sentence with too many high frequency words might not be considered as poetic, while adequately ambiguous sentences with low frequency words can possibly be novel and creative. Inspired by this, we present Lingxi, a diversity-aware Chinese modern poetry generation system. We propose nucleus sampling with randomized head (NS-RH) algorithm, which randomizes the high frequency part ("head") of the predicted dis…
Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry
College of Computer Science, Inner Mongolia University, China（内蒙古大学计算机学院，中国）；National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, China（蒙古语智能信息处理技术国家与地方联合工程研究中心，中国）；NLP²CT Lab, Department of Computer and Information Science, University of Macau, China（澳门大学计算机与信息科学学院NLP²CT实验室，中国）
Can Machine Generate Traditional Chinese Poetry? A Feigenbaum Test
Tsinghua University（清华大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Huilan Limited（汇兰有限公司）；Tsinghua National Lab for Information Science and Technology（清华大学信息科学技术国家实验室）
Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry
NLP 2 CT Lab, Department of Computer and Information Science, University of Macau（计算机与信息科学系，澳门大学）；Department of Portuguese, Faculty of Arts and Humanities, University of Macau（文学与人文学院，澳门大学）
Chinese traditional poetry is an important intangible cultural heritage of China and an artistic carrier of thought, culture, spirit and emotion. However, due to the strict rules of ancient poetry, it is very difficult to write poetry by machine. This paper proposes an automatic generation method of Chinese traditional poetry based on deep learning technology, which extracts keywords from each poem and matches them with the previous text to make the poem conform to the theme, and when a user inputs a paragraph of text, the machine obtains the theme and generates poem sentence by sentence. Usin…
In order to test whether artificial intelligence can create qualified classical poetry like humans, the author proposes a study of Chinese classical poetry generation based on a pre-trained model. This paper mainly tries to use BART and other pre training models, proposes FS2TEXT and RR2TEXT to generate metrical poetry text and even specific style poetry text, and solves the problem that the user's writing intention gradually reduces the relevance of the generated poetry text. In order to test the model's results, the authors selected ancient poets, by combining it with BART's poetic model wor…
CCiV: A Benchmark for Structure, Rhythm and Quality in LLM-Generated Chinese \textit{Ci} Poetry
School of Computer Science and Technology, East China Normal University, China（东华师范大学计算机科学与技术学院）；Shanghai Institute of Artificial Intelligence for Education, East China Normal University, China（东华师范大学教育人工智能研究所）；Department of Chinese Language and Literature, East China Normal University, China（东华师范大学中文语言文学系）
Can ChatGPT Really Understand Modern Chinese Poetry?
NLP；CT Lab, Department of Computer and Information Science, University of Macau（计算机与信息科学系，澳门科技大学CT实验室）；Department of Portuguese, Faculty of Arts and Humanities, University of Macau（人文学院葡萄牙语系，澳门科技大学）
FrameNet( Fillmore and Baker [2009] ) is well-known for its wide use for knowledge representation in the form of inheritance-based ontologies and lexica( Trott et al. [2020] ). Although FrameNet is usually applied to languages like English, Spanish and Italian, there are still plenty of FrameNet data sets available for other languages like Chinese, which differs significantly from those languages based on Latin alphabets. In this paper, the translation from ancient Chinese Poetry to modern Chinese will be first conducted to further apply the Chinese FrameNet(CFN, provided by Shanxi University)…
Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs
Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）；University of Rochester（罗切斯特大学）；Sichuan University（四川大学）；Department of Portuguese, Faculty of Arts and Humanities, University of Macau（澳门大学人文学院葡萄牙语系）
Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry
Shanghai Jiao Tong University（上海交通大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
MixPoet: Diverse Poetry Generation via Learning Controllable Mixed Latent Space
Tsinghua University（清华大学）；ESTATES PTE LTD（6ESTATES PTE LTD）；Beijing University of Posts and Telecommunications（北京邮电大学）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Institute for Artificial Intelligence, Tsinghua University（清华大学人工智能研究院）；State Key Lab on Intelligent Technology and Systems, Tsinghua University（智能技术与系统国家重点实验室）
Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?
NLP；CT Lab, Department of Computer and Information Science, University of Macau（计算机与信息科学系，澳门大学CT实验室）；Department of Portuguese, Faculty of Arts and Humanities, University of Macau（人文学院葡萄牙语系，澳门大学）
Chinese couplet is a special form of poetry composed of complex syntax with ancient Chinese language. Due to the complexity of semantic and grammatical rules, creation of a suitable couplet is a formidable challenge. This paper presents a transformer-based sequence-to-sequence couplet generation model. With the utilization of AnchiBERT, the model is able to capture ancient Chinese language understanding. Moreover, we evaluate the Glyph, PinYin and Part-of-Speech tagging on the couplet grammatical rules to further improve the model.
In this work, we construct the largest dataset for multimodal pretraining in Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide range of domains. We propose a cross-modal pretraining method called M6, referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. We scale the model size up to 10 billion and 100 billion parameters, and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding perfor…
Multi-modal pretraining for learning high-level multi-modal representation is a further step towards deep learning and artificial intelligence. In this work, we propose a novel model, namely InterBERT (BERT for Interaction), which is the first model of our series of multimodal pretraining methods M6 (MultiModality-to-MultiModality Multitask Mega-transformer). The model owns strong capability of modeling interaction between the information flows of different modalities. The single-stream interaction module is capable of effectively processing information of multiple modalilties, and the two-str…
Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Lab of Artificial Intelligence for Education, East China Normal University（华东师范大学教育人工智能实验室）；Shanghai Institute of Artificial Intelligence for Education, East China Normal University（华东师范大学上海教育人工智能研究院）
VATEX Captioning Challenge 2019: Multi-modal Information Fusion and Multi-stage Training Strategy for Video Captioning
CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）；National Laboratory of Pattern Recognition, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；PeopleAI, Inc., China（中国PeopleAI公司）
Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）深圳大数据研究院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Automatic Cued Speech Recognition (ACSR) provides an intelligent human-machine interface for visual communications, where the Cued Speech (CS) system utilizes lip movements and hand gestures to code spoken language for hearing-impaired people. Previous ACSR approaches often utilize direct feature concatenation as the main fusion paradigm. However, the asynchronous modalities i.e., lip, hand shape and hand position) in CS may cause interference for feature concatenation. To address this challenge, we propose a transformer based cross-modal mutual learning framework to prompt multi-modal interac…
