arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.10416cs.CLcs.AI
收藏

AI与人类--科学内容生成的差异分析

AI vs. Human -- Differentiation Analysis of Scientific Content Generation

Wuhan University(武汉大学);Worcester Polytechnic Institute(伍斯特理工学院)

Yongqiang Ma, Jiawei Liu, Fan Yi, Qikai Cheng, Yong Huang, Wei Lu, Xiaozhong Liu

摘要:

最近的神经语言模型在产生非常可控、流畅和合乎语法的文本方面迈出了重要的一步。尽管研究发现,人工智能生成的文本与众包工作人员的人类书面文本无法区分,但人工智能生成的文本中仍然存在更微妙、更难发现的错误。我们主要关注科学AI写作助手深度参与的场景。首先,我们构建了一个特征描述框架,以区分人工智能生成的文本和人类写的文本从语法,语义和语用的基础上人类的评价。然后我们利用这些特征,即,写作风格、连贯性、一致性和论证逻辑,从提出的两类内容框架来分析。最后,我们采用了几种公开可用的方法,通过人工智能生成的科学文本检测模型来研究人工智能生成的科学文本与人类撰写的科学文本之间的差距。结果表明,虽然人工智能有潜力生成与人类书面内容一样准确的科学内容,但在深度和整体质量方面仍存在差距。人工智能生成的科学内容更有可能包含事实问题的错误。我们发现,人工智能生成的科学文本和人类撰写的科学文本之间存在着“写作风格”的差距。在此基础上,我们总结了一系列适用于其他领域的模型无关和分布无关检测任务的特征。本文的研究结果有助于指导人工智能模型的优化,以产生高质量的内容,并解决相关的道德和安全问题。

英文摘要:

Recent neural language models have taken a significant step forward in producing remarkably controllable, fluent, and grammatical text. Although studies have found that AI-generated text is not distinguishable from human-written text for crowd-sourcing workers, there still exist errors in AI-generated text which are even subtler and harder to spot. We primarily focus on the scenario in which scientific AI writing assistant is deeply involved. First, we construct a feature description framework to distinguish between AI-generated text and human-written text from syntax, semantics, and pragmatics based on the human evaluation. Then we utilize the features, i.e., writing style, coherence, consistency, and argument logistics, from the proposed framework to analyze two types of content. Finally, we adopt several publicly available methods to investigate the gap of between AI-generated scientific text and human-written scientific text by AI-generated scientific text detection models. The results suggest that while AI has the potential to generate scientific content that is as accurate as human-written content, there is still a gap in terms of depth and overall quality. The AI-generated scientific content is more likely to contain errors in factual issues. We find that there exists a "writing style" gap between AI-generated scientific text and human-written scientific text. Based on the analysis result, we summarize a series of model-agnostic and distribution-agnostic features for detection tasks in other domains. Findings in this paper contribute to guiding the optimization of AI models to produce high-quality content and addressing related ethical and security concerns.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑