Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs' generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3's capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities, and reveal its contribution as a writing "collaborator" under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs' promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.
CoAuthorAI: A Human in the Loop System For Scientific Book Writing
Kexin Technology, Beijing（凯欣科技，北京）；Institute for Sustainable Industries and Liveable Cities, Victoria University（可持续产业与宜居城市研究所，维多利亚大学）；School of Information Technology, Deakin University（信息科技学院，德坎大学）
Columbia University（哥伦比亚大学）；New York University（纽约大学）
Recent work in training large language models (LLMs) to follow natural language instructions has opened up exciting opportunities for natural language interface design. Building on the prior success of LLMs in the realm of computer-assisted creativity, we aim to study if LLMs can improve the quality of user-generated content through collaboration. We present CoPoet, a collaborative poetry writing system. In contrast to auto-completing a user's text, CoPoet is controlled by user instructions that specify the attributes of the desired text, such as Write a sentence about `love' or Write a senten…
Exploring how EFL students talk to and through AI to develop texts
Everwrite Limited（Everwrite有限公司）；Shanghai Jiao Tong University（上海交通大学）；The Education University of Hong Kong（香港教育大学）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）
Google Research（谷歌研究院）；University of Pennsylvania（宾夕法尼亚大学）
NLP researchers need more, higher-quality text datasets. Human-labeled datasets are expensive to collect, while datasets collected via automatic retrieval from the web such as WikiBio are noisy and can include undesired biases. Moreover, data sourced from the web is often included in datasets used to pretrain models, leading to inadvertent cross-contamination of training and test sets. In this work we introduce a novel method for efficient dataset curation: we use a large language model to provide seed generations to human raters, thereby changing dataset authoring from a writing task to an ed…
Expert Preference-based Evaluation of Automated Related Work Generation
Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI) Technical University of Darmstadt（乌普萨拉知识处理实验室（UKP实验室）计算机科学系海斯赛人工智能中心（hessian.AI）达姆施塔特技术大学）；Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)（克诺尔兹塞学校卓越学习和智能系统（ELIZA））
From the earliest days of computing, there have been tools to help shape narrative. Spell-checking, word counts, and readability analysis, give today's novelists tools that Dickens, Austen, and Shakespeare could only have dreamt of. However, such tools have focused on the word, or phrase levels. In the last decade, research focus has shifted to support for collaborative editing of documents. This work considers more sophisticated attempts to visualise the semantics, pace and rhythm within a narrative through data mining. We describe real life applications in two related domains.
Collaborative writing is essential for teams that create documents together. Creating documents in large-scale collaborations is a challenging task that requires an efficient workflow. The design of such a workflow has received comparatively little attention. Conventional solutions such as working on a single Microsoft Word document or a shared online document are still widely used. In this paper, we propose a new workflow consisting of a combination of the lightweight markup language AsciiDoc together with the state-of-the-art version control system Git. The proposed process makes use of well…
Collaborative creativity is the approach of employing crowd to accomplish creative tasks. In this paper, we present a collaborative crowdsourcing platform for writing stories by means of connecting a series of `images'. These connected images are termed as Image Chains, reflecting successive scenarios. Users can either start or extend an Image Chain by uploading their own image or choosing from the available ones. These users are allowed to pen their stories from the Image Chains. Finally, stories get published based on the number of votes obtained. This provides an organized framework of stor…
Microsoft（微软）；Microsoft Research AI（微软研究院人工智能）；University of Waterloo（滑铁卢大学）
Writing is a complex non-linear process that begins with a mental model of intent, and progresses through an outline of ideas, to words on paper (and their subsequent refinement). Despite past research in understanding writing, Web-scale consumer and enterprise collaborative digital writing environments are yet to greatly benefit from intelligent systems that understand the stages of document evolution, providing opportune assistance based on authors' situated actions and context. In this paper, we present three studies that explore temporal stages of document authoring. We first survey inform…
FAID: Fine-Grained AI-Generated Text Detection Using Multi-Task Auxiliary and Multi-Level Contrastive Learning
BKAI Research Center, Hanoi University of Science and Technology（河内科学技术大学BKAI研究中心）；MBZUAI；INSAIT, Sofia University "St. Kliment Ohridski"（索菲亚大学"圣克莱门特·奥赫里德斯"INSAIT）
Getting machines to generate text perceived as creative is a long-pursued goal. A growing body of research directs this goal towards augmenting the creative writing abilities of human authors. In this paper, we pursue this objective by analyzing how observing examples of automatically generated text influences writing. In particular, we examine a task referred to as sentence infilling, which involves transforming a list of words into a complete sentence. We emphasize "storiability" as a desirable feature of the resulting sentences, where "storiable" sentences are those that suggest a story a r…
Why Human Guidance Matters in Collaborative Vibe Coding
Cornell University（康奈尔大学）；Princeton University（普林斯顿大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Cambridge（剑桥大学）；New York University（纽约大学）
