Traditionally, writing assistance systems have focused on short or even single-word suggestions. Recently, large language models like GPT-3 have made it possible to generate significantly longer natural-sounding suggestions, offering more advanced assistance opportunities. This study explores the trade-offs between sentence- vs. message-level suggestions for AI-mediated communication. We recruited 120 participants to act as staffers from legislators' offices who often need to respond to large volumes of constituent concerns. Participants were asked to reply to emails with different types of assistance. The results show that participants receiving message-level suggestions responded faster and were more satisfied with the experience, as they mainly edited the suggested drafts. In addition, the texts they wrote were evaluated as more helpful by others. In comparison, participants receiving sentence-level assistance retained a higher sense of agency, but took longer for the task as they needed to plan the flow of their responses and decide when to use suggestions. Our findings have implications for designing task-appropriate communication assistance systems.
A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews
Department of Computer Science & Information Systems, Birla Institute of Technology and Science, Pilani, India(印度比拉理工学院计算机科学与信息系统系);Birdeye Inc.(Birdeye公司)
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, Peking University(北京大学数据科学中心);Microsoft(微软);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Laboratory of Media Convergence Production Technology and Systems(媒体融合生产技术与系统国家重点实验室)
本文提出Smart Word Suggestions(SWS)任务与基准,通过端到端评估识别写作中需改进的词汇并提供替换建议,构建了含人工标注测试集与远监督训练集的评估框架,实验表明该任务具挑战性并指明未来方向。
Performance of ChatGPT-3.5 and GPT-4 on the United States Medical Licensing Examination With and Without Distractions
Magee-Womens Research Institute(马吉妇女研究所);University of Pittsburgh School of Medicine(匹兹堡大学医学院);Chaim Sheba Medical Center(哈伊姆·谢巴医疗中心);Sackler Faculty of Medicine, Tel Aviv University(特拉维夫大学萨克勒医学院);Ariel University(阿里尔大学)
Department of Linguistics, Stanford University(斯坦福大学语言学系);BIO-X Interdisciplinary Biosciences Institute, Stanford University(斯坦福大学生物交叉科学研究所);Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology(麻省理工学院脑科学与认知科学系)