arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.14511cs.HC
收藏

使用生成式人工智能个人助理工具对任务特定生产力提升的评估

Evaluation of Task Specific Productivity Improvements Using a Generative Artificial Intelligence Personal Assistant Tool

Brian S. Freeman, Kendall Arriola, Dan Cottell, Emmett Lawlor, Matt Erdman, Trevor Sutherland, Brian Wells

AI总结:

本研究通过对照实验评估基于GPT-3.5的生成式AI个人助理工具在四类办公任务中的生产力提升效果,发现其在总结和说明创建任务上提升3.3%至69%,并验证了LLM-as-a-judge质量评估方法的有效性。

摘要:

本研究评估了使用由Trane Technologies开发的生成式人工智能个人助理工具(PAT)所实现的生产力提升。该PAT基于OpenAI的GPT 3.5模型,部署在Microsoft Azure上,以确保安全访问和知识产权保护。为评估该工具的生产力效果,开展了一项实验,比较四项常见办公任务的完成时间和内容质量:撰写电子邮件、总结文章、为简单任务创建说明以及准备演示文稿大纲。六十三(63)名参与者被随机分为使用PAT的测试组和手动执行任务的对照组。结果表明生产力有显著提升,尤其是在涉及总结和说明创建的任务中,提升幅度从3.3%到69%不等。研究进一步分析了用户年龄、回答字数以及回答质量等因素,发现PAT用户生成了更冗长且质量更高的内容。采用GPT-4的“LLM-as-a-judge”方法对回答质量进行评分,该方法有效区分了高质量和低质量输出。研究结果强调了PAT在提升工作场所生产力方面的潜力,并指出了进一步研究和优化的方向。

英文摘要:

This study evaluates the productivity improvements achieved using a generative artificial intelligence personal assistant tool (PAT) developed by Trane Technologies. The PAT, based on OpenAI's GPT 3.5 model, was deployed on Microsoft Azure to ensure secure access and protection of intellectual property. To assess the tool's productivity effectiveness, an experiment was conducted comparing the completion times and content quality of four common office tasks: writing an email, summarizing an article, creating instructions for a simple task, and preparing a presentation outline. Sixty-three (63) participants were randomly divided into a test group using the PAT and a control group performing the tasks manually. Results indicated significant productivity enhancements, particularly for tasks involving summarization and instruction creation, with improvements ranging from 3.3% to 69%. The study further analyzed factors such as the age of users, response word counts, and quality of responses, revealing that the PAT users generated more verbose and higher-quality content. An 'LLM-as-a-judge' method employing GPT-4 was used to grade the quality of responses, which effectively distinguished between high and low-quality outputs. The findings underscore the potential of PATs in enhancing workplace productivity and highlight areas for further research and optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑