BotEval：促进交互式人类评估
BotEval: Facilitating Interactive Human Evaluation
AI总结：
针对NLP模型在复杂交互任务中难以充分评估的问题，开发了开源工具包BotEval，通过支持人机交互评估、提供多复杂度模板及众包平台兼容性，平衡定制灵活性与用户友好性。
摘要：
随着自然语言处理（NLP）模型的快速发展，语言模型被应用于日益复杂的交互式任务，如谈判和对话审核。让人类评估者直接与这些NLP模型进行交互，对于充分评估其在此类交互式任务上的表现至关重要。我们开发了BotEval，这是一个易于定制、开源的评估工具包，重点是在评估过程中实现人机交互，而非由人类评估者对静态输入做出判断。BotEval通过提供涵盖不同复杂程度的常见用例模板，以及与流行众包平台的内置兼容性，在定制灵活性与用户友好性之间取得了平衡。我们通过一项评估各种聊天机器人在对话审核有效性方面表现的研究，展示了BotEval的众多实用功能，并讨论了BotEval与其他标注工具的不同之处。
英文摘要：
Following the rapid progress in natural language processing (NLP) models, language models are applied to increasingly more complex interactive tasks such as negotiations and conversation moderations. Having human evaluators directly interact with these NLP models is essential for adequately evaluating the performance on such interactive tasks. We develop BotEval, an easily customizable, open-source, evaluation toolkit that focuses on enabling human-bot interactions as part of the evaluation process, as opposed to human evaluators making judgements for a static input. BotEval balances flexibility for customization and user-friendliness by providing templates for common use cases that span various degrees of complexity and built-in compatibility with popular crowdsourcing platforms. We showcase the numerous useful features of BotEval through a study that evaluates the performance of various chatbots on their effectiveness for conversational moderation and discuss how BotEval differs from other annotation tools.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
