Goal-directed interactive agents, which autonomously complete tasks through interactions with their environment, can assist humans in various domains of their daily lives. Recent advances in large language models (LLMs) led to a surge of new, more and more challenging tasks to evaluate such agents. To properly contextualize performance across these tasks, it is imperative to understand the different challenges they pose to agents. To this end, this survey compiles relevant tasks and environments for evaluating goal-directed interactive agents, structuring them along dimensions relevant for understanding current obstacles. An up-to-date compilation of relevant resources can be found on our project website: https://coli-saar.github.io/interactive-agents.
GODEL: Large-Scale Pre-Training for Goal-Directed Dialog
Microsoft Corp.(微软公司);Columbia University(哥伦比亚大学)
我们介绍了GODEL(Grounded Open Dialogue语言模型),一个大型的预先训练的对话语言模型。与DialoGPT等早期模型相比,GODEL利用了一个新的基础预训练阶段,旨在更好地支持使GODEL适应广泛的下游对话任务,这些任务需要当前对话外部的信息(例如,数据库或文档)以产生良好的响应。针对一系列包括面向任务的对话,会话QA和接地开放域对话的基准测试的实验表明,GODEL在Few-Shot微调设置中,在人工和自动评估方面都优于最先进的预训练对话模型。我们的评估方法的一个新的特点是引入了效用的概念,评估的有用性的反应(外在评价),除了他们的沟通功能(内在评价)。我们表明,外部评估提供了改进的注释者之间的协议和自动化指标的相关性。代码和数据处理脚本是公开的。
A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents
University of Pennsylvania(宾夕法尼亚大学);New York University(纽约大学);Indiana University, Bloomington(印第安纳大学,布卢明顿);Northeastern University(东北大学);University College London(伦敦大学学院)
Human Attention during Goal-directed Reading Comprehension Relies on Task Optimization
College of Biomedical Engineering and Instrument Sciences, Zhejiang University(浙江大学生物医学工程与仪器科学学院);Zhejiang lab(浙江实验室);Division of Arts and Sciences, New York University Shanghai(上海纽约大学文理学院)
StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement
University of Notre Dame(圣母大学);University of California, Irvine(加州大学欧文分校);Georgia Institute of Technology(佐治亚理工学院);Rensselaer Polytechnic Institute(伦斯勒理工学院);University of Washington(华盛顿大学);IBM Research(IBM研究院)