Recent advances in large language models (LLMs) have empowered AI agents capable of performing various sequential decision-making tasks. However, effectively guiding LLMs to perform well in unfamiliar domains like web navigation, where they lack sufficient knowledge, has proven to be difficult with the demonstration-based in-context learning paradigm. In this paper, we introduce a novel framework, called AutoGuide, which addresses this limitation by automatically generating context-aware guidelines from offline experiences. Importantly, each context-aware guideline is expressed in concise natural language and follows a conditional structure, clearly describing the context where it is applicable. As a result, our guidelines facilitate the provision of relevant knowledge for the agent's current decision-making process, overcoming the limitations of the conventional demonstration-based learning paradigm. Our evaluation demonstrates that AutoGuide significantly outperforms competitive baselines in complex benchmark domains, including real-world web navigation.
Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study
The Graduate University for Advanced Studies, SOKENDAI(综合研究大学院大学);National Institute of Informatics(国立信息学研究所);National Library of Medicine(美国国家医学图书馆);Joint Support-Center for Data Science Research(数据科学联合支持中心);Japanese-French Laboratory of Informatics, CNRS(日法信息学联合实验室(法国国家科学研究中心));Nantes University(南特大学)
A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations
Microsoft Research Asia(微软亚洲研究院);Hong Kong University of Science and Technology(香港科技大学);Peking University(北京大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Annotation Guidelines-Based Knowledge Augmentation: Towards Enhancing Large Language Models for Educational Text Classification
National Engineering Research Center of Educational Big Data, Faculty of Artificial Intelligence in Education, Central China Normal University(国家教育大数据工程研究中心,教育人工智能学院,中央财经大学)
SpecEval: Evaluating Model Adherence to Behavior Specifications
Department of Computer Science, Stanford University(计算机科学系,斯坦福大学);The Bradley Department of Electrical and Computer Engineering, Virginia Tech(电气与计算机工程系,弗吉尼亚理工学院)
Applied Linguistics and Technology Program(应用语言学与技术项目);Iowa State University, USA(美国爱荷华州立大学);Department of Informatics(信息学系);University of Oslo, Norway(挪威奥斯陆大学)
AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding
Yonsei University College of Medicine(延世大学医学院);Yonsei Institute for Digital Health(延世大学数字健康研究所);Yonsei University(延世大学);Institute of Behavioral Sciences in Medicine(医学行为科学研究所)