Learning to Ask: When LLM Agents Meet Unclear Instruction
Renmin University of China(中国人民大学);The Chinese University of Hong Kong(香港中文大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学深圳校区);Johns Hopkins University(约翰霍普金斯大学);Xiaohongshu Inc.(小红书公司)
Wenxuan Wang, Juluan Shi, Zixuan Ling, Yuk-Kit Chan, Chaozheng Wang, Cheryl Lee, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, Michael R. Lyu
Equipped with the capability to call functions, modern large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone. However, the effective execution of these tools relies heavily not just on the advanced capabilities of LLMs but also on precise user instructions, which often cannot be ensured in the real world. To evaluate the performance of LLMs tool-use under imperfect instructions, we meticulously examine the real-world instructions queried from users, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench (NoisyToolBench). We find that due to the next-token prediction training objective, LLMs tend to arbitrarily generate the missed argument, which may lead to hallucinations and risks. To address this issue, we propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions. Moreover, to reduce the manual labor involved in user-LLM interaction and assess LLMs performance in tool utilization from both accuracy and efficiency perspectives, we design an automated evaluation tool named ToolEvaluator. Our experiments demonstrate that the AwN significantly outperforms existing frameworks for tool learning in the NoisyToolBench. We will release all related code and datasets to support future research.
Ask, Relax, or Act? Evaluating Actionable Indeterminacy in LLM Preference Reasoning
The Chinese University of Hong Kong Shenzhen(香港中文大学(深圳));Beijing University of Posts and Telecommunications(北京邮电大学);Halmstad University College(哈尔姆斯塔德大学学院);Tongji University(同济大学);University of Waterloo(滑铁卢大学)
LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants?
University of California San Diego La Jolla California United States;Northeastern University Boston Massachusetts United States;North Carolina State University Raleigh North Carolina United States;Independent Researcher Seattle Washington United States;Independent Researcher Palo Alto California United States;University of California San Diego;Northeastern University;North Carolina State University;Independent Researcher
Oxford Internet Institute, University of Oxford(牛津大学互联网研究所);Centre for Digital Governance, Hertie School(数字治理中心);Weizenbaum Institute(魏泽瑙研究所);Technical University of Munich(慕尼黑技术大学)