The rapid development in the field of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents to assist humans in their daily tasks. However, a significant gap remains in assessing whether LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the pressing need for methods to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval provides an implementation for the math problems, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria. We present a comprehensive analysis of the robustness of quantifier's work.
From Code to Compliance: Assessing ChatGPT's Utility in Designing an Accessible Webpage -- A Case Study
Department of Computer Science, Norwegian University of Science and Technology（计算机科学系，挪威科学技术大学）；Department of Design, Norwegian University of Science and Technology（设计系，挪威科学技术大学）
Conversation AI Dialog for Medicare powered by Finetuning and Retrieval Augmented Generation
University of Florida（佛罗里达大学）；National Institute of Technology Calicut（卡利卡特国家理工学院）；SRM, Chennai（SRM大学（金奈））；University of Cincinnati（辛辛那提大学）；Dartmouth College（达特茅斯学院）；Arizona State University（亚利桑那州立大学）
SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）；Carnegie Mellon University（卡内基梅隆大学）
FinSphere, a Real-Time Stock Analysis Agent Powered by Instruction-Tuned LLMs and Domain Tools
JF SmartInvest Holdings Ltd.（JF SmartInvest控股有限公司）；Columbia University（哥伦比亚大学）；Shanghai University of Finance and Economics（上海财经大学）；Johns Hopkins University（约翰霍普金斯大学）
Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain
Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models
The State Key Laboratory of Blockchain and Data Security, Zhejiang University（区块链与数据安全国家重点实验室，浙江大学）；Sun Yat-sen University（中山大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
Assessing the Impact of Conspiracy Theories Using Large Language Models
Arizona State University（亚利桑那州立大学）；University of Washington（华盛顿大学）；USC Information Sciences Institute（南加州大学信息科学研究所）；ASU Institute for Social Science Research（亚利桑那州立大学社会科学研究所）
AssistantX: An LLM-Powered Proactive Assistant in Collaborative Human-Populated Environment
Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）
The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
NLP Department, Mohamed Bin Zayed University of Artificial Intelligence（马尔代夫比兹艾兹大学人工智能学院自然语言处理系）；Ubiquitous Knowledge Processing Lab, Department of Computer Science（计算机科学系通用知识处理实验室）；Hessian Center for AI (hessian.AI), TU Darmstadt（图尔努尔德马斯特大学海斯塞人工智能中心）
