By simply composing prompts, developers can prototype novel generative applications with Large Language Models (LLMs). To refine prototypes into products, however, developers must iteratively revise prompts by evaluating outputs to diagnose weaknesses. Formative interviews (N=8) revealed that developers invest significant effort in manually evaluating outputs as they assess context-specific and subjective criteria. We present EvalLM, an interactive system for iteratively refining prompts by evaluating multiple outputs on user-defined criteria. By describing criteria in natural language, users can employ the system's LLM-based evaluator to get an overview of where prompts excel or fail, and improve these based on the evaluator's feedback. A comparative study (N=12) showed that EvalLM, when compared to manual evaluation, helped participants compose more diverse criteria, examine twice as many outputs, and reach satisfactory prompts with 59% fewer revisions. Beyond prompts, our work can be extended to augment model evaluation and alignment in specific application contexts.
Read, Diagnose and Chat: Towards Explainable and Interactive LLMs-Augmented Depression Detection in Social Media
Hefei University of Technology（合肥工业大学）；University of Science and Technology of China（中国科学技术大学）；East China Normal University（华东师范大学）；Pennsylvania State University（宾夕法尼亚州立大学）
PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）；Laboratory of Art and Archaeology Image (Zhejiang University), Ministry of Education（浙江大学艺术与考古图像实验室（教育部））；Hong Kong University of Science and Technology（香港科技大学）；Zhejiang University（浙江大学）
MemoCare: An Interactive Multimodal Mobile System for Automated Cognitive Screening
Lausanne University Hospital（洛桑大学医院）；Faculty of Biology and Medicine, University of Lausanne（洛桑大学生物与医学学院）；VNU University of Engineering and Technology（越南国立大学工程技术大学）；VinUni-Illinois Smart Health Center, VinUniversity（文大-伊利诺伊智能健康中心，文大）；University of Science, VNU-HCM（胡志明市越南国立大学科学大学）；Hanoi Medical University（河内医科大学）；National Geriatric Hospital（国家老年病医院）；Military Hospital 175（第175军医院）；School of Medicine, University of Medicine and Pharmacy at Ho Chi Minh City（胡志明市医药大学医学院）；International University, VNU-HCM（胡志明市越南国立大学国际大学）；Vietnam National University Ho Chi Minh City（胡志明市越南国立大学）
