AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradients
Keck Data Science Institute, Pepperdine University(佩珀代因大学凯克数据科学研究所);Department of Medicine, University of California, Los Angeles(加利福尼亚大学洛杉矶分校医学院);Biological Sciences Division, University of Chicago(芝加哥大学生物科学部);Brain Research Institute, University of California, Los Angeles(加利福尼亚大学洛杉矶分校脑研究所)
Large language models (LLMs) have demonstrated increasingly sophisticated performance in medical and other fields of knowledge. Traditional methods of creating specialist LLMs require extensive fine-tuning and training of models on large datasets. Recently, prompt engineering, instead of fine-tuning, has shown potential to boost the performance of general foundation models. However, prompting methods such as chain-of-thought (CoT) may not be suitable for all subspecialty, and k-shot approaches may introduce irrelevant tokens into the context space. We present AutoMedPrompt, which explores the use of textual gradients to elicit medically relevant reasoning through system prompt optimization. AutoMedPrompt leverages TextGrad's automatic differentiation via text to improve the ability of general foundation LLMs. We evaluated AutoMedPrompt on Llama 3, an open-source LLM, using several QA benchmarks, including MedQA, PubMedQA, and the nephrology subspecialty-specific NephSAP. Our results show that prompting with textual gradients outperforms previous methods on open-source LLMs and surpasses proprietary models such as GPT-4, Claude 3 Opus, and Med-PaLM 2. AutoMedPrompt sets a new state-of-the-art (SOTA) performance on PubMedQA with an accuracy of 82.6$\%$, while also outperforming previous prompting strategies on open-sourced models for MedQA (77.7$\%$) and NephSAP (63.8$\%$).
REVOLVE: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization
Hong Kong University of Science and Technology(香港科技大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Brown University(布朗大学);University of Michigan - Ann Arbor(密歇根大学安娜堡分校);Hong Kong Polytechnic University(香港理工大学);Intel Lab(英特尔实验室)
From Weak Task Specifications to Scientific Extraction Agents: Optimizing Task Construction
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Cognitive Intelligence(认知智能全国重点实验室);CCCC Second Highway Consultants Co., Ltd.(中交第二公路勘察设计研究院有限公司)
SciTopic: Enhancing Topic Discovery in Scientific Literature through Advanced LLM
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China(中国科学院计算机网络信息中心);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学);Department of Computer Science, Portland State University, Portland, US(波特兰州立大学计算机科学系);Information Science and Technology College, Dalian Maritime University, Dalian, China(大连海事大学信息科学与技术学院)
Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning
Sun Yat-sen University(中山大学);Shenzhen Loop Area Institute(深圳河套学院);Beijing Zhongguancun Academy(北京中关村研究院);Huawei Technologies Co., Ltd(华为技术有限公司);Harbin Institute of Technology(哈尔滨工业大学)
LL4G: Self-Supervised Dynamic Optimization for Graph-Based Personality Detection
University of Southampton(南安普顿大学);University of Essex(埃塞克斯大学);Birkbeck, University of London(伦敦大学伯克贝克学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
A Similarity Paradigm Through Textual Regularization Without Forgetting
Shanghai Jiao Tong University(上海交通大学);Hong Kong Baptist University(香港浸会大学);Northeastern University(东北大学);University of Science and Technology of China(中国科学技术大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Leveraging Group Relative Policy Optimization to Advance Large Language Models in Traditional Chinese Medicine
University of Missouri(密苏里大学);Christopher S. Bond Life Sciences Center, University of Missouri(密苏里大学克里斯托弗·S·邦德生命科学中心);Community Health Service Center Shanghai Pudong New Area(上海市浦东新区社区卫生服务中心);Shanghai University of Traditional Chinese Medicine(上海中医药大学)