MedHopQA 挑战为大型语言模型(LLM)提供了一个关键测试:在高风险的生物医学领域中进行复杂的多跳推理。本文详细介绍了我们对 Google Gemini Flash 模型的直接基于 API 的评估,重点关注高级提示工程的影响。我们为 Gemini 2.0 Flash 设计了一个复杂的多组件提示,结合了角色扮演、显式的多步思维链(CoT)示例和详细的格式规则。使用这个复杂提示的最佳运行获得了0.720的概念级得分。这一结果显著优于仅得0.565的基线提示。值得注意的是,在高效的 Gemini 2.0 Flash 上的性能与下一代 Gemini 2.5 Flash 的结果几乎相同。我们的发现表明,复杂的提示设计是释放现代LLM全部推理能力的关键因素。
英文摘要:
The MedHopQA challenge presents a critical test for Large Language Models (LLMs): complex, multi-hop reasoning in the high-stakes biomedical domain. This paper details our direct API-based evaluation of Google's Gemini Flash models, focusing on the impact of advanced prompt engineering. We designed a sophisticated, multi-component prompt for Gemini 2.0 Flash that combined role-playing, explicit multi-shot Chain-of-Thought (CoT) examples, and detailed formatting rules. Our best run, using this complex prompt, achieved a Concept Level Score of 0.720. This result dramatically outperformed a baseline prompt which scored only 0.565. Remarkably, this performance on the efficient Gemini 2.0 Flash was almost identical to the result from the next-generation Gemini 2.5 Flash. Our findings demonstrate that sophisticated prompt design is a critical factor for unlocking the full reasoning capabilities of modern LLMs.
Gender and content bias in Large Language Models: a case study on Google Gemini 2.0 Flash Experimental
University of Perugia(佩鲁贾大学);Chapin Hall at University of Chicago(芝加哥大学查平学院);Berdyansk State Pedagogical University(别尔江斯克国立师范大学);Università di Bologna(博洛尼亚大学);University of North Texas(北得克萨斯大学);Chapin Hall(查平学院)
H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking
Duke University(杜克大学);Accenture(埃森哲);Accenture Security(埃森哲安全)
Benchmarking Large Language Models for Biomedical Relation Extraction
Interdisciplinary School of Doctoral Studies(跨学科博士研究学院);Faculty of Mathematics and Computer Science, University of Bucharest(布加勒斯特大学数学与计算机科学学院);HLT Research Center(HLT研究中心)
Benchmarking Next-Generation Reasoning-Focused Large Language Models in Ophthalmology: A Head-to-Head Evaluation on 5,888 Items
National University of Singapore(新加坡国立大学);Singapore Eye Research Institute(新加坡眼科研究所);Singapore National Eye Centre(新加坡国家眼科中心);The Chinese University of Hong Kong(香港中文大学);Yale University(耶鲁大学);Cleveland Clinic(克利夫兰诊所);Montreal(蒙特利尔);Tan Tock Seng Hospital(陈笃生医院);Stanford University(斯坦福大学)
Multimodal Human-AI Synergy for Medical Imaging Quality Control: A Hybrid Intelligence Framework with Adaptive Dataset Curation and Closed-Loop Evaluation
Zhongda Hospital(中大医院);School of Medicine, Southeast University(东南大学医学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases
Center for Alzheimer’s and Related Dementias, NIA, NIH(阿尔茨海默病及相关痴呆症研究中心,国家老龄化研究所,国立卫生研究院);DataTecnica LLC(DataTecnica公司);Johns Hopkins University(约翰霍普金斯大学);Laboratory of Neurogenetics, NIA, NIH(神经遗传学实验室,国家老龄化研究所,国立卫生研究院)
Prompt Engineering Strategies for LLM-based Qualitative Coding of Psychological Safety in Software Engineering Communities: A Controlled Empirical Study
Federal University of Bahia(巴伊亚联邦大学);State University of Feira de Santana(费拉德桑塔纳州立大学);Federal University of Sergipe(塞格皮联邦大学);Federal Institute of Ceara(塞阿拉联邦理工学院)
ChatGPT and Gemini participated in the Korean College Scholastic Ability Test -- Earth Science I
Institute for Research Excellence in Learning Sciences(学习科学卓越研究所);Seoul National University(首尔大学);National Taiwan Normal University(台湾师范大学);Universitas Negeri Malang(玛琅国立大学)
Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning
Tencent AI for Life Sciences Lab(腾讯AI生命科学实验室);Sun Yat-sen University(中山大学);The Hong Kong Polytechnic University(香港理工大学);Westlake University(西湖大学);The Hong Kong University of Science and Technology(香港科技大学)
Ensembles of Large Language Models for Identifying EQ-5D Studies in PubMed Based on Their Abstracts
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University(欧布达大学应用信息学与应用数学博士学院);John von Neumann Faculty of Informatics, Obuda University(欧布达大学约翰·冯·诺伊曼信息学学院);Doctoral School of Innovation Management, Obuda University(欧布达大学创新管理博士学院)
Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
Department of Computer Engineering, Hacettepe University(哈切塔佩大学计算机工程系);Department of Computer Engineering, Ankara University(安卡拉大学计算机工程系);Zephlen AI and Information Technologies Inc.(泽夫伦人工智能与信息技术公司)