JMedEthicBench：用于评估日语大语言模型医疗安全性的多轮对话基准
JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models
AI总结：
提出首个多轮对话基准JMedEthicBench，基于日本医学会67条指南和7种自动越狱策略生成5万+对抗对话，评估27个模型发现医疗专用模型安全性脆弱，且多轮交互中安全性显著下降。
摘要：
随着大语言模型（LLM）在医疗领域的部署日益增多，在临床使用前仔细评估其医疗安全性变得至关重要。然而，现有的安全基准仍然以英语为中心，并且仅使用单轮提示进行测试，尽管临床咨询是多轮的。为了解决这些差距，我们引入了JMedEthicBench，这是第一个用于评估日语医疗LLM医疗安全性的多轮对话基准。我们的基准基于日本医学会的67条指南，包含使用七种自动发现的越狱策略生成的超过50,000个对抗性对话。使用双LLM评分协议，我们评估了27个模型，发现商业模型保持了稳健的安全性，而医疗专用模型表现出更高的脆弱性。此外，安全分数在对话轮次中显著下降（中位数：9.5降至5.0，p < 0.001）。对我们的基准的日语和英语版本进行的跨语言评估表明，医疗模型的脆弱性跨语言持续存在，表明存在固有的对齐限制，而非语言特定因素。这些发现表明，领域特定的微调可能会意外削弱安全机制，并且多轮交互代表了一个需要专门对齐策略的独特威胁面。
英文摘要：
As Large Language Models (LLMs) are increasingly deployed in healthcare field, it becomes essential to carefully evaluate their medical safety before clinical use. However, existing safety benchmarks remain predominantly English-centric, and test with only single-turn prompts despite multi-turn clinical consultations. To address these gaps, we introduce JMedEthicBench, the first multi-turn conversational benchmark for evaluating medical safety of LLMs for Japanese healthcare. Our benchmark is based on 67 guidelines from the Japan Medical Association and contains over 50,000 adversarial conversations generated using seven automatically discovered jailbreak strategies. Using a dual-LLM scoring protocol, we evaluate 27 models and find that commercial models maintain robust safety while medical-specialized models exhibit increased vulnerability. Furthermore, safety scores decline significantly across conversation turns (median: 9.5 to 5.0, $p < 0.001$). Cross-lingual evaluation on both Japanese and English versions of our benchmark reveals that medical model vulnerabilities persist across languages, indicating inherent alignment limitations rather than language-specific factors. These findings suggest that domain-specific fine-tuning may accidentally weaken safety mechanisms and that multi-turn interactions represent a distinct threat surface requiring dedicated alignment strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
