Bioalignment: 评估和改进LLM对生物系统的倾向以提高AI安全性
Bioalignment: Measuring and Improving LLM Disposition Toward Biological Systems for AI Safety
AI总结:
本研究通过微调改进LLM对生物解决方案的偏好,表明少量微调可提升模型对生物与合成方法的权衡能力。
摘要:
大型语言模型(LLMs)在互联网级语料上进行训练时可能会表现出系统性偏见,这种偏见会增加产生不期望行为的可能性。在本研究中,我们检查了在四个领域(材料、能源、制造和算法)中合成与生物技术解决方案之间的潜在偏见。通过使用50个精心挑选的Bioalignment提示和受凯利准则启发的评估框架,对5个前沿模型和5个开源模型进行了测量。根据这一指标,大多数模型并未实现生物对齐,即它们倾向于支持合成(非生物)解决方案。我们接下来检查了微调是否能增加两个开源模型(Llama 3.2-3B-Instruct和Qwen2.5-3B-Instruct)对基于生物的方法的偏好。首先使用约2200万token的定制语料库(来自6636篇强调生物问题解决的PMC文章)对Llama 3B进行微调,混合了持续训练和指令格式的语料库。然后扩展到Qwen 3B,仅使用指令格式的语料库。我们发现,QLoRA微调显著提高了两个模型对生物解决方案的评分,而不会降低一般能力(Holm-Bonferroni校正后的p < 0.001和p < 0.01)。这表明,即使少量微调也能改变模型对生物和生物启发式与合成方法相对价值的权衡方式。尽管本研究集中在小型开源LLM上,但可能可以扩展到更大模型,并可用于开发更倾向于生物方法的模型。我们发布了基准、语料库、代码和适配器权重。
英文摘要:
Large language models (LLMs) trained on internet-scale corpora can exhibit systematic biases that increase the probability of unwanted behavior. In this study, we examined potential biases towards synthetic vs. biological technological solutions across four domains (materials, energy, manufacturing, and algorithms). A sample of 5 frontier and 5 open-weight models were measured using 50 curated Bioalignment prompts with a Kelly criterion-inspired evaluation framework. According to this metric, most models were not bioaligned in that they exhibit biases in favor of synthetic (non-biological) solutions. We next examined if fine-tuning could increase the preferences of two open-weight models, Llama 3.2-3B-Instruct and Qwen2.5-3B-Instruct, for biological-based approaches. A curated corpus of ~22M tokens from 6,636 PMC articles emphasizing biological problem-solving was used first to fine-tune Llama 3B with a mixed corpus of continued training and instruction-formatted. This was then extended to Qwen 3B using instruction-formatted only. We found that QLoRA fine-tuning significantly increased the scoring of biological solutions for both models without degrading general capabilities (Holm-Bonferroni-corrected p < 0.001 and p < 0.01, respectively). This suggests that even a small amount of fine-tuning can change how models weigh the relative value of biological and bioinspired vs. synthetic approaches. Although this work focused on small open-weight LLMs, it may be extensible to much larger models and could be used to develop models that favor bio-based approaches. We release the benchmark, corpus, code, and adapter weights.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。