鲁棒的模拟硬件参数高效大语言模型适配
Robust Parameter-Efficient LLM Adaptation on Analog Hardware
浏览论文内容
中文总结 AI 辅助
针对模拟硬件上大语言模型微调受硬件非理想性影响的问题，提出基于LoRA的参数高效适配方法，通过输入重塑和更新累积改善噪声MVM和有限状态编程下的适配性能。
中文摘要 AI 辅助
模拟存内计算是一种有前景的用于设备端执行大语言模型的平台，因为它能在内存中并行执行矩阵-向量乘法（MVM），从而减少数据移动。然而，有限的数模转换器精度、输入噪声和有限的电导状态会降低模型精度，而针对这些影响进行全模型重训练可能代价高昂。我们开发了一种基于低秩适配（LoRA）的、与优化器无关的参数高效适配方法，将存储在模拟阵列上的预训练权重固定，同时训练LoRA权重以适应下游任务和硬件非理想性。可靠的适配需要处理前向和后向MVM以及物理权重更新中的错误。我们使用输入重塑来减少输入引起的MVM误差，并使用更新累积来保留小的更新，然后再将其编程到有限状态的模拟设备中。在Llama-3.2-1B-Instruct和Llama-3-8B上，使用Muon和AdamW优化器，输入重塑在噪声MVM计算下改善了模拟LoRA微调。更新累积单独保留了低于阈值的更新，并在有限分辨率编程下显著改善了适配，包括电导状态少至20种的配置。额外的实验表明，在噪声模拟设置下，保留的负对数似然度一致改善。
英文摘要
Analog in-memory computing is a promising platform for on-device execution of large language models because it performs matrix--vector multiplications (MVMs) in memory and in parallel, reducing data movement. However, limited digital-to-analog converter precision, input noise, and finite conductance states can degrade model accuracy, while full-model retraining to address these effects can be costly. We develop an optimizer-agnostic, parameter-efficient adaptation method based on Low-Rank Adaptation (LoRA), keeping the pretrained weights stored on analog arrays fixed while training the LoRA weights to adapt to downstream tasks and hardware non-idealities. Reliable adaptation requires handling errors in both forward and backward MVMs and physical weight updates. We use input reshaping to reduce input-induced MVM errors and update accumulation to retain small updates before programming them to finite-state analog devices. Across Llama-3.2-1B-Instruct and Llama-3-8B with both Muon and AdamW, input reshaping improves analog LoRA fine-tuning under noisy MVM computation. Update accumulation separately preserves sub-threshold updates and substantially improves adaptation under finite-resolution programming, including configurations with as few as 20 conductance states. Additional experiments show consistent held-out negative log-likelihood improvements across noisy analog settings.
发表机构
- Cornell University（康奈尔大学）
机构由 AI 辅助整理，请以论文原文为准。
