量化表格LLM中微调多重性下的预测一致性
Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
AI总结:
针对表格LLM微调多重性导致预测冲突的问题,本文提出一种基于嵌入空间局部采样的度量方法,无需重训练即可量化预测一致性,并提供概率保证,在多个真实数据集上验证其优于现有方法。
摘要:
在表格分类任务上微调大语言模型(LLM)可能导致微调多重性现象,即性能同样良好的模型对相同输入做出相互冲突的预测。微调多重性可能源于训练过程中的变化,例如随机种子、权重初始化、训练数据的微小改动等,这引发了人们对表格LLM在金融、招聘、教育、医疗等高风险应用中可靠性的担忧。我们的工作形式化了表格LLM中这一独特的微调多重性挑战,并提出一种新的度量方法,用于在无需昂贵模型重训练的情况下量化单个预测的一致性。我们的度量通过分析(采样)模型在嵌入空间中该输入周围的局部行为来量化预测的一致性。有趣的是,我们表明可以利用局部邻域中的采样,在一大类微调模型下为预测一致性提供概率保证,即具有足够高局部稳定性(由我们的度量定义)的输入也以高概率在多个微调模型之间保持一致。我们在多个真实世界数据集上进行实验,表明我们的局部稳定性度量能够预先捕捉多个微调模型在实际多重性下的一致性,优于其他竞争度量。
英文摘要:
Fine-tuning LLMs on tabular classification tasks can lead to the phenomenon of fine-tuning multiplicity where equally well-performing models make conflicting predictions on the same input. Fine-tuning multiplicity can arise due to variations in the training process, e.g., seed, weight initialization, minor changes to training data, etc., raising concerns about the reliability of Tabular LLMs in high-stakes applications such as finance, hiring, education, healthcare. Our work formalizes this unique challenge of fine-tuning multiplicity in Tabular LLMs and proposes a novel measure to quantify the consistency of individual predictions without expensive model retraining. Our measure quantifies a prediction's consistency by analyzing (sampling) the model's local behavior around that input in the embedding space. Interestingly, we show that sampling in the local neighborhood can be leveraged to provide probabilistic guarantees on prediction consistency under a broad class of fine-tuned models, i.e., inputs with sufficiently high local stability (as defined by our measure) also remain consistent across several fine-tuned models with high probability. We perform experiments on multiple real-world datasets to show that our local stability measure preemptively captures consistency under actual multiplicity across several fine-tuned models, outperforming competing measures.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。