arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.03271cs.LGcs.AI
收藏

决策潜力面:大型语言模型决策边界的理论与实用近似

Decision Potential Surface: A Theoretical and Practical Approximation of Large Language Model Decision Boundary

The Hong Kong Polytechnic University(香港理工大学);University of Macau(澳门大学);Shanghai Jiaotong University(上海交通大学);Huawei(华为);Rochester Institute of Technology(罗切斯特理工学院);PolyU Research Centre for Privacy and Security Technologies in Future Smart Systems(PolyU未来智能系统隐私与安全技术研究中心)

Zi Liang, Zhiyao Wu, Haoyang Shang, Yulin Jin, Qingqing Ye, Huadi Zheng, Peizhao Hu, Haibo Hu

AI总结:

本文提出决策潜力面(DPS)作为一种新的分析大型语言模型决策性质的方法,通过K-DPS算法以有限样本近似决策边界,理论推导了误差上限,展示了误差与采样次数的权衡。

摘要:

决策边界,即模型赋予两个类别相等分类概率的输入子空间,在揭示核心模型属性和解释行为中起关键作用。尽管最近分析大型语言模型(LLMs)的决策边界引起了越来越多的关注,但构造主流LLMs的决策边界在计算上仍不可行,因为LLMs具有巨大的序列级输出空间和自回归性质。为了解决这个问题,本文提出决策潜力面(DPS),这是一种新的分析LLMs决策性质的概念。DPS来源于每个输入区分不同类别的置信度,自然捕捉了决策边界的潜力。我们证明了DPS中的零高度等高线等同于LLM的决策边界,封闭区域代表决策区域。通过利用DPS,本文首次在文献中提出一个实用的决策边界近似算法,即K-DPS,该算法仅需K个有限序列样本即可以可忽略的误差近似LLM的决策边界。我们理论推导了K-DPS与理想DPS之间绝对误差、期望误差和误差集中度的上限,证明了这些误差可以与采样次数进行权衡。

英文摘要:

Decision boundary, the subspace of inputs where a machine learning model assigns equal classification probabilities to two classes, is pivotal in revealing core model properties and interpreting behaviors. While analyzing the decision boundary of large language models (LLMs) has attracted increasing attention recently, constructing it for mainstream LLMs remains computationally infeasible due to the enormous sequence-level output spaces and the autoregressive nature of LLMs. To address this issue, in this paper we propose Decision Potential Surface (DPS), a new notion for analyzing the properties of LLM decisions. DPS is derived from the confidence in distinguishing different classes for each input, which naturally captures the potential of the decision boundary. We prove that the zero-height isohypse in DPS is equivalent to the decision boundary of an LLM, with enclosed regions representing decision regions. By leveraging DPS, for the first time in the literature, we propose a practical decision boundary approximation algorithm, namely K-DPS, which only requires only K finite sequence samples to approximate an LLM's decision boundary with negligible error. We theoretically derive the upper bounds for the absolute error, expected error, and the error concentration between K-DPS and the ideal DPS, demonstrating that such errors can be traded off against sampling times.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑