PCS：基于蜕变关系的黑盒大语言模型感知置信度评分
PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relations
AI总结：
本文提出基于蜕变关系的感知置信度评分PCS，通过分析大语言模型在语义等价文本变体上预测标签的一致性来评估置信度，从而提升零样本LLM文本分类性能，并在单模型和多模型投票场景中分别实现9.3%和5.8%的性能提升。
摘要：
零样本大语言模型（LLMs）如今也被用于文本分类任务，例如对句子或文章进行情感与偏见检测。然而，在这类数据标注任务中，它们的表现可能并非最优。我们提出一种新技术，通过利用蜕变关系（Metamorphic Relations, MRs）来评估大语言模型对文本输入进行分类时的置信度。这些蜕变关系会生成输入在语义上等价但文本上存在差异的版本。遵循蜕变测试（Metamorphic Testing, MT）的原则，变异后的版本预期应具有与原始输入相似的标注标签。通过分析大语言模型在这些变体上响应的一致性，我们根据预测标签的出现频率计算感知置信度分数（Perceived Confidence Score, PCS）。PCS 既可用于单一大语言模型场景，也可用于多模型场景（例如在多数投票机制下对多个大语言模型进行审查时）。实证评估表明，我们基于 PCS 的方法将零样本大语言模型在文本分类任务中的性能提升了 9.3%。当多个大语言模型以多数投票方式使用时，借助 PCS 我们获得了 5.8% 的性能提升。
英文摘要：
Zero-shot LLMs are now also used for textual classification tasks, e.g., sentiment and bias detection in a sentence or article. However, their performance can be suboptimal in such data annotation tasks. We introduce a novel technique that evaluates an LLM's confidence for classifying a textual input by leveraging Metamorphic Relations (MRs). The MRs generate semantically equivalent yet textually divergent versions of the input. Following the principles of Metamorphic Testing (MT), the mutated versions are expected to have annotation labels similar to the input. By analyzing the consistency of an LLM's responses across these variations, we compute a perceived confidence score (PCS) based on the frequency of the predicted labels. PCS can be used for both single and multiple LLM settings (e.g., when multiple LLMs are vetted in a majority-voting setup). Empirical evaluation shows that our PCS-based approach improves the performance of zero-shot LLMs by 9.3% in textual classification tasks. When multiple LLMs are used in a majority-voting setup, we obtain a performance boost of 5.8% with PCS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
