LogProber：从LLM响应中的污染中解耦置信度
LogProber: Disentangling confidence from contamination in LLM responses
AI总结：
LogProber是一种高效的黑盒算法，通过关注问题熟悉度而非答案来检测LLM响应中的训练数据污染，并分析其与现有方法的优劣及局限性。
摘要：
在机器学习中，污染指的是测试数据泄漏到训练集中的情况。这一问题对于评估大型语言模型（LLMs）的性能尤为关键，因为LLMs通常是在从万维网抓取的庞大且通常不透明的文本语料库上进行训练的。因此，开发检测污染的工具对于公平且恰当地追踪LLMs性能的演变至关重要。迄今为止，只有少数近期研究尝试解决在短文本序列（如基准测试中常见的序列）中量化和检测污染的问题。然而，这些方法存在局限性，有时会使其不切实际。在本文中，我们介绍了LogProber，一种新颖且高效的算法，我们证明它能够在黑盒设置中检测污染，该算法试图通过关注对问题的熟悉度而非答案来克服上述一些缺点。在此，我们探讨了所提出方法与同期方法相比的特性，识别了其优势和局限性，并说明了根据检测算法的设计，不同形式的污染如何可能未被检测到。
英文摘要：
In machine learning, contamination refers to situations where testing data leak into the training set. The issue is particularly relevant for the evaluation of the performance of Large Language Models (LLMs), which are generally trained on gargantuan, and generally opaque, corpora of text scraped from the world wide web. Developing tools to detect contamination is therefore crucial to be able to fairly and properly track the evolution of the performance of LLMs. To date, only a few recent studies have attempted to address the issue of quantifying and detecting contamination in short text sequences, such as those commonly found in benchmarks. However, these methods have limitations that can sometimes render them impractical. In the present paper, we introduce LogProber, a novel, efficient algorithm that we show to be able to detect contamination in a black box setting that tries to tackle some of these drawbacks by focusing on the familiarity with the question rather than the answer. Here, we explore the properties of the proposed method in comparison with concurrent approaches, identify its advantages and limitations, and illustrate how different forms of contamination can go undetected depending on the design of the detection algorithm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
