下游知识基准测试无需数据集：响应离散度与特定领域问答准确率呈负相关
No Dataset Needed for Downstream Knowledge Benchmarking: Response Dispersion Inversely Correlates with Accuracy on Domain-specific QA
AI总结：
本研究提出“响应离散度”指标，通过反复询问LLM意见问题并计算响应嵌入矩阵的奇异值数量，替代传统QA数据集评估，发现其与准确率呈负相关，为终端用户免数据集比较LLM知识提供了新方案。
摘要：
本研究旨在比较LLM在特定主题领域的知识时，免除创建QA数据集和为（聊天机器人）LLM响应评分的需求。该方法完全以终端用户为中心，无需访问LLM的任何内部机制，只要能对其进行提示并提供随机种子以对同一提示生成不同响应即可。本文针对给定主题领域，通过反复向LLM询问关于该领域的同一意见问题，定义了LLM的“响应离散度”。即，响应离散度是解释LLM响应嵌入矩阵95%方差所需的奇异值数量。研究发现，响应离散度与相关QA评估的准确率呈负相关（平均Spearman秩相关强于-.59）。用例分析表明，在同一主题领域比较两个不同LLM时，比较它们的响应离散度在74%到89%的情况下是替代比较其QA准确率的合适方案，该范围取决于终端用户可能接受的合理准确率差异容忍度，以此换取使用响应离散度代替QA准确率进行比较所节省的劳动力。本研究探讨了两种响应嵌入来创建嵌入矩阵，一种来自OpenAI的API，另一种是本文提出的名为参考句子相似性嵌入的新型嵌入，它可在本地计算，且在计算响应离散度时表现几乎同样出色。此外，本研究还将一个名为IRC-Wiki Trivia dataset的现有数据集（最初为问答游戏开发）重新利用和整理，并将该整理版本命名为IRC-WikiTriviaQA，供本研究使用。
英文摘要：
This research seeks to obviate the need for creating QA datasets and grading (chatbot) LLM responses when comparing LLMs' knowledge in specific topic domains. This is done in an entirely end-user centric way without need for access to any inner workings of the LLM, so long as it can be prompted and given a random seed to create different generations to the same prompt. The paper does this by, for a given topic domain, defining the "response dispersion" of an LLM by repeatedly asking an LLM the same opinion question about that topic domain. Namely, the response dispersion is the count of singular values needed to explain 95% of the variance in the embedding matrix of the LLM's responses. It is found that the response dispersion is inversely correlated with accuracy on relevant QA evaluations (average spearman rank correlation stronger than -.59). A use-case analysis shows that when comparing two different LLMs on the same topic domain, comparing their response dispersion is a suitable replacement for comparing their QA accuracy between 74% and 89% of the time, the range depending on certain reasonable accuracy-difference tolerances that may be acceptable to an end-user in exchange for the labor being saved using response dispersion instead of QA accuracy for comparison. Two response embeddings are studied for creating the embedding matrix in this study, one is from OpenAI's APIs and one is a novel embedding, here named reference sentence similarity embeddings, that can be computed locally and performs very nearly as well in calculating response dispersion. Also in this research, a pre-existing dataset called the IRC-Wiki Trivia dataset, originally developed for trivia games, has been re-purposed, curated, and the curation, called IRC-WikiTriviaQA, is made available for the purpose of this research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
