用于大语言模型生成文本的嵌入式条件独立性检验及其在德国议会演讲中的应用
Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speeches
浏览论文内容
中文总结 AI 辅助
该研究提出嵌入式条件独立性检验（eCITs），将其应用于德国议会演讲，发现大语言模型的摘要除来源演讲外，还携带发言者党派和性别的额外信息。
中文摘要 AI 辅助
条件独立性检验（CITs）用于检验在给定第三个随机对象Z的情况下，两个随机对象X和Y之间的条件依赖关系。现有CITs对高维数据的适用性有限，尤其是文本这类多模态数据。不过，我们表明这类检验对大语言模型（LLM）的输出具有研究价值，我们在此检验从源文本Z生成的输出X是否携带了Z本身之外关于属性Y的信息。为此，我们提出嵌入式CITs（eCITs），该方法对X和Z进行嵌入，并将现有CIT应用于得到的表示及Y。我们证明，若Z的嵌入是充分的（即保留了Z携带的关于Y或X表示的信息），则原假设会从X和Z转移到它们的表示，因此对嵌入假设有效的CIT对原假设也有效。我们还给出了两个假设等价的条件，并表明当嵌入检验针对条件均值独立性时，充分性会弱化为均值充分性。我们提出一种半合成模拟设计，以评估特定数据集和任务下，针对给定嵌入映射的eCITs的I类错误（T1E）控制和功效，并将其用于评估我们的应用。将eCITs应用于德国议会演讲，我们发现，对于所考虑的所有嵌入映射组合，两个LLM的摘要除了其生成来源的演讲之外，还包含关于发言者党派和性别的信息。
英文摘要
Conditional independence tests (CITs) test for conditional dependence between two random objects $X$ and $Y$ given a third random object $Z$. Existing CITs have limited applicability to high-dimensional data, especially multimodal data like text. However, we show that such tests are of interest for large language model (LLM) outputs, where we test whether an output $X$ generated from a source text $Z$ carries information about an attribute $Y$ beyond $Z$ itself. For this purpose, we propose embedded CITs (eCITs), which embed $X$ and $Z$ and apply an existing CIT to the resulting representations and to $Y$. We show that, provided the embedding of $Z$ is sufficient, i.e. retains the information $Z$ carries about either $Y$ or the representation of $X$, the null hypothesis transfers from $X$ and $Z$ to their representations, so that a CIT valid for the embedded hypothesis is valid for the original one. We further give conditions for equivalence of the two hypotheses, and show that sufficiency weakens to mean sufficiency when the embedded test targets conditional mean independence. We propose a semi-synthetic simulation design to assess type I error (T1E) control and power of the eCITs for given embedding maps on a specific dataset and task, and use it to evaluate them on our application. Applying the eCITs to German Parliament speeches, we find for all combinations of embedding maps considered that the summaries of two LLMs contain information about the speaker's faction and gender beyond the speech they were generated from.
发表机构
- Humboldt-Universität zu Berlin（柏林洪堡大学）
- Hasso-Plattner-Institut for Digital Engineering（哈索·普拉特纳数字工程研究所）
机构由 AI 辅助整理，请以论文原文为准。
