揭示文本的内在维度：从学术摘要到创意故事
Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story
AI总结：
本研究针对大语言模型分析中文本内在维度的决定因素不明的问题，结合交叉编码器、语言学特征与稀疏自编码器开展分析，明确了内在维度的特性、体裁分层规律与因果影响特征，为内在维度的应用与解读提供指导。
摘要：
内在维度（ID）是现代大语言模型（LLM）分析中的重要工具，可为训练动态、缩放行为与数据集结构相关研究提供参考，但其对应的文本决定因素仍未得到充分探索。本研究首次开展综合性研究，通过交叉编码器分析、语言学特征与稀疏自编码器（SAE）将内在维度锚定到可解释的文本属性上。本研究得到三项核心发现：第一，内在维度与基于熵的指标互补：控制文本长度后二者无相关性，内在维度捕捉的是与预测质量正交的几何复杂度。第二，内在维度呈现稳健的体裁分层特性：在所有测试模型中，科学散文的内在维度较低（约8），百科内容为中等水平（约9），创意/观点类写作的内在维度较高（约10.5）；这表明当代大语言模型认为科学文本“表征简单”，而小说类内容需要额外的自由度。第三，借助稀疏自编码器，我们识别出因果特征：科学类信号（正式语气、报告模板、统计数据）会降低内在维度，拟人化信号（个性化表达、情感、叙事）会提升内在维度，引导实验证实了这些效应的因果性。因此对当代模型而言，科学写作相对“简单”，而小说、观点与情感类内容会增加表征自由度。本研究的多维度分析为合理使用内在维度、正确解读基于内在维度的结果提供了实践指导。
英文摘要：
Intrinsic dimension (ID) is an important tool in modern LLM analysis, informing studies of training dynamics, scaling behavior, and dataset structure, yet its textual determinants remain underexplored. We provide the first comprehensive study grounding ID in interpretable text properties through cross-encoder analysis, linguistic features, and sparse autoencoders (SAEs). In this work, we establish three key findings. First, ID is complementary to entropy-based metrics: after controlling for length, the two are uncorrelated, with ID capturing geometric complexity orthogonal to prediction quality. Second, ID exhibits robust genre stratification: scientific prose shows low ID (~8), encyclopedic content medium ID (~9), and creative/opinion writing high ID (~10.5) across all models tested. This reveals that contemporary LLMs find scientific text "representationally simple" while fiction requires additional degrees of freedom. Third, using SAEs, we identify causal features: scientific signals (formal tone, report templates, statistics) reduce ID; humanized signals (personalization, emotion, narrative) increase it. Steering experiments confirm these effects are causal. Thus, for contemporary models, scientific writing appears comparatively "easy", whereas fiction, opinion, and affect add representational degrees of freedom. Our multi-faceted analysis provides practical guidance for the proper use of ID and the sound interpretation of ID-based results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
