arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.01825cs.CLcs.AI
收藏

分形模式或可揭示下一词元预测的成功之道

Fractal Patterns May Illuminate the Success of Next-Token Prediction

Google Deepmind(谷歌DeepMind)

Ibrahim Alabdulmohsin, Vinh Q. Tran, Mostafa Dehghani

AI总结:

本文通过证明语言的自相似性和长程依赖性(H≈0.7),揭示分形参数可预测LLM下游性能,为下一词元预测的成功提供新解释。

摘要:

我们研究语言的分形结构,旨在为可能先前已被猜测但未得到正式证明的属性提供精确的形式化描述。我们证实语言具有:(1) 自相似性,即在所有粒度层级上都表现出复杂性,且不存在特定的特征上下文长度;(2) 长程依赖性(LRD),其赫斯特参数约为 H=0.7。基于这些发现,我们认为语言中的短期模式/依赖关系(如段落内)反映了更大范围(如整篇文档)中的模式/依赖关系。这可能有助于阐明下一词元预测如何跨多个粒度层级(从单词和从句到更广泛的上下文和意图)捕捉文本结构。此外,我们跨不同领域和架构进行了广泛分析,表明分形参数具有稳健性。最后,我们证明,在预测大型语言模型(LLM)的下游性能时,各LLM之间分形参数的微小变化优于基于困惑度的每字节比特数(BPB)指标。我们希望这些发现能为语言及LLM成功背后的机制提供新的视角。

英文摘要:

We study the fractal structure of language, aiming to provide a precise formalism for quantifying properties that may have been previously suspected but not formally shown. We establish that language is: (1) self-similar, exhibiting complexities at all levels of granularity, with no particular characteristic context length, and (2) long-range dependent (LRD), with a Hurst parameter of approximately H=0.7. Based on these findings, we argue that short-term patterns/dependencies in language, such as in paragraphs, mirror the patterns/dependencies over larger scopes, like entire documents. This may shed some light on how next-token prediction can capture the structure of text across multiple levels of granularity, from words and clauses to broader contexts and intents. In addition, we carry out an extensive analysis across different domains and architectures, showing that fractal parameters are robust. Finally, we demonstrate that the tiny variations in fractal parameters seen across LLMs improve upon perplexity-based bits-per-byte (BPB) in predicting their downstream performance. We hope these findings offer a fresh perspective on language and the mechanisms underlying the success of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑