约化密度下的语言建模
Language Modeling with Reduced Densities
AI总结：
针对非结构化文本的数学结构问题，本文用充实范畴论刻画有限字母符号序列，并构造到约化密度算子范畴的函子，以保留范畴结构并通过 Loewner 序表示蕴涵。
摘要：
本工作源于如下观察：当今最先进的统计语言模型令人印象深刻，不仅在于其性能，而且——也相当关键——在于它们完全由非结构化文本数据中的相关性构建而成。后一观察引出了本文核心的一个根本问题：非结构化文本数据中存在何种数学结构？我们提出以充实范畴论作为自然的回答。我们表明，来自有限字母表的符号序列（例如文本语料库中发现的序列）构成一个在概率上充实的范畴。随后，我们处理第二个根本问题：如何以保留范畴结构的方式存储和建模这些信息？为此，我们构造了一个从文本的充实范畴到某一特定的约化密度算子充实范畴的函子。后者利用了半正定算子上的 Loewner 序，该序还可进一步解释为蕴涵关系的一个玩具示例。
英文摘要：
This work originates from the observation that today's state-of-the-art statistical language models are impressive not only for their performance, but also - and quite crucially - because they are built entirely from correlations in unstructured text data. The latter observation prompts a fundamental question that lies at the heart of this paper: What mathematical structure exists in unstructured text data? We put forth enriched category theory as a natural answer. We show that sequences of symbols from a finite alphabet, such as those found in a corpus of text, form a category enriched over probabilities. We then address a second fundamental question: How can this information be stored and modeled in a way that preserves the categorical structure? We answer this by constructing a functor from our enriched category of text to a particular enriched category of reduced density operators. The latter leverages the Loewner order on positive semidefinite operators, which can further be interpreted as a toy example of entailment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
