Tabby：一种用于表格和结构化数据合成的语言模型架构
Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis
AI总结：
Tabby是一种用于表格和结构化数据合成的语言模型架构，通过门控专家混合机制提升数据生成质量，结合新型训练技术实现44%的质量提升。
摘要：
尽管大语言模型（LLMs）的进步近年来显著提升了合成文本数据的质量，但合成表格数据却受到相对较少的关注。我们通过Tabby，一种简单的但强大的后训练修改标准Transformer语言模型架构的方法，来解决这一不平衡问题，使该架构能够用于表格数据集的合成。Tabby使用门控专家混合机制来表示列之间的差异，每个列有特定的参数集。实证结果显示，Tabby生成的数据质量接近或等于真实数据。通过将我们的新型LLM表格训练技术Plain与Tabby结合，我们观察到相对于先前方法的质量提升高达44%。我们还展示Tabby不仅限于表格，还能扩展到更一般的结构化数据，在嵌套JSON数据集上也达到了与真实数据相媲美的水平。
英文摘要：
While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
