PixelBytes：捕捉多模态生成的统一表示
PixelBytes: Catching Unified Representation for Multimodal Generation
AI总结：
本文提出PixelBytes统一多模态表示方法，融合文本、音频、动作-状态和像素图像，通过PxBy嵌入及自回归模型实验证明其优于预测模型，并展示扩散模型在控制问题中的应用潜力。
摘要：
本报告提出了PixelBytes，一种用于统一多模态表示学习的方法。受图像Transformer、PixelCNN和Mamba-Bytes等序列模型的启发，我们探索将文本、音频、动作-状态以及像素化图像（精灵图）整合到一个统一的表示中。我们在PixelBytes Pokemon数据集和一个最优控制数据集上进行了实验。我们的研究涵盖了多种模型架构，包括循环神经网络（RNN）、状态空间模型（SSM）和基于注意力的模型，重点关注双向处理以及我们的PxBy嵌入技术。我们基于数据缩减策略和自回归学习对模型进行了评估，特别考察了长短期记忆（LSTM）网络在预测模式和自回归模式下的表现。结果表明，在此情境下自回归模型优于预测模型。此外，我们发现扩散模型可以应用于控制问题并实现并行化生成。PixelBytes旨在为多模态数据处理与生成的基础模型发展做出贡献。该项目的代码、模型和数据集均可在线上获取。
英文摘要：
This report presents PixelBytes, an approach for unified multimodal representation learning. Drawing inspiration from sequence models like Image Transformers, PixelCNN, and Mamba-Bytes, we explore integrating text, audio, action-state, and pixelated images (sprites) into a cohesive representation. We conducted experiments on a PixelBytes Pokemon dataset and an Optimal-Control dataset. Our investigation covered various model architectures, including Recurrent Neural Networks (RNNs), State Space Models (SSMs), and Attention-based models, with a focus on bidirectional processing and our PxBy embedding technique. We evaluated models based on data reduction strategies and autoregressive learning, specifically examining Long Short-Term Memory (LSTM) networks in predictive and autoregressive modes. Our results indicate that autoregressive models perform better than predictive models in this context. Additionally, we found that diffusion models can be applied to control problems and parallelized generation. PixelBytes aims to contribute to the development of foundation models for multimodal data processing and generation. The project's code, models, and datasets are available online.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
