ByteAction：字节空间动作识别基础模型

中文摘要 AI 辅助

字节空间动作识别（BAR）旨在直接从压缩图像比特流中识别人类动作，无需任何像素解码。由于完全在字节空间中运行，BAR本质上不依赖文件完整性和像素级重建，因此天然适用于隐私敏感场景，且对比特流损坏具有鲁棒性。本文提出ByteAction，一种可在损坏的图像比特流上实现准确动作识别的BAR基础模型。ByteAction采用双视图字节级识别框架，构建弱损坏和强损坏的比特流视图，通过比特流模式增强（BPA）进行数据增强，并由共享的ByteFormer骨干网络编码。该模型通过分类和损坏一致性两个目标进行优化。具体而言，本文提出比特流模式增强（BPA），将一维字节序列重塑为二维字节矩阵，并应用区域级擦除，以鼓励模型学习鲁棒的跨区域字节依赖关系。本文进一步提出损坏一致性训练策略，通过双向KL散度约束模型在不同损坏程度下保持稳定的预测。在Stanford40、PPMI和PASCAL VOC 2012 Action的图像比特流上开展的实验表明，ByteAction在所有场景下均实现了最先进的损坏鲁棒性，同时保持了与完整比特流相当的性能。

英文摘要

Byte-space Action Recognition (BAR) aims to recognize human actions directly from compressed image bitstreams without any pixel decoding. By operating entirely in byte space, BAR is inherently independent of file integrity and pixel-level reconstruction, making it naturally applicable to privacy-sensitive scenarios and robust against bitstream corruption. In this paper, we propose ByteAction, a BAR foundation model that achieves accurate action recognition on corrupted image bitstreams. ByteAction follows a dual-view byte-level recognition framework. It constructs weakly and strongly corrupted bitstream views, which are augmented by Bitstream Pattern Augmentation (BPA) and encoded with a shared ByteFormer backbone. The model is optimized with both classification and corruption consistency objectives. Specifically, we propose Bitstream Pattern Augmentation (BPA), which reshapes one-dimensional byte sequences into two-dimensional byte matrix and applies region-level erasure to encourage the model to learn robust cross-region byte dependencies. We further propose a Corruption Consistency Training strategy that constrains the model to maintain stable predictions across different corruption severities through bidirectional KL divergence. Experiments on the image bitstream from Stanford40, PPMI, and PASCAL VOC 2012 Action demonstrate that ByteAction achieves state-of-the-art corruption robustness across all scenarios while maintaining competitive intact bitstream performance.

发表机构

• School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）

机构由 AI 辅助整理，请以论文原文为准。