arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.16918cs.CVcs.AI
收藏

Xray-Visual模型:在产业级数据上扩展视觉模型

Xray-Visual Models: Scaling Vision models on Industry Scale Data

Meta-AI;MIT(麻省理工学院);University of Maryland(马里兰大学)
Shlok Mishra, Tsung-Yu Lin, Linda Wang, Hongli Xu, Yimin Liu, Michael Hsu, Chaitanya Ahuja, Hao Yuan, Jianpeng Cheng, Hong-You Chen, Haoyuan Xu, Chao Li, Sreya … 展开作者

Shlok Mishra, Tsung-Yu Lin, Linda Wang, Hongli Xu, Yimin Liu, Michael Hsu, Chaitanya Ahuja, Hao Yuan, Jianpeng Cheng, Hong-You Chen, Haoyuan Xu, Chao Li, Sreya Dutta Roy, Abhijeet Awasthi, Jihye Moon, Don Husa, Michael Ge, Sumedha Singla, Arkabandhu Chowdhury, Phong Dingh, Satya Narayan Shukla, Yonghuan Yang, David Jacobs, Qi Guo, Jun Xiao, Xiangjun Fan, Aashu Singh

AI总结:

Xray-Visual通过三阶段训练流程和LLM2CLIP技术,在产业级数据上实现高效多模态视觉模型,取得最佳性能并提升鲁棒性与泛化能力。

摘要:

我们提出了Xray-Visual,一种统一的视觉模型架构,用于大规模图像和视频理解,该模型在Facebook和Instagram上的产业级社交媒体数据上进行训练。我们的模型利用超过150亿个经过精心挑选的图像-文本对和100亿个视频-标签对,采用稳健的数据整理流程,结合平衡和噪声抑制策略,以最大化语义多样性同时最小化标签噪声。我们引入了一个三阶段训练流程,结合自监督MAE、半监督标签分类和CLIP风格的对比学习,以联合优化图像和视频模态。我们的架构基于增强的视觉Transformer骨干网络,结合高效的令牌重组(EViT)以提高计算效率。广泛的实验表明,Xray-Visual在多种基准测试中均取得最佳性能,包括ImageNet图像分类、Kinetics和HMDB51视频理解以及MSCOCO跨模态检索。该模型在领域偏移和对抗扰动方面表现出强大的鲁棒性。我们进一步证明,将大型语言模型作为文本编码器(LLM2CLIP)显著提高了检索性能和泛化能力,特别是在现实环境中。Xray-Visual为可扩展的多模态视觉模型建立了新的基准,同时保持了优越的准确性和计算效率。

英文摘要:

We present Xray-Visual, a unified vision model architecture for large-scale image and video understanding trained on industry-scale social media data. Our model leverages over 15 billion curated image-text pairs and 10 billion video-hashtag pairs from Facebook and Instagram, employing robust data curation pipelines that incorporate balancing and noise suppression strategies to maximize semantic diversity while minimizing label noise. We introduce a three-stage training pipeline that combines self-supervised MAE, semi-supervised hashtag classification, and CLIP-style contrastive learning to jointly optimize image and video modalities. Our architecture builds on a Vision Transformer backbone enhanced with efficient token reorganization (EViT) for improved computational efficiency. Extensive experiments demonstrate that Xray-Visual achieves state-of-the-art performance across diverse benchmarks, including ImageNet for image classification, Kinetics and HMDB51 for video understanding, and MSCOCO for cross-modal retrieval. The model exhibits strong robustness to domain shift and adversarial perturbations. We further demonstrate that integrating large language models as text encoders (LLM2CLIP) significantly enhances retrieval performance and generalization capabilities, particularly in real-world environments. Xray-Visual establishes new benchmarks for scalable, multimodal vision models, while maintaining superior accuracy and computational efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑