Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding
Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Sensetime
Image pyramids are widely adopted in top-performing methods to obtain multi-scale features for precise visual perception and understanding. However, current image pyramids use the same large-scale model to process multiple resolutions of images, leading to significant computational cost. To address this challenge, we propose a novel network architecture, called Parameter-Inverted Image Pyramid Networks (PIIP). Specifically, PIIP uses pretrained models (ViTs or CNNs) as branches to process multi-scale images, where images of higher resolutions are processed by smaller network branches to balance computational cost and performance. To integrate information from different spatial scales, we further propose a novel cross-branch feature interaction mechanism. To validate PIIP, we apply it to various perception models and a representative multimodal large language model called LLaVA, and conduct extensive experiments on various tasks such as object detection, segmentation, image classification and multimodal understanding. PIIP achieves superior performance compared to single-branch and existing multi-resolution approaches with lower computational cost. When applied to InternViT-6B, a large-scale vision foundation model, PIIP can improve its performance by 1%-2% on detection and segmentation with only 40%-60% of the original computation, finally achieving 60.0 box AP on MS COCO and 59.7 mIoU on ADE20K. For multimodal understanding, our PIIP-LLaVA achieves 73.0% accuracy on TextVQA and 74.5% on MMBench with only 2.8M training data. Our code is released at https://github.com/OpenGVLab/PIIP.
Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);SenseTime Research(商汤科技研究院);Shanghai AI Laboratory(上海人工智能实验室)
Deep Sensor Fusion with Pyramid Fusion Networks for 3D Semantic Segmentation
Friedrich-Alexander University (FAU) Erlangen-Nürnberg(埃尔朗根-纽伦堡弗里德里希-亚历山大大学);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Technische Hochschule Ingolstadt(因戈尔施塔特应用技术大学);Research Institute AImotion Bavaria(巴伐利亚AImotion研究所)
CLFTv2: Efficient Camera-LiDAR Fusion for Semantic Segmentation via Hierarchical Feature Pyramids
Tallinn University of Technology(塔林理工大学);FinEst Centre for Smart Cities, Tallinn University of Technology(塔林理工大学芬埃斯特智慧城市中心);Universitas Mercatorum(梅尔卡托鲁姆大学)
SPFFNet: Strip Perception and Feature Fusion Spatial Pyramid Pooling for Fabric Defect Detection
Waterford Institute South East Technological University Waterford, Ireland(沃特福德研究所 南东技术大学 爱尔兰沃特福德);Faculty of Innovation Engineering Macau University of Science(创新工程学院 澳门科学技术大学)
Multi-Modal Semantic Segmentation of Electrolyzer Components for Sustainable Hydrogen Technologies: A Dual-Branch Deep Learning Approach
Applied Artificial Intelligence and INtelligent Systems (AAIINS) Laboratory(应用人工智能与智能系统(AAIINS)实验室);University of Scholars(学者大学);Murdoch University(莫道克大学);Charles Darwin University(查尔斯达尔文大学)
LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer
Tsinghua University(清华大学);National University of Singapore(新加坡国立大学);Alibaba Group(阿里巴巴集团);University of Chinese Academy of Sciences(中国科学院大学);The Chinese University of Hong Kong(香港中文大学)
LeafInst - Unified Instance Segmentation Network for Fine-Grained Forestry Leaf Phenotype Analysis: A New UAV based Benchmark
organization= College of Information Science;Technology \& Artificial Intelligence, Nanjing Forestry University , city= Nanjing , postcode= 210037 , state= Jiangsu , country= China;organization= Department of Geography, College of Natural Resources;organization= Co-Innovation Center for Sustainable Forestry in Southern China, Nanjing Forestry University , city= Nanjing , postcode= 210037 , state= Jiangsu , country= China;organization= School of Architecture \& Urban Planning, Shenzhen University , city= Shenzhen , postcode= 518060 , state= Guangdong , country= China;organization= School of Software, Hefei University of Technology , city= Hefei , postcode= 230601 , state= Anhui , country= China
PVAFN: Point-Voxel Attention Fusion Network with Multi-Pooling Enhancing for 3D Object Detection
College of Computer Science and Technology, Taiyuan University of Technology(太原理工大学计算机科学与技术学院);Key Laboratory of Machine Perception, Peking University(北京大学机器感知实验室);University of Pisa(比萨大学);University of Trento(特伦托大学)
MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapters
School of Information and Electronics, Beijing Institute of Technology(信息与电子学院,北京理工大学);Beijing Institute of Technology Chongqing Innovation Center(北京理工大学重庆创新中心);National Key Laboratory for Space-Born Intelligent Information Processing(空间智能信息处理国家级重点实验室);School of Automation, Beijing Institute of Technology(自动化学院,北京理工大学)
Rethinking Medical Landmark Localization with Prototype Learning-based Progressive Offset Correction
Shenzhen University(深圳大学);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人中心);Boston Children’s Hospital(波士顿儿童医院);Harvard Medical School(哈佛医学院);School of Artificial Intelligence, Shenzhen University(深圳大学人工智能学院);School of Biomedical Engineering and Informatics, Nanjing Medical University(南京医科大学生物医学工程与信息学院);National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University(深圳大学大数据系统计算技术国家工程实验室)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大模型与智能治理重点实验室);Beihang University(北京航空航天大学);Beijing University of Posts and Telecommunications(北京邮电大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);AresoX
ScanFormer: Referring Expression Comprehension by Iteratively Scanning
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Software Technology, Zhejiang University(浙江大学软件学院);Zhejiang-Singapore Innovation and Al Joint Research Lab(浙新创新与人工智能联合研究实验室)
When Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perception
National University of Singapore(新加坡国立大学);Wuhan University(武汉大学);ByteDance(字节跳动);Nankai University(南开大学);JD Research, JD.com, Inc.(京东研究院(京东有限公司));Zhejiang University(浙江大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);The University of Hong Kong(香港大学)
4K-Resolution Photo Exposure Correction at 125 FPS with ~8K Parameters
Nankai University(南开大学);Tianjin University(天津大学);Lappeenranta-Lahti University of Technology(拉彭兰塔-拉赫蒂理工大学);Guangdong Provincial Key Laboratory of Big Data Computing, CUHK (Shenzhen)(广东省大数据计算重点实验室,香港中文大学(深圳))
AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation
School of Information and Communication Engineering, Dalian University of Technology(大连理工大学信息与通信工程学院);School of Innovation and Entrepreneurship, Dalian University of Technology(大连理工大学创新创业学院);School of Artificial Intelligence, Dalian University of Technology(大连理工大学人工智能学院)