UniHCP: A Unified Model for Human-Centric Perceptions
The University of Sydney（悉尼大学）；Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
Human-centric perceptions (e.g., pose estimation, human parsing, pedestrian detection, person re-identification, etc.) play a key role in industrial applications of visual models. While specific human-centric tasks have their own relevant semantic aspect to focus on, they also share the same underlying semantic structure of the human body. However, few works have attempted to exploit such homogeneity and design a general-propose model for human-centric tasks. In this work, we revisit a broad range of human-centric tasks and unify them in a minimalist manner. We propose UniHCP, a Unified Model for Human-Centric Perceptions, which unifies a wide range of human-centric tasks in a simplified end-to-end manner with the plain vision transformer architecture. With large-scale joint training on 33 human-centric datasets, UniHCP can outperform strong baselines on several in-domain and downstream tasks by direct evaluation. When adapted to a specific task, UniHCP achieves new SOTAs on a wide range of human-centric tasks, e.g., 69.8 mIoU on CIHP for human parsing, 86.18 mA on PA-100K for attribute prediction, 90.3 mAP on Market1501 for ReID, and 85.8 JI on CrowdHuman for pedestrian detection, performing better than specialized models tailored for each task.
HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception
Zhejiang University（浙江大学）；Baidu VIS（百度视觉技术部）；University of Science and Technology Beijing（北京科技大学）；South China University of Technology（华南理工大学）；Shanghai Jiao Tong University（上海交通大学）；Jilin University（吉林大学）
In recent years, vision-centric perception has flourished in various autonomous driving tasks, including 3D detection, semantic map construction, motion forecasting, and depth estimation. Nevertheless, the latency of vision-centric approaches is too high for practical deployment (e.g., most camera-based 3D detectors have a runtime greater than 300ms). To bridge the gap between ideal research and real-world applications, it is necessary to quantify the trade-off between performance and efficiency. Traditionally, autonomous-driving perception benchmarks perform the offline evaluation, neglecting…
ShanghaiTech University（上海科技大学）；Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）；Shanghai AI Lab（上海人工智能实验室）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Kentucky（肯塔基大学）；University of Maryland, College Park（马里兰大学帕克分校）
Shanghai Jiao Tong University（上海交通大学）；University of Southern California（南加州大学）；Shanghai AI Laboratory（上海人工智能实验室）
Vision-centric joint perception and prediction (PnP) has become an emerging trend in autonomous driving research. It predicts the future states of the traffic participants in the surrounding environment from raw RGB images. However, it is still a critical challenge to synchronize features obtained at multiple camera views and timestamps due to inevitable geometric distortions and further exploit those spatial-temporal features. To address this issue, we propose a temporal bird's-eye-view pyramid transformer (TBP-Former) for vision-centric PnP, which includes two novel designs. First, a pose-sy…
The University of Sydney（悉尼大学）；SenseTime Research（商汤科技研究院）；Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
Human-centric perceptions include a variety of vision tasks, which have widespread industrial applications, including surveillance, autonomous driving, and the metaverse. It is desirable to have a general pretrain model for versatile human-centric downstream tasks. This paper forges ahead along this path from the aspects of both benchmark and pretraining methods. Specifically, we propose a \textbf{HumanBench} based on existing datasets to comprehensively evaluate on the common ground the generalization abilities of different pretraining methods on 19 datasets from 6 diverse downstream tasks, i…
Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Human-centric perception plays a vital role in vision and graphics. But their data annotations are prohibitively expensive. Therefore, it is desirable to have a versatile pre-train model that serves as a foundation for data-efficient downstream tasks transfer. To this end, we propose the Human-Centric Multi-Modal Contrastive Learning framework HCMoCo that leverages the multi-modal nature of human data (e.g. RGB, depth, 2D keypoints) for effective representation learning. The objective comes with two main challenges: dense pre-train for multi-modality data, efficient usage of sparse human prior…
In this paper, we present BEVerse, a unified framework for 3D perception and prediction based on multi-camera systems. Unlike existing studies focusing on the improvement of single-task approaches, BEVerse features in producing spatio-temporal Birds-Eye-View (BEV) representations from multi-camera videos and jointly reasoning about multiple tasks for vision-centric autonomous driving. Specifically, BEVerse first performs shared feature extraction and lifting to generate 4D BEV representations from multi-timestamp and multi-view images. After the ego-motion alignment, the spatio-temporal encode…
Unified Vision-Centric Pedestrian Crossing Action Prediction via Adaptive Patch Projection and Proactive Spatial Rectification
School of Electronics and Control Engineering, Chang’an University（长安大学电子与控制工程学院）；School of Astronautics, Northwestern Polytechnical University（西北工业大学航天学院）
NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Variants of accuracy and precision are the gold-standard by which the computer vision community measures progress of perception algorithms. One reason for the ubiquity of these metrics is that they are largely task-agnostic; we in general seek to detect zero false negatives or positives. The downside of these metrics is that, at worst, they penalize all incorrect detections equally without conditioning on the task or scene, and at best, heuristics need to be chosen to ensure that different mistakes count differently. In this paper, we propose a principled metric for 3D object detection specifi…
Human-Centric Perception for Child Sexual Abuse Imagery
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学）；Federal Police of Brazil（巴西联邦警察局）；Universidade Estadual de Campinas (UNICAMP)（坎皮纳斯州立大学）；University of Sheffield（谢菲尔德大学）
Scale-Aware Pre-Training for Human-Centric Visual Perception: Enabling Lightweight and Generalizable Models
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（深圳先进研究院，电子科学与技术大学）；Center for Future Media, University of Electronic Science and Technology of China（未来媒体中心，电子科学与技术大学）；TongJi University（同济大学）
Improving Vision-language Models with Perception-centric Process Reward Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院）；Bytedance（字节跳动）；University of California, San Diego（加州大学圣地亚哥分校）；The Hong Kong University of Science and Technology（香港科技大学）
EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
X-Humanoid；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；University of Manchester（曼彻斯特大学）；Monash University（墨尔本大学）；Celonis AI；University of New South Wales（新南威尔士大学）
Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models
Fudan University（复旦大学）；Meituan（美团）；Shanghai Key Lab of Intell. Info. Processing（上海智能信息处理重点实验室）；Shanghai Collaborative Innovation Center on Intelligent Visual Computing（上海智能视觉计算协同创新中心）
Vehicle-centric Perception via Multimodal Structured Pre-training
Information Materials and Intelligent Sensing Laboratory of Anhui Province（安徽省信息材料与智能感知实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；the School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
University of Michigan（密歇根大学）；Robotics Institute（机器人研究所）
Visual perception tasks often require vast amounts of labelled data, including 3D poses and image space segmentation masks. The process of creating such training data sets can prove difficult or time-intensive to scale up to efficacy for general use. Consider the task of pose estimation for rigid objects. Deep neural network based approaches have shown good performance when trained on large, public datasets. However, adapting these networks for other novel objects, or fine-tuning existing models for different environments, requires significant time investment to generate newly labelled instanc…
ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大模型与智能治理重点实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Jiaotong University（北京交通大学）；State Key Laboratory of Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；AresoX
CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution
Nanjing University（南京大学）；LLM Team, Shopee Pte. Ltd.（Shopee 联邦学习团队）；East China Normal University（华东师范大学）；Nanjing University of Science and Technology（南京理工大学）；Central South University（中南大学）
Robotic manipulation in complex open-world scenarios requires both reliable physical manipulation skills and effective and generalizable perception. In this paper, we propose a method where general purpose pretrained visual models serve as an object-centric prior for the perception system of a learned policy. We devise an object-level attentional mechanism that can be used to determine relevant objects from a few trajectories or demonstrations, and then immediately incorporate those objects into a learned policy. A task-independent meta-attention locates possible objects in the scene, and a ta…
WaterVideoQA: ASV-Centric Perception and Rule-Compliant Reasoning via Multi-Modal Agents
Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)（香港理工大学（广州）人工智能研究所）；Hubei Key Laboratory of Inland Shipping Technology (Wuhan University of Technology)（湖北内河航运技术重点实验室（武汉理工大学））；School of Navigation, Wuhan University of Technology（武汉理工大学航海学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学先进科技学院）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；School of Information Engineering, Yancheng Institute of Technology（盐城职业技术学院信息工程学院）；School of Engineering, Stanford University（斯坦福大学工程学院）；Centre for AI and Data Science Innovation and the School of Science and Engineering, James Cook University（詹姆斯库克大学人工智能与数据科学创新中心及科学与工程学院）
EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University（北京未来区块链与隐私计算先进创新中心，人工智能学院，北京航空航天大学）；Hangzhou International Innovation Institute, Beihang University（杭州国际创新研究院，北京航空航天大学）；Paradigm Inc.（4Paradigm公司）
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China（中国科学院智能信息处理重点实验室（中国科学院）、计算技术研究所、中国）
OSRAM GmbH（欧司朗股份有限公司）；Istituto Italiano di Tecnologia (IIT)（意大利理工学院）；University of Verona (UNIVR)（维罗纳大学）
Lighting design in indoor environments is of primary importance for at least two reasons: 1) people should perceive an adequate light; 2) an effective lighting design means consistent energy saving. We present the Invisible Light Switch (ILS) to address both aspects. ILS dynamically adjusts the room illumination level to save energy while maintaining constant the light level perception of the users. So the energy saving is invisible to them. Our proposed ILS leverages a radiosity model to estimate the light level which is perceived by a person within an indoor environment, taking into account …
