解决密集3D视觉问题的基于学习的方法通常在3D传感器数据上进行训练。分别使用的测量距离的原理提供了优点和缺点。由于缺乏多模态数据集,这些通常不会在文献中进行比较或讨论。无纹理区域对于来自运动运动恢复结构和立体声的结构是有问题的,反射材料对主动感测造成问题,并且半透明对象的距离对于用现有硬件测量是复杂的。对不准确或损坏的数据进行训练会导致模型偏差,并妨碍泛化能力。如果在评估过程中将传感器测量值视为地面实况,则这些影响仍然不会被注意到。本文研究了传感器误差对密集三维视觉任务深度估计和重建的影响。我们严格显示传感器特性对学习预测的重大影响,并注意到日常家庭环境中各种技术所产生的泛化问题。为了进行评估,我们引入了精心设计的数据集\footnote{dataset available at https://github.com/Junggy/HAMMER-dataset},其中包括来自商用传感器的测量结果,即D-ToF、I-ToF、无源/有源立体声和单目RGB+ P。我们的研究量化了相当大的传感器噪声影响,并为改善密集视觉估计和有针对性的数据融合铺平了道路。
英文摘要:
Learning-based methods to solve dense 3D vision problems typically train on 3D sensor data. The respectively used principle of measuring distances provides advantages and drawbacks. These are typically not compared nor discussed in the literature due to a lack of multi-modal datasets. Texture-less regions are problematic for structure from motion and stereo, reflective material poses issues for active sensing, and distances for translucent objects are intricate to measure with existing hardware. Training on inaccurate or corrupt data induces model bias and hampers generalisation capabilities. These effects remain unnoticed if the sensor measurement is considered as ground truth during the evaluation. This paper investigates the effect of sensor errors for the dense 3D vision tasks of depth estimation and reconstruction. We rigorously show the significant impact of sensor characteristics on the learned predictions and notice generalisation issues arising from various technologies in everyday household environments. For evaluation, we introduce a carefully designed dataset\footnote{dataset available at https://github.com/Junggy/HAMMER-dataset} comprising measurements from commodity sensors, namely D-ToF, I-ToF, passive/active stereo, and monocular RGB+P. Our study quantifies the considerable sensor noise impact and paves the way to improved dense vision estimates and targeted data fusion.
Interpretable Vision Transformers in Monocular Depth Estimation via SVDA
Dept. Electrical and Computer Engineering(电子与计算机工程系);Democritus University of Thrace(德米特里乌斯大学);Athena Research Center(雅典研究中心);University Campus at Kimmeria(基米里亚大学校园)
DensePercept-NCSSD: Vision Mamba towards Real-time Dense Visual Perception with Non-Causal State Space Duality
Machine Intelligence Group, Department of Computer Science & Information Systems, Birla Institute of Technology and Science, Pilani, Hyderabad Campus(机器智能组,计算机科学与信息系统系,比拉理工学院,帕利尼,海得拉巴校区)
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Nanjing University(南京大学);The University of Hong Kong(香港大学);Nanjing University of Science and Technology(南京理工大学);IIAI(智能信息处理与应用研究院);SenseTime Research(商汤科技研究院)
CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction
Nanyang Technological University(南洋理工大学);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);SenseTime Research and Tetras.AI(商汤科技研究院与Tetras.AI);Shanghai AI Laboratory(上海人工智能实验室)
针对 CLIP ViT 从全局图像迁移到局部区域时的域偏移问题,本文提出 CLIPSelf 自蒸馏方法,在无需区域-文本对的情况下增强区域-语言对齐,并在多项开放词汇密集预测基准上达到最优性能。
Adapting Dense Vision-Language Relationships for Multi-label Classification with Partial Label
State Key Laboratory of Virtual Reality Technology and Systems(虚拟现实技术与系统国家重点实验室);School of Computer Science and Engineering(计算机科学与工程学院);Beihang University(北京航空航天大学)
Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression
School of Information Science and Technology, Fudan University(复旦大学信息科学与技术学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Academy for Engineering and Technology, Fudan University(复旦大学工程技术学院);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学)
Dino-NestedUNet: Unlocking Foundation Vision Encoders for Pathology Tumor Bulk Segmentation via Dense Decoding
Department of Pathology, College of Medicine, The Ohio State University Wexner Medical Center, Columbus, OH, USA(病理学系,医学学院,俄亥俄州立大学韦克斯纳医学中心,哥伦布,OH,USA);Department of Biomedical Engineering, The Ohio State University, Columbus, OH, USA(生物医学工程系,俄亥俄州立大学,哥伦布,OH,USA)
Dense Vision Transformer Compression with Few Samples
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)
DenVisCoM: Dense Vision Correspondence Mamba for Efficient and Real-time Optical Flow and Stereo Estimation
Machine Intelligence Group, Birla Institute of Technology and Science, Pilani, Hyderabad Campus, India(机智人工智能组,比拉理工学院,比拉理工学院,海得拉巴校区,印度);Université Côte d’Azur, Inria, France(蔚蓝大学,法国国家信息与自动化研究所)
Revitalizing Dense Material Segmentation: Stabilized Vision Transformers and the Generalization Paradox
Bahcesehir University, Istanbul, Turkey(巴塞希尔大学,伊斯坦布尔,土耳其);Poder Bilişim Teknolojileri Sanayi ve Ticaret A.Ş., Istanbul, Turkey(Poder信息科技工业和贸易股份有限公司,伊斯坦布尔,土耳其);Galatasaray University, Istanbul, Turkey(加拉塔萨雷大学,伊斯坦布尔,土耳其)
Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistance
New York University Tandon School of Engineering(纽约大学坦登工程学院);New York University Abu Dhabi(纽约大学阿布扎比分校);NYU Grossman School of Medicine(纽约大学格罗斯曼医学院);NYU Langone Health(纽约大学兰贡医疗中心);Harvard University(哈佛大学)