Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Northwestern Polytechnical University（西北工业大学）；Zhejiang University（浙江大学）
Haoyi Zhu, Yating Wang, Di Huang, Weicai Ye, Wanli Ouyang, Tong He
In robot learning, the observation space is crucial due to the distinct characteristics of different modalities, which can potentially become a bottleneck alongside policy design. In this study, we explore the influence of various observation spaces on robot learning, focusing on three predominant modalities: RGB, RGB-D, and point cloud. We introduce OBSBench, a benchmark comprising two simulators and 125 tasks, along with standardized pipelines for various encoders and policy baselines. Extensive experiments on diverse contact-rich manipulation tasks reveal a notable trend: point cloud-based methods, even those with the simplest designs, frequently outperform their RGB and RGB-D counterparts. This trend persists in both scenarios: training from scratch and utilizing pre-training. Furthermore, our findings demonstrate that point cloud observations often yield better policy performance and significantly stronger generalization capabilities across various geometric and visual conditions. These outcomes suggest that the 3D point cloud is a valuable observation modality for intricate robotic tasks. We also suggest that incorporating both appearance and coordinate information can enhance the performance of point cloud methods. We hope our work provides valuable insights and guidance for designing more generalizable and robust robotic models. Codes are available at https://github.com/HaoyiZhu/PointCloudMatters.
Rethinking Attention Module Design for Point Cloud Analysis
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB（弗劳恩霍夫光学技术、系统技术和图像利用研究所IOSB）
Rethinking Metrics and Diffusion Architecture for 3D Point Cloud Generation
Mines Paris, Université PSL（Mines Paris，Université PSL）；Centre des Matériaux (MAT), UMR7633 CNRS（Centre des Matériaux（MAT），UMR7633 CNRS）；Centre de Morphologie Mathématique (CMM)（Centre de Morphologie Mathématique（CMM））；Centre de Mise en Forme des Matériaux (CEMEF), UMR7635 CNRS（Centre de Mise en Forme des Matériaux（CEMEF），UMR7635 CNRS）
本文针对三维点云生成评估指标鲁棒性不足的问题，提出样本对齐、DCD及新指标SNC，并设计Diffusion Point Transformer架构，在ShapeNet上取得领先性能。
SPAC-Net: Rethinking Point Cloud Completion with Structural Prior
School of Digital Media Technology, Hangzhou Dianzi University（杭州电子科技大学数字媒体技术学院）；School of Informatics, Xiamen University（厦门大学信息学院）；School of Information, Renmin University of China（中国人民大学信息学院）
Rethinking Transferable Adversarial Attacks on Point Clouds from a Compact Subspace Perspective
Cyberspace Institute of Advanced Technology, Guangzhou University（广州大学网络安全研究院）；School of Computer Science and Cyber Engineering, Guangzhou University（广州大学计算机科学与网络工程学院）；Department of Automation, University of Science and Technology of China（中国科学技术大学自动化系）；Institute for Math & AI, Wuhan University（武汉大学数学与人工智能研究院）；School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能、光学与电子学院（iOPEN））
Rethinking the Detail-Preserved Completion of Complex Tubular Structures based on Point Cloud: a Dataset and a Benchmark
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（新一代人工智能技术及其交叉应用重点实验室（东南大学））；Jiangsu Province Joint International Research Laboratory of Medical Information Processing（江苏省医学信息处理联合国际研究实验室）；The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院）
Federated Imitation Learning: A Novel Framework for Cloud Robotic Systems with Heterogeneous Sensor Data
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong University of Science and Technology（香港科技大学）；University of Macau（澳门大学）
Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework
Nanjing University of Science and Technology（南京理工大学）；Shanghai Jiao Tong University（上海交通大学）；Hangzhou City University（杭州城市学院）；East China Normal University（华东师范大学）
Federated Imitation Learning: A Privacy Considered Imitation Learning Framework for Cloud Robotic Systems with Heterogeneous Sensor Data
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong University of Science and Technology（香港科技大学）
Peer-Assisted Robotic Learning: A Data-Driven Collaborative Learning Approach for Cloud Robotic Systems
Chinese Academy of Sciences（中国科学院）；Shenzhen Institutes of Advanced Technology (SIAT)（中国科学院深圳先进技术研究院）；Joint Laboratory of Artificial Intelligence and Robotics of SIAT and University of Macau（中国科学院深圳先进技术研究院与澳门大学人工智能与机器人联合实验室）；University of Macau（澳门大学）；University of Chinese Academy of Sciences（中国科学院大学）
GaussianPU: A Hybrid 2D-3D Upsampling Framework for Enhancing Color Point Clouds via 3D Gaussian Splatting
Sun Yat-Sen University（中山大学）；Nanjing University（南京大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Peking University（北京大学）；Xi’an Jiaotong University（西安交通大学）
University of California, Berkeley（加利福尼亚大学伯克利分校）；Robert Bosch Research and Technology Center North America（罗伯特·博世北美研究与技术中心）；Carnegie Mellon University（卡内基梅隆大学）
Rethinking the Multi-view Stereo from the Perspective of Rendering-based Augmentation
Fudan University（复旦大学）
GigaMVS因其大规模、复杂遮挡和千兆像素图像而对现有的多视图立体视觉（MVS）算法提出了几项挑战。为了解决这些问题，我们首先应用了一种最先进的基于学习的MVS方法--MVS Former，以克服传统PatchMatch方法所遇到的无纹理和反射区域等棘手场景，但在一些大场景的重建中失败了。此外，传统的PatchMatch算法，如ACMMP，OpenMVS和RealityCapture被用来进一步提高大场景的完整性。此外，为了统一深度学习方法和传统PatchMatch的优点，我们建议渲染深度和彩色图像，以进一步微调MVSFormer模型。值得注意的是，我们发现，MVS方法可以通过渲染图像产生更好的预测，由于重合照明，我们认为这是重要的MVS社区。因此，MVSFormer能够推广到大规模场景，并补充解决了无纹理重建问题。最后，我们组装了上述所有点云\textit{except ones from RealityCapture}，并在竞争对手GigaReconstruction上排名前1。
