3D scene modeling techniques serve as the bedrocks in the geospatial engineering and computer science, which drives many applications ranging from automated driving, terrain mapping, navigation, virtual, augmented, mixed, and extended reality (for gaming and movie industry etc.). This dissertation presents a fraction of contributions that advances 3D scene modeling to its state of the art, in the aspects of both appearance and geometry modeling. In contrast to the prevailing deep learning methods, as a core contribution, this thesis aims to develop algorithms that follow first principles, where sophisticated physic-based models are introduced alongside with simpler learning and inference tasks. The outcomes of these algorithms yield processes that can consume much larger volume of data for highly accurate reconstructing 3D scenes at a scale without losing methodological generality, which are not possible by contemporary complex-model based deep learning methods. Specifically, the dissertation introduces three novel methodologies that address the challenges of inferring appearance and geometry through physics-based modeling.
Overall, the research encapsulated in this dissertation marks a series of methodological triumphs in the processing of complex datasets. By navigating the confluence of deep learning, computational geometry, and photogrammetry, this work lays down a robust framework for future exploration and practical application in the rapidly evolving field of 3D scene reconstruction. The outcomes of these studies are evidenced through rigorous experiments and comparisons with existing state-of-the-art methods, demonstrating the efficacy and scalability of the proposed approaches.
SA-Person: Text-Based Person Retrieval with Scene-aware Re-ranking
School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人中心);Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统(MAIS));Wuhan University(武汉大学);University of Chinese Academy of Sciences(中国科学院大学)
Beihang University(北京航空航天大学);Shandong University(山东大学);Beijing Technology and Business University(北京工商大学);Jingdezhen Research Institute of Beihang University(北京航空航天大学景德镇研究院);Shandong University of Science and Technology(山东科技大学)
University of Pennsylvania(宾夕法尼亚大学);Carnegie Mellon University(卡内基梅隆大学);Motional(莫申公司);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);Georgia Tech(佐治亚理工学院)
School of Computer Science, Australian Artificial Intelligence Institute (AAII), University of Technology Sydney(悉尼科技大学计算机科学学院、澳大利亚人工智能研究所);School of Automobile, Chang’an University(长安大学汽车学院)
Direct and Adaptable Mesh-Gaussian Scene Reconstruction from Multi-View Images
School of Computer Science, Australian Artificial Intelligence Institute (AAII)(计算机科学学院,澳大利亚人工智能研究所);University of Technology Sydney(悉尼技术大学);School of Automobile(汽车学院);Chang’an University(长安大学)
3D-USE: From Image-Level to Scene-Level Underwater Enhancement
College of Computer Science, Nankai University(南开大学计算机学院);NKIARI(南开大学人工智能研究院(深圳));School of Automation, Southeast University(东南大学自动化学院);Advanced Ocean Institute of Southeast University, Nantong(东南大学南通先进海洋研究院)
Injecting Frame-Event Complementary Fusion into Diffusion for Optical Flow in Challenging Scenes
National Key Lab of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(多谱段信息智能处理国家级实验室,人工智能与自动化学院,华中科技大学);School of Computing, National University of Singapore(计算机学院,新加坡国立大学)
Dense Scene Multiple Object Tracking with Box-Plane Matching
Tencent Youtu Lab(腾讯优图实验室)
多目标跟踪(MOT)是计算机视觉中的一项重要任务。由于遮挡问题,MOT仍然具有挑战性,特别是在密集场景中。在检测跟踪的框架下,我们提出了盒平面匹配(BPM)方法来提高密集场景下的MOT性能。首先,我们设计了分层聚集判别模型(LADM)来过滤噪声检测。然后,为了正确地关联剩余检测,我们引入全局注意力特征模型(GAFM)来提取外观特征,并使用它来计算历史轨迹和当前检测之间的外观相似性。最后,我们提出了盒平面匹配策略,以实现数据关联,根据运动相似性和外观相似性之间的轨迹和检测。凭借这三个模块的有效性,我们的团队在ACM MM Grand Challenge Hieve 2020中获得了第一名。
Monocular Occupancy Prediction for Scalable Indoor Scenes
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);Centre for Artificial Intelligence and Robotics (HKISI_CAS)(人工智能与机器人中心)
Underwater360: Reconstructing Underwater Scenes from Panoramic Images with Omnidirectional Gaussian Splatting
School of Software, Dalian University of Technology(大连理工大学软件学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)