Exploiting multiple modalities for semantic scene parsing has been shown to improve accuracy over the singlemodality scenario. However multimodal datasets often suffer from problems such as data misalignment and label inconsistencies, where the existing methods assume that corresponding regions in two modalities must have identical labels. We propose to address this issue, by formulating multimodal semantic labeling as inference in a CRF and introducing latent nodes to explicitly model inconsistencies between two modalities. These latent nodes allow us not only to leverage information from both domains to improve their labeling, but also to cut the edges between inconsistent regions. We propose to learn intradomain and inter-domain potential functions from training data to avoid hand-tuning of the model parameters. We evaluate our approach on two publicly available datasets containing 2D and 3D data. Thanks to our latent nodes and our learning strategy, our method outperforms the state-of-the-art in both cases. Moreover, in order to highlight the benefits of the geometric information and the potential of our method in simultaneous 2D/3D semantic and geometric inference, we performed simultaneous inference of semantic and geometric classes both in 2D and 3D that led to satisfactory improvements of the labeling results in both datasets.
Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing
Zhengzhou University(郑州大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peng Cheng Laboratory(鹏城实验室)
OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models
School of Information Science and Engineering, East China University of Science and Technology(东华大学信息科学与工程学院);School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件工程学院);School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院);Alibaba Group(阿里巴巴集团)
Predicting Scene Parsing and Motion Dynamics in the Future
NUS Graduate School for Integrative Science and Engineering (NGS), NUS(新加坡国立大学综合科学与工程研究生院);Department of ECE, NUS(新加坡国立大学电子与计算机工程系);Adobe Research(Adobe 研究院);Tencent AI Lab(腾讯人工智能实验室);Qihoo 360 AI Institute(奇虎360人工智能研究院)
PereStruct: Multimodal Semantic Assembly for Robust Historical Document Parsing
IGIC RAS(俄罗斯科学院信息传输问题研究所);Yandex Cloud;National University of Science and Technology MISIS(莫斯科国立钢铁合金学院);Nekrasov Central Universal Scientific Library(涅克拉索夫中央综合科学图书馆)
Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Tencent(腾讯);Nankai University(南开大学);University of Chinese Academy of Sciences(中国科学院大学);Nanjing University of Science and Technology(南京理工大学)
Multimodal Medical Image Parsing with Task-Specific Low-Rank Adapters
Gettysburg College(盖蒂斯堡学院);Instituto Politécnico Nacional (IPN)(国立理工学院);Michigan State University(密歇根州立大学);HSE University(高等经济学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks
College of Electronic and Information Engineering, Tongji University(同济大学电子信息学院);Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学智能自主系统研究所);College of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);Department of Vehicle Control System and Software Development, NIO(蔚来汽车车辆控制系统与软件开发部门);Department of Automotive Engineering, Jilin University(吉林大学汽车工程学院);Key Laboratory of Embedded System and Service Computing (Ministry of Education), Tongji University(同济大学嵌入式系统与服务计算重点实验室);College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology(上海智能科学与技术研究院电子信息学院);Shanghai Key Laboratory of Intelligent Autonomous Systems(上海智能自主系统重点实验室)
A fine-grained attention and geometric correspondence model for musculoskeletal risk classification in athletes using multimodal visual and skeletal features
Department of Computer Science and Engineering, United International University(计算机科学与工程系,国际联合大学);Department of Data Science and Artificial Intelligence, Monash University(数据科学与人工智能系,墨尔本大学);Faculty of Science and Technology, Charles Darwin University(科学与技术学院,查尔斯达尔文大学);Applied Artificial Intelligence and Intelligent Systems (AAIINS) Laboratory, Dhaka(应用人工智能与智能系统实验室,达卡)
GPT-4o: Visual perception performance of multimodal large language models in piglet activity understanding
China Agricultural University(中国农业大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);AniEye Technology(安尼眼科技);State Key Laboratory of Animal Nutrition and Feeding(动物营养与饲料国家重点实验室)
Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model
University of Washington(华盛顿大学);Tsinghua University(清华大学);Tencent(腾讯);Google Deepmind(谷歌DeepMind);Allen Institute for AI(艾伦人工智能研究所);Cornell University(康奈尔大学)
ReCon: Enhancing True Correspondence Discrimination through Relation Consistency for Robust Noisy Correspondence Learning
Huaqiao University(华侨大学);Hong Kong Baptist University(香港浸会大学);University of Electronic Science and Technology of China(电子科技大学);Xidian University(西安电子科技大学)
FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrieval
School of Software, Shandong University(山东大学软件学院);School of Data Science, City University of Hong Kong(香港城市大学数据科学学院);School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院)