Multi-modal knowledge graphs have emerged as a powerful approach for information representation, combining data from different modalities such as text, images, and videos. While several such graphs have been constructed and have played important roles in applications like visual question answering and recommendation systems, challenges persist in their development. These include the scarcity of high-quality Chinese knowledge graphs and limited domain coverage in existing multi-modal knowledge graphs. This paper introduces MMPKUBase, a robust and extensive Chinese multi-modal knowledge graph that covers diverse domains, including birds, mammals, ferns, and more, comprising over 50,000 entities and over 1 million filtered images. To ensure data quality, we employ Prototypical Contrastive Learning and the Isolation Forest algorithm to refine the image data. Additionally, we have developed a user-friendly platform to facilitate image attribute exploration.
VATEX Captioning Challenge 2019: Multi-modal Information Fusion and Multi-stage Training Strategy for Video Captioning
CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）；National Laboratory of Pattern Recognition, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；PeopleAI, Inc., China（中国PeopleAI公司）
CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language
School of Informatics, Xiamen University, China（厦门大学信息学院）；Key Lab of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian-Taiwan (XMU), Ministry of Culture and Tourism, China（福建省-台湾非物质文化遗产数字化保护与智能处理重点实验室（XMU），文化和旅游部，中国）；National Language Resources Monitoring and Research Center for Education and Teaching Media, Xiamen University, China（教育与教学媒体语言资源监测与研究中心，厦门大学，中国）
InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
Alibaba Group（阿里巴巴集团）；Peking University（北京大学）
用于学习高级多模态表示的多模态预训练是迈向深度学习和人工智能的又一步。在这项工作中，我们提出了一个新的模型，即InterBERT（BERT for Interaction），这是我们的多模态预训练方法M6（MultiModality-to-MultiModality Multitask Mega-transformer）系列的第一个模型。该模型具有很强的建模能力，不同模态的信息流之间的相互作用。单流交互模块能够有效地处理多模态的信息，而上层的双流交互模块保持了各模态的独立性，避免了单模态任务的性能下降。我们使用三个预训练任务对模型进行预训练，包括掩蔽段建模（MSM）、掩蔽区域建模（MRM）和图像-文本匹配（ITM）;并在一系列视觉和语言下游任务上对模型进行微调。实验结果表明，InterBERT的性能优于一系列强基线，包括最新的多模态预训练方法，分析表明MSM和MRM是有效的预训练方法，我们的方法在单模态任务中可以达到与BERT相当的性能。此外，我们提出了一个大规模的数据集，用于中文多模态预训练，我们开发了中文InterBERT，这是第一个中文多模态预训练模型。我们在我们提出的来自中国最大的电子商务平台手机淘宝的310万图像-文本对数据集上预训练了中文InterBERT。我们微调的模型，基于文本的图像检索，最近，我们部署了基于主题的推荐模型在线。
Cross-Modal Mutual Learning for Cued Speech Recognition
Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）深圳大数据研究院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China（多媒体可信感知与高效计算教育部重点实验室）；Xiamen University（厦门大学）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）
ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Freedom AI；National Health Data Institute, Shenzhen（深圳国家健康数据研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Guangdong University of Technology（广东工业大学）
No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks
Yunnan University（云南大学）；The University of Manchester（曼彻斯特大学）；Wuhan University（武汉大学）；University of Florida（佛罗里达大学）；The Chinese University of Hong Kong（香港中文大学）；The Fin AI
LongHisDoc: A Comprehensive Benchmark for Chinese Long Historical Document Understanding
School of Electronic and Information Engineering, South China University of Technology（华南理工大学电子与信息工程学院）；College of Literature and Journalism, Sichuan University（四川大学文学与新闻学院）
Foundation Model-based Evaluation of Neuropsychiatric Disorders: A Lifespan-Inclusive, Multi-Modal, and Multi-Lingual Study
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；Shenzhen Research Institute, Hunan University（湖南大学深圳研究院）；Ministry of Education Key Laboratory of Fusion Computing of Supercomputing and Artificial Intelligence, Hunan University（湖南大学教育部长春计算机与人工智能融合计算重点实验室）
Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies
Shanghai Institute of Infectious Disease and Biosecurity, Fudan University（复旦大学附属上海市感染与生物安全研究院）；Shanghai Health Development Research Center(Shanghai Medical Information Center)（上海卫生发展研究中心（上海市医学信息中心））；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASR
National Institute of Information and Communications Technology（信息通信研究机构）；Research Center for Information Technology Innovation（资讯科技创新研究中心）；Academia Sinica（中央研究院）
MMBench: Is Your Multi-modal Model an All-around Player?
Shanghai AI Laboratory（上海人工智能实验室）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；The Chinese University of Hong Kong（香港中文大学）
MANBench: Is Your Multimodal Model Smarter than Human?
School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通讯学院，华中科技大学）；School of Basic Medicine, Huazhong University of Science and Technology（基础医学院，华中科技大学）
