AI Video Assistant emerges as a new paradigm for Real-time Communication (RTC), where one peer is a Multimodal Large Language Model (MLLM) deployed in the cloud. This makes interaction between humans and AI more intuitive, akin to chatting with a real person. However, a fundamental mismatch exists between current RTC frameworks and AI Video Assistants, stemming from the drastic shift in Quality of Experience (QoE) and more challenging networks. Measurements on our production prototype also confirm that current RTC fails, causing latency spikes and accuracy drops.
To address these challenges, we propose Artic, an AI-oriented RTC framework for MLLM Video Assistants, exploring the shift from "humans watching video" to "AI understanding video." Specifically, Artic proposes: (1) Response Capability-aware Adaptive Bitrate, which utilizes MLLM accuracy saturation to proactively cap bitrate, reserving bandwidth headroom to absorb future fluctuations for latency reduction; (2) Zero-overhead Context-aware Streaming, which allocates limited bitrate to regions most important for the response, maintaining accuracy even under ultra-low bitrates; and (3) Degraded Video Understanding Benchmark, the first benchmark evaluating how RTC-induced video degradation affects MLLM accuracy. Prototype experiments using real-world uplink traces show that compared with existing methods, Artic significantly improves accuracy by 15.12% and reduces latency by 135.31 ms. We will release the benchmark and codes at https://github.com/pku-netvideo/DeViBench.
RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；School of Computing, Clemson University（克莱姆森大学计算机学院）；Washington University School of Medicine in St. Louis（圣路易斯华盛顿大学医学院）；The Keck School of Medicine of the University of Southern California（南加州大学凯克医学院）；Mayo Clinic（梅奥诊所）；Banner Alzheimer’s Institute（班纳阿尔茨海默病研究所）
ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；School of Engineering, Westlake University（西湖大学工程学院）；University of Minnesota – Twin Cities（明尼苏达大学双城分校）；University of Toronto（多伦多大学）
AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison
Department of Computer Science and Engineering, Southern University of Science and Technology（计算机科学与工程系，南方科技大学）；Macau University of Science and Technology（澳门科技大学）；Tencent YouTu Lab（腾讯YouTu实验室）；Nanjing University（南京大学）；Shanghai Jiao Tong University（上海交通大学）；Centre for Frontier AI Research, A*STAR（前沿人工智能研究中心，A*STAR）
CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language
School of Informatics, Xiamen University, China（厦门大学信息学院）；Key Lab of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian-Taiwan (XMU), Ministry of Culture and Tourism, China（福建省-台湾非物质文化遗产数字化保护与智能处理重点实验室（XMU），文化和旅游部，中国）；National Language Resources Monitoring and Research Center for Education and Teaching Media, Xiamen University, China（教育与教学媒体语言资源监测与研究中心，厦门大学，中国）
Multimodal Large Language Models for Bioimage Analysis
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；Leibniz-Institut für Analytische Wissenschaften – ISAS – e.V.（莱布尼茨分析科学研究所）
Event-Aware Instructed Assistant for Referring Video Segmentation
Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与技术学院大数据研究所）；Institute of Trustworthy Embodied AI, Fudan University（复旦大学可信具身智能研究所）；Shanghai University of Finance and Economics（上海财经大学）
EgoBlind: Towards Egocentric Visual Assistance for the Blind
National University of Singapore（新加坡国立大学）；Communication University of China（中国传媒大学）；University of Science and Technology of China（中国科学技术大学）；Hefei University of Technoloy（合肥工业大学）
LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC学院）；University of Washington（华盛顿大学）；Zhejiang University School of Medicine Sir Run Run Shaw Hospital（浙江大学医学院附属邵逸夫医院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness
Department of Computer Science（计算机科学系）；Engineering, Ohio State University, Ohio, US（工程系，俄亥俄州立大学，俄亥俄，美国）；Department of Computer Science, University of Central Florida, Florida, US（计算机科学系，中央佛罗里达大学，佛罗里达，美国）
MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；International Center of Future Science, Jilin University（吉林大学未来科学国际中心）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（教育部知识驱动人机智能工程研究中心）
Multimodal LLM Integrated Semantic Communications for 6G Immersive Experiences
School of Electronic and Information Engineering, Beijing Jiaotong University（电子信息工程学院，北京交通大学）；School of Electrical and Electronic Engineering, Imperial College London（电子电气工程学院，帝国理工学院伦敦分校）
MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；East China Normal University（东华大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Institute of High Performance Computing, A*STAR（高性能计算研究所，A*STAR）
MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues
University of Notre Dame（诺丁汉大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Georgia Institute of Technology（佐治亚理工学院）；University of Montreal（蒙特利尔大学）
LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；East China Normal University（华东师范大学）；Shanghai University of Electric Power（上海电力大学）
AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions
TMLR Group, Hong Kong Baptist University（香港 Baptist 大学 TMLR 团体）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sydney AI Centre, The University of Sydney（悉尼大学 AI 中心）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Advancing AI Research Assistants with Expert-Involved Learning
Yale University（耶鲁大学）；Broad Institute of MIT and Harvard（麻省理工学院-哈佛大学博德研究所）；Google DeepMind（谷歌DeepMind）；Stanford University（斯坦福大学）；Genentech（基因泰克）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Cornell University（康奈尔大学）；Harvard University（哈佛大学）
