By combining voice and touch interactions, multimodal interfaces can surpass the efficiency of either modality alone. Traditional multimodal frameworks require laborious developer work to support rich multimodal commands where the user's multimodal command involves possibly exponential combinations of actions/function invocations. This paper presents ReactGenie, a programming framework that better separates multimodal input from the computational model to enable developers to create efficient and capable multimodal interfaces with ease. ReactGenie translates multimodal user commands into NLPL (Natural Language Programming Language), a programming language we created, using a neural semantic parser based on large-language models. The ReactGenie runtime interprets the parsed NLPL and composes primitives in the computational model to implement complex user commands. As a result, ReactGenie allows easy implementation and unprecedented richness in commands for end-users of multimodal apps. Our evaluation showed that 12 developers can learn and build a nontrivial ReactGenie application in under 2.5 hours on average. In addition, compared with a traditional GUI, end-users can complete tasks faster and with less task load using ReactGenie apps.
WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code
Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI)、中国电信);Northwestern Polytechnical University(西北工业大学);Beijing Jiaotong University(北京交通大学);Nanjing University(南京大学)
Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality
Centrum Wiskunde & Informatica(数学与计算机科学中心);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);University College London(伦敦大学学院);Shandong University(山东大学);TU Delft(代尔夫特理工大学)
Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective
Shanghai AI Laboratory(上海人工智能实验室);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
Shenzhen Institute of Advanced Technology Chinese Academy of Sciences(深圳先进技术研究院中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Dalian University of Technology(大连理工大学);UNSW Sydney(悉尼大学);Shenzhen University of Advanced Technology(深圳先进技术大学)
ClassMind: Scaling Classroom Observation and Instructional Feedback with Multimodal AI
Massachusetts Institute of Technology(麻省理工学院);Michigan State University(密歇根州立大学);National University of Singapore(新加坡国立大学);New York University(纽约大学);M3S, Singapore-MIT Alliance for Research and Technology(新加坡-麻省理工学院研究与技术创新中心);MIT Open Learning, Massachusetts Institute of Technology(麻省理工学院开放学习项目)
Smart Device Development for Gait Monitoring: Multimodal Feedback in an Interactive Foot Orthosis, Walking Aid, and Mobile Application
Frankfurt University of Applied Sciences(法兰克福应用科学大学);University of Cadiz(加的斯大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校);Arizona State University(亚利桑那州立大学);Instituto de Investigación e Innovación Biomédica de Cádiz (INiBICA)(加的斯生物医学研究与创新研究所)
HEART-Watch: A multimodal physiological dataset from a Google Pixel Watch across different physical states
Institute of Biomedical Engineering, University of Toronto(多伦多大学生物医学工程研究所);KITE Research Institute, University Health Network(大学健康网络基特研究所);College of Engineering and Technology, American University of the Middle East(中东美国大学工程技术学院)
K2MUSE: A human lower-limb multimodal walking dataset spanning task and acquisition variability for rehabilitation robotics
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences(机器人与智能系统国家重点实验室,沈阳自动化研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);College of Artificial Intelligence, Tianjin Key Laboratory of Intelligent Robotics, Nankai University(人工智能学院,天津智能机器人重点实验室,南开大学);School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(人工智能与自动化学院,华中科技大学)