An image is worth a thousand words, conveying information that goes beyond the physical visual content therein. In this paper, we study the intent behind social media images with an aim to analyze how visual information can help the recognition of human intent. Towards this goal, we introduce an intent dataset, Intentonomy, comprising 14K images covering a wide range of everyday scenes. These images are manually annotated with 28 intent categories that are derived from a social psychology taxonomy. We then systematically study whether, and to what extent, commonly used visual information, i.e., object and context, contribute to human motive understanding. Based on our findings, we conduct further study to quantify the effect of attending to object and context classes as well as textual information in the form of hashtags when training an intent classifier. Our results quantitatively and qualitatively shed light on how visual and textual information can produce observable effects when predicting intent.
In multimodal assistant, where vision is also one of the input modalities, the identification of user intent becomes a challenging task as visual input can influence the outcome. Current digital assistants take spoken input and try to determine the user intent from conversational or device context. So, a dataset, which includes visual input (i.e. images or videos for the corresponding questions targeted for multimodal assistant use cases, is not readily available. The research in visual question answering (VQA) and visual question generation (VQG) is a great step forward. However, they do not …
Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Michigan（密歇根大学）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Stanford University（斯坦福大学）
Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions
Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）；University of Science and Technology Beijing (USTB)（北京科技大学）
In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
State Key Laboratory of VR Technology and Systems, School of CSE, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学计算机科学与工程学院）；College of AI, Tsinghua University（清华大学人工智能学院）
Memory-Augmented Query Intent Understanding for Efficient Chat-based Image Retrieval
School of Computer Science and Technology, and the School of Statistics and Mathematics, Zhejiang Gongshang University（计算机科学与技术学院，和统计与数学学院，浙江工商大学）；School of Computer Science and Technology, Zhejiang Gongshang University, and the Zhejiang Key Laboratory of Big Data and Future E-Commerce Technology（计算机科学与技术学院，浙江工商大学，和大数据与未来电子商务技术浙江省重点实验室）；Wangxuan Institute of Computer Technology, Peking University（王璇计算机技术研究所，北京大学）；School of Information and Electronic Engineering, Zhejiang Gongshang University（信息与电子工程学院，浙江工商大学）；School of Computer Science and Technology, East China Normal University（计算机科学与技术学院，华东师范大学）；Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences , Institute of Computing Technology, CAS（中国科学院智能信息处理重点实验室，计算技术研究所，中国科学院）；School of Information Science and Technology, University of Science and Technology of China（信息科学与技术学院，中国科学技术大学）
TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film
LIX, Ecole Polytechnique, IP Paris（巴黎综合理工学院LIX实验室、巴黎IP大学）；New York University（纽约大学）；Ecole nationale supérieure Louis-Lumière（路易卢米耶高等国家学院）；Stony Brook University（石溪大学）
OMR-Diffusion:Optimizing Multi-Round Enhanced Training in Diffusion Models for Improved Intent Understanding
Xiamen University（厦门大学）；University of Electronic Science and Technology of China（电子科技大学）；Shenzhen International Graduate School, Tsinghua University（深圳国际研究生院，清华大学）；Xidian University（西安电子科技大学）
Understanding passenger intents from spoken interactions and car's vision (both inside and outside the vehicle) are important building blocks towards developing contextual dialog systems for natural interactions in autonomous vehicles (AV). In this study, we continued exploring AMIE (Automated-vehicle Multimodal In-cabin Experience), the in-cabin agent responsible for handling certain multimodal passenger-vehicle interactions. When the passengers give instructions to AMIE, the agent should parse such commands properly considering available three modalities (language/text, audio, video) and tri…
IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning
University of Science and Technology of China（中国科学技术大学）；Hefei Institutes of Physical Science,Chinese Academy of Sciences（合肥物理研究所，中国科学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究所，合肥综合性国家科学中心）；State Key Lab. for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；University of Chicago（芝加哥大学）；National University of Defense Technology（国防科技大学）；Harbin Institute of Technology（哈尔滨工业大学）
