This paper presents an implemented system for recognizing the occurrence of events described by simple spatial-motion verbs in short image sequences. The semantics of these verbs is specified with event-logic expressions that describe changes in the state of force-dynamic relations between the participants of the event. An efficient finite representation is introduced for the infinite sets of intervals that occur when describing liquid and semi-liquid events. Additionally, an efficient procedure using this representation is presented for inferring occurrences of compound events, described with event-logic expressions, from occurrences of primitive events. Using force dynamics and event logic to specify the lexical semantics of events allows the system to be more robust than prior systems based on motion profile.
Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling
McGovern Institute for Brain Research, MIT(麻省理工学院麦戈文脑科学研究所);Department of Brain and Cognitive Sciences, MIT(麻省理工学院脑与认知科学系);The Program in Speech and Hearing Bioscience and Technology, Harvard University(哈佛大学言语与听觉生物科学与技术项目);CSAIL, MIT(麻省理工学院计算机科学与人工智能实验室)