Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Data
UC Berkeley（加州大学伯克利分校）；ICSI（国际计算机科学研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Saarland University（萨尔大学）；Technische Universität München（慕尼黑工业大学）；Stanford University（斯坦福大学）；SPIEGEL-Verlag（明镜出版社）
Marcus Rohrbach, Anna Rohrbach, Michaela Regneri, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, Bernt Schiele
备注：
in International Journal of Computer Vision (IJCV) 2015
Activity recognition has shown impressive progress in recent years. However, the challenges of detecting fine-grained activities and understanding how they are combined into composite activities have been largely overlooked. In this work we approach both tasks and present a dataset which provides detailed annotations to address them. The first challenge is to detect fine-grained activities, which are defined by low inter-class variability and are typically characterized by fine-grained body motions. We explore how human pose and hands can help to approach this challenge by comparing two pose-based and two hand-centric features with state-of-the-art holistic features. To attack the second challenge, recognizing composite activities, we leverage the fact that these activities are compositional and that the essential components of the activities can be obtained from textual descriptions or scripts. We show the benefits of our hand-centric approach for fine-grained activity classification and detection. For composite activity recognition we find that decomposition into attributes allows sharing information across composites and is essential to attack this hard task. Using script data we can recognize novel composites without having training data for them.
Facebook AI Research (FAIR)（Facebook人工智能研究院（FAIR））
To understand the visual world, a machine must not only recognize individual object instances but also how they interact. Humans are often at the center of such interactions and detecting human-object interactions is an important practical and scientific problem. In this paper, we address the task of detecting <human, verb, object> triplets in challenging everyday photos. We propose a novel model that is driven by a human-centric approach. Our hypothesis is that the appearance of a person -- their pose, clothing, action -- is a powerful cue for localizing the objects they are interacting with.…
Huazhong University of Science and Technology（华中科技大学）
Ethiopic/Amharic script is one of the oldest African writing systems, which serves at least 23 languages (e.g., Amharic, Tigrinya) in East Africa for more than 120 million people. The Amharic writing system, Abugida, has 282 syllables, 15 punctuation marks, and 20 numerals. The Amharic syllabic matrix is derived from 34 base graphemes/consonants by adding up to 12 appropriate diacritics or vocalic markers to the characters. The syllables with a common consonant or vocalic markers are likely to be visually similar and challenge text recognition tasks. In this work, we presented the first compre…
Challenges in Multi-centric Generalization: Phase and Step Recognition in Roux-en-Y Gastric Bypass Surgery
University of Basel（巴塞尔大学）；University of Strasbourg（斯特拉斯堡大学）；University of Verona（维罗纳大学）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；Inselspital Bern University Hospital（伯尔尼大学医院）；IRCAD France（法国消化外科和腹腔镜影像研究所）
EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction
Beihang University（北京航空航天大学）；University of Science and Technology Beijing（北京科技大学）；School of Computer Science and Engineering（计算机科学与工程学院）；State Key Laboratory of Virtual Reality Technology and Systems（虚拟现实技术与系统国家重点实验室）
The cuneiform script constitutes one of the earliest systems of writing and is realized by wedge-shaped marks on clay tablets. A tremendous number of cuneiform tablets have already been discovered and are incrementally digitalized and made available to automated processing. As reading cuneiform script is still a manual task, we address the real-world application of recognizing cuneiform signs by two graph based methods with complementary runtime characteristics. We present a graph model for cuneiform signs together with a tailored distance measure based on the concept of the graph edit distanc…
VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanjing University（南京大学）；Shanghai Innovation Institute（上海创新研究院）；City University of Hong Kong（香港城市大学）
Recognizing fonts has become an important task in document analysis, due to the increasing number of available digital documents in different fonts and emphases. A generic font-recognition system independent of language, script and content is desirable for processing various types of documents. At the same time, categorizing calligraphy styles in handwritten manuscripts is important for palaeographic analysis, but has not been studied sufficiently in the literature. We address the font-recognition problem as analysis and categorization of textures. We extract features using complex wavelet tra…
Recent years have witnessed rapid progress in detecting and recognizing individual object instances. To understand the situation in a scene, however, computers need to recognize how humans interact with surrounding objects. In this paper, we tackle the challenging task of detecting human-object interactions (HOI). Our core idea is that the appearance of a person or an object instance contains informative cues on which relevant parts of an image to attend to for facilitating interaction prediction. To exploit these cues, we propose an instance-centric attention module that learns to dynamically…
Detecting Children with Autism Spectrum Disorder based on Script-Centric Behavior Understanding with Emotional Enhancement
School of Computer Science, Wuhan University（武汉大学计算机学院）；Digital Innovation Research Center, Duke Kunshan University（杜克昆山大学数字创新研究中心）；School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息学院）；Child Development and Behavior Center, Third Affiliated Hospital of Sun Yat-sen University（中山大学第三附属医院儿童发展与行为中心）
A Cross-Font Image Retrieval Network for Recognizing Undeciphered Oracle Bone Inscriptions
Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能学院）；School of Informatics, Xiamen University（厦门大学信息学院）；Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism（文化和旅游部闽南（厦门大学）非物质文化遗产数字化保护与智能处理重点实验室）
LIRMM；Université de Montpellier（蒙彼利埃大学）；CNRS（法国国家科学研究中心）；Technische Universiteit Delft（代尔夫特理工大学）
Intuitive user interfaces are indispensable to interact with the human centric smart environments. In this paper, we propose a unified framework that recognizes both static and dynamic gestures, using simple RGB vision (without depth sensing). This feature makes it suitable for inexpensive human-robot interaction in social or industrial settings. We employ a pose-driven spatial attention strategy, which guides our proposed Static and Dynamic gestures Network - StaDNet. From the image of the human upper body, we estimate his/her depth, along with the region-of-interest around his/her hands. The…
Institute of Engineering & Management（工程与管理学院）；Indian Institute of Technology Roorkee（印度理工学院鲁尔基分校）；Indian Statistical Institute（印度统计研究所）
Handwritten word recognition and spotting of low-resource scripts are difficult as sufficient training data is not available and it is often expensive for collecting data of such scripts. This paper presents a novel cross language platform for handwritten word recognition and spotting for such low-resource scripts where training is performed with a sufficiently large dataset of an available script (considered as source script) and testing is done on other scripts (considered as target script). Training with one source script and testing with another script to have a reasonable result is not ea…
The main challenge in on-line handwritten character recognition in Indian lan- guage is the large size of the character set, larger similarity between different characters in the script and the huge variation in writing style. In this paper we propose a framework for on-line handwitten script recognition taking cues from speech signal processing literature. The framework is based on identify- ing strokes, which in turn lead to recognition of handwritten on-line characters rather that the conventional character identification. Though the framework is described for Devanagari script, the framewo…
Center for Visual Information Technology, IIIT Hyderabad（海得拉巴国际信息技术研究所视觉信息技术中心）
Building robust recognizers for Arabic has always been challenging. We demonstrate the effectiveness of an end-to-end trainable CNN-RNN hybrid architecture in recognizing Arabic text in videos and natural scenes. We outperform previous state-of-the-art on two publicly available video text datasets - ALIF and ACTIV. For the scene text recognition task, we introduce a new Arabic scene text dataset and establish baseline results. For scripts like Arabic, a major challenge in developing robust recognizers is the lack of large quantity of annotated data. We overcome this by synthesising millions of…
Handwriting recognition refers to the identification of written characters. Handwriting recognition has become an acute research area in recent years for the ease of access of computer science. In this paper primarily discussed On-line and Off-line handwriting recognition methods for Arabic words which are often used among then across the Middle East and North Africa People. Arabic word online handwriting recognition is a very challenging task due to its cursive nature. Because of the characteristic of the whole body of the Arabic script, namely connectivity between the characters, thereby the…
University of Science and Technology of China（中国科学技术大学）
Affordance-centric Question-driven Task Completion for Egocentric Assistant(AQTC) is a novel task which helps AI assistant learn from instructional videos and scripts and guide the user step-by-step. In this paper, we deal with the AQTC via a two-stage Function-centric approach, which consists of Question2Function Module to ground the question with the related function and Function2Answer Module to predict the action based on the historical steps. We evaluated several possible solutions in each module and obtained significant gains compared to the given baselines. Our code is available at \url…
Several techniques have been proposed to address the problem of recognizing activities of daily living from signals. Deep learning techniques applied to inertial signals have proven to be effective, achieving significant classification accuracy. Recently, research in human activity recognition (HAR) models has been almost totally model-centric. It has been proven that the number of training samples and their quality are critical for obtaining deep learning models that both perform well independently of their architecture, and that are more robust to intraclass variability and interclass simila…
University of Southern California（南加州大学）；Google（谷歌）
Longform media such as movies have complex narrative structures, with events spanning a rich variety of ambient visual scenes. Domain specific challenges associated with visual scenes in movies include transitions, person coverage, and a wide array of real-life and fictional scenarios. Existing visual scene datasets in movies have limited taxonomies and don't consider the visual scene transition within movie clips. In this work, we address the problem of visual scene recognition in movies by first automatically curating a new and extensive movie-centric taxonomy of 179 scene labels derived fro…
Arabic text recognition is a challenging task because of the cursive nature of Arabic writing system, its joint writing scheme, the large number of ligatures and many other challenges. Deep Learning DL models achieved significant progress in numerous domains including computer vision and sequence modelling. This paper presents a model that can recognize Arabic text that was printed using multiple font types including fonts that mimic Arabic handwritten scripts. The proposed model employs a hybrid DL network that can recognize Arabic printed text without the need for character segmentation. The…
Object-Centric Instruction Augmentation for Robotic Manipulation
School of Computer Science, East China Normal University（华东师范大学计算机科学学院）；Midea Group（美的集团）；Department of Mathematics, School of Science, Shanghai University（上海大学理学院数学系）
Recognizing Actions from Robotic View for Natural Human-Robot Interaction
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School（1 人工智能通用基础理论国家重点实验室，北京大学深圳研究生院）；Sun Yat-sen University（2 中山大学）；Kiel University（3 科尔堡大学）；Lancaster University（4 拉斯特纳大学）；State University of New York at Buffalo（5 纽约州立大学布法罗分校）
