We present DocFormer -- a multi-modal transformer based architecture for the task of Visual Document Understanding (VDU). VDU is a challenging problem which aims to understand documents in their varied formats (forms, receipts etc.) and layouts. In addition, DocFormer is pre-trained in an unsupervised fashion using carefully designed tasks which encourage multi-modal interaction. DocFormer uses text, vision and spatial features and combines them using a novel multi-modal self-attention layer. DocFormer also shares learned spatial embeddings across modalities which makes it easy for the model to correlate text to visual tokens and vice versa. DocFormer is evaluated on 4 different datasets each with strong baselines. DocFormer achieves state-of-the-art results on all of them, sometimes beating models 4x its size (in no. of parameters).
DocTr: Document Image Transformer for Geometric Unwarping and Illumination Correction
University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
Automated Parsing of Engineering Drawings for Structured Information Extraction Using a Fine-tuned Document Understanding Transformer
Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR)(新加坡制造技术研究院(SIMTech),新加坡科技研究局(A*STAR));Advanced Remanufacturing and Technology Centre (ARTC), Agency for Science, Technology and Research (A*STAR)(先进再制造与技术中心(ARTC),新加坡科技研究局(A*STAR));Nanyang Technological University(南洋理工大学)
High-Fidelity Document Stain Removal via A Large-Scale Real-World Dataset and A Memory-Augmented Transformer
School of Computer Science and Engineering, Huizhou University(惠州学院计算机科学与工程学院);Faculty of Science and Technology, University of Macau(澳门大学科技学院);School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);School of Electronic and Communication Engineering, Shenzhen Polytechnic University(深圳职业技术学院电子与通信工程学院)
DocDeshadower: Frequency-Aware Transformer for Document Shadow Removal
Faculty of Science and Technology, University of Macau(澳门大学科技学院);Tp-Link International Shenzhen Co.,Ltd.(普联国际(深圳)有限公司);School of Computer Science and Engineering, Huizhou University(惠州学院计算机科学与工程学院)
DocBinFormer: A Two-Level Transformer Network for Effective Document Image Binarization
Optiks Innovations Pvt. Ltd(奥普蒂克斯创新私人有限公司);Alipurduar Government Engineering and Management College(阿里布尔杜阿尔政府工程与管理学院);Chongqing College of Mobile Communication(重庆移通学院);Indian Statistical Institute(印度统计研究所);Nanyang Technological University(南洋理工大学)
DocEnTr: An End-to-End Document Image Enhancement Transformer
Computer Vision Center Universitat Autònoma de Barcelona(巴塞罗那自治大学计算机视觉中心);Digital Research Center of Sfax(斯法克斯数字研究中心);University of Sfax(斯法克斯大学);Indian Statistical Institute(印度统计研究所)
ST-KeyS: Self-Supervised Transformer for Keyword Spotting in Historical Handwritten Documents
Digital Research Center of Sfax(斯法克斯数字研究中心);Computer Vision Center(计算机视觉中心);Universitat Autònoma de Barcelona(巴塞罗那自治大学);Blekinge Institute of Technology(布莱金厄理工学院)
Bridging the Performance Gap between DETR and R-CNN for Graphical Object Detection in Document Images
Technical University of Kaiserslautern(凯泽斯劳滕工业大学);German Research Institute for Artificial Intelligence (DFKI)(德国人工智能研究中心);Luleå University of Technology(吕勒奥理工大学)
Towards End-to-End Semi-Supervised Table Detection with Semantic Aligned Matching Transformer
Technical University of Kaiserslautern(凯泽斯劳滕工业大学);Mindgarage, Technical University of Kaiserslautern(凯泽斯劳滕工业大学Mindgarage);German Research Institute for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI))