Handwritten text recognition aims to convert visual input into machine-readable text, and it remains challenging due to the evolving and context-dependent nature of handwriting. Character sets change over time, and character frequency distributions shift across historical periods or regions, often causing models trained on broad, heterogeneous corpora to underperform on specific subsets. To tackle this, we propose a novel loss function that incorporates the Wasserstein distance between the character frequency distribution of the predicted text and a target distribution empirically derived from training data. By penalizing divergence from expected distributions, our approach enhances both accuracy and robustness under temporal and contextual intra-dataset shifts. Furthermore, we demonstrate that character distribution alignment can also improve existing models at inference time without requiring retraining by integrating it as a scoring function in a guided decoding scheme. Experimental results across multiple datasets and architectures confirm the effectiveness of our method in boosting generalization and performance. We open source our code at https://github.com/pkaliosis/fada.
LITIS Laboratory（LITIS实验室）；Rouen University（鲁昂大学）；Normandie University（诺曼底大学）；INSA of Rouen（鲁昂国立应用科学学院）
Unconstrained handwriting recognition is an essential task in document analysis. It is usually carried out in two steps. First, the document is segmented into text lines. Second, an Optical Character Recognition model is applied on these line images. We propose the Simple Predict & Align Network: an end-to-end recurrence-free Fully Convolutional Network performing OCR at paragraph level without any prior segmentation stage. The framework is as simple as the one used for the recognition of isolated lines and we achieve competitive results on three popular datasets: RIMES, IAM and READ 2016. The…
Duke University（杜克大学）；The Statistical and Applied Mathematical Sciences Institute（统计与应用数学科学研究所）；NVIDIA Corporation（英伟达公司）
The paper approaches the task of handwritten text recognition (HTR) with attentional encoder-decoder networks trained on sequences of characters, rather than words. We experiment on lines of text from popular handwriting datasets and compare different activation functions for the attention mechanism used for aligning image pixels and target characters. We find that softmax attention focuses heavily on individual characters, while sigmoid attention focuses on multiple characters at each step of the decoding. When the sequence alignment is one-to-one, softmax attention is able to learn a more pr…
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
The Transformer-based encoder-decoder architecture has recently made significant advances in recognizing handwritten mathematical expressions. However, the transformer model still suffers from the lack of coverage problem, making its expression recognition rate (ExpRate) inferior to its RNN counterpart. Coverage information, which records the alignment information of the past steps, has proven effective in the RNN models. In this paper, we propose CoMER, a model that adopts the coverage information in the transformer decoder. Specifically, we propose a novel Attention Refinement Module (ARM) t…
Arba Minch University（阿尔巴明奇大学）；Ethiopian Institute of Technology-Mekelle（默克莱埃塞俄比亚理工学院）
Amharic is the official language of the Federal Democratic Republic of Ethiopia. There are lots of historic Amharic and Ethiopic handwritten documents addressing various relevant issues including governance, science, religious, social rules, cultures and art works which are very reach indigenous knowledge. The Amharic language has its own alphabet derived from Ge'ez which is currently the liturgical language in Ethiopia. Handwritten character recognition for non Latin scripts like Amharic is not addressed especially using the advantages of the state of the art techniques. This research work de…
Entropy-Aware Structural Alignment for Zero-Shot Handwritten Chinese Character Recognition
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Shenzhen Key Laboratory of Embedded System Design（深圳嵌入式系统设计重点实验室）；Undergraduate School of Artificial Intelligence, Shenzhen Polytechnic University（深圳职业技术学院人工智能本科生学院）；Institute of Applied Artificial Intelligence of the Guangdong-Hong Kong Macao Greater Bay Area（粤港澳大湾区应用人工智能研究院）
National Institute of Technology, Hamirpur（哈米尔普尔国家理工学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Line segmentation from handwritten text images is one of the challenging task due to diversity and unknown variations as undefined spaces, styles, orientations, stroke heights, overlapping, and alignments. Though abundant researches, there is a need of improvement to achieve robustness and higher segmentation rates. In the present work, an adaptive approach is used for the line segmentation from handwritten text images merging the alignment of connected component coordinates and text height. The mathematical justification is provided for measuring the text height respective to the image size. …
University of Science and Technology of China（中国科学技术大学）
In this study, we present a novel end-to-end approach based on the encoder-decoder framework with the attention mechanism for online handwritten mathematical expression recognition (OHMER). First, the input two-dimensional ink trajectory information of handwritten expression is encoded via the gated recurrent unit based recurrent neural network (GRU-RNN). Then the decoder is also implemented by the GRU-RNN with a coverage-based attention model. The proposed approach can simultaneously accomplish the symbol recognition and structural analysis to output a character sequence in LaTeX format. Vali…
Optimal Transport for Handwritten Text Recognition in a Low-Resource Regime
Robotics Institute, Athena Research Center（机器人研究所，亚特兰蒂斯研究中心）；School of ECE, National Technical University of Athens（电子工程学院，国家技术大学雅典）；Department of SGE, University of West Attica（SGE系，西阿提卡大学）；HERON - Hellenic Robotics Center of Excellence, Athens, Greece（HERON - 希腊机器人卓越中心）
Advancing Offline Handwritten Text Recognition: A Systematic Review of Data Augmentation and Generation Techniques
Artificial Intelligence and Innovation Centre, University of Kurdistan Hewler（库尔德斯坦胡勒尔大学人工智能与创新中心）；Computer Science and Engineering Department, School of Science and Engineering, University of Kurdistan Hewler（库尔德斯坦胡勒尔大学科学与工程学院计算机科学与工程系）；School of Computing and Engineering, University of Huddersfield（赫德斯菲尔德大学计算与工程学院）；Department of Computer Science, College of Science, Charmo University（查莫大学科学学院计算机科学系）；Software and Informatics Engineering, College of Engineering, Salahaddin University-Erbil（萨拉赫丁大学埃尔比尔工程学院软件与信息技术工程）
Cooperative Madianet Innovation Center, Shanghai Jiao Tong University（上海交通大学Madianet协同创新中心）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）；PingAn Technology US Research Lab（平安科技美国研究实验室）；Huawei Technologies Noah’s Ark Lab（华为技术诺亚方舟实验室）
Automatic character generation is an appealing solution for new typeface design, especially for Chinese typefaces including over 3700 most commonly-used characters. This task has two main pain points: (i) handwritten characters are usually associated with thin strokes of few information and complex structure which are error prone during deformation; (ii) thousands of characters with various shapes are needed to synthesize based on a few manually designed characters. To solve those issues, we propose a novel convolutional-neural-network-based model with three main techniques: collaborative stro…
University of Science and Technology of China（中国科学技术大学）；National Engineering Laboratory for Speech and Language Information Processing（国家语音与语言信息处理工程实验室）
In this paper, we propose a novel stroke constrained attention network (SCAN) which treats stroke as the basic unit for encoder-decoder based online handwritten mathematical expression recognition (HMER). Unlike previous methods which use trace points or image pixels as basic units, SCAN makes full use of stroke-level information for better alignment and representation. The proposed SCAN can be adopted in both single-modal (online or offline) and multi-modal HMER. For single-modal HMER, SCAN first employs a CNN-GRU encoder to extract point-level features from input traces in online mode and em…
Indian Institute of Technology Roorkee（印度理工学院鲁尔基分校）；Institute of Engineering & Management, Kolkata（加尔各答工程与管理学院）；Indian Statistical Institute, Kolkata（印度统计学院加尔各答分校）
This paper presents a novel approach towards Indic handwritten word recognition using zone-wise information. Because of complex nature due to compound characters, modifiers, overlapping and touching, etc., character segmentation and recognition is a tedious job in Indic scripts (e.g. Devanagari, Bangla, Gurumukhi, and other similar scripts). To avoid character segmentation in such scripts, HMM-based sequence modeling has been used earlier in holistic way. This paper proposes an efficient word recognition framework by segmenting the handwritten word images horizontally into three zones (upper, …
Recurrent neural network (RNN) and connectionist temporal classification (CTC) have showed successes in many sequence labeling tasks with the strong ability of dealing with the problems where the alignment between the inputs and the target labels is unknown. Residual network is a new structure of convolutional neural network and works well in various computer vision tasks. In this paper, we take advantage of the architectures mentioned above to create a new network for handwritten digit string recognition. First we design a residual network to extract features from input images, then we employ…
Stroke order and velocity are helpful features in the fields of signature verification, handwriting recognition, and handwriting synthesis. Recovering these features from offline handwritten text is a challenging and well-studied problem. We propose a new model called TRACE (Trajectory Recovery by an Adaptively-trained Convolutional Encoder). TRACE is a differentiable approach that uses a convolutional recurrent neural network (CRNN) to infer temporal stroke information from long lines of offline handwritten text with many characters and dynamic time warping (DTW) to align predictions and grou…
University of Rostock（罗斯托克大学）；PLANET artificial intelligence GmbH（PLANET人工智能有限公司）
Encoder-decoder models have become an effective approach for sequence learning tasks like machine translation, image captioning and speech recognition, but have yet to show competitive results for handwritten text recognition. To this end, we propose an attention-based sequence-to-sequence model. It combines a convolutional neural network as a generic feature extractor with a recurrent neural network to encode both the visual information, as well as the temporal context between characters in the input image, and uses a separate recurrent neural network to decode the actual character sequence. …
Leveraging Transfer Learning and Mobile-enabled Convolutional Neural Networks for Improved Arabic Handwritten Character Recognition
Systems Theory and Informatics Laboratory, University Moulay Ismail, Meknes, Morocco（穆拉伊姆大学系统理论与信息实验室）；LMC, Polydisciplinary Faculty of Safi, Cadi Ayyad University, Morocco（卡迪·艾亚德大学萨菲多学科 faculty）；College of Engineering and Information Technology, University of Dubai, Dubai, UAE（迪拜大学工程与信息技术学院）
Fuzzy-Geometric Branch-Point Modeling for Structure-Aware Augmentation of Handwritten Chinese Characters
School of Information Science and Engineering, Lanzhou University（兰州大学信息科学与工程学院）；Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系广东省类脑智能计算重点实验室）
South China University of Technology（华南理工大学）；Lenovo Research（联想研究院）
Text recognition has attracted considerable research interests because of its various applications. The cutting-edge text recognition methods are based on attention mechanisms. However, most of attention methods usually suffer from serious alignment problem due to its recurrency alignment operation, where the alignment relies on historical decoding results. To remedy this issue, we propose a decoupled attention network (DAN), which decouples the alignment operation from using historical decoding results. DAN is an effective, flexible and robust end-to-end text recognizer, which consists of thr…
In most machine learning training paradigms a fixed, often handcrafted, loss function is assumed to be a good proxy for an underlying evaluation metric. In this work we assess this assumption by meta-learning an adaptive loss function to directly optimize the evaluation metric. We propose a sample efficient reinforcement learning approach for adapting the loss dynamically during training. We empirically show how this formulation improves performance by simultaneously optimizing the evaluation metric and smoothing the loss landscape. We verify our method in metric learning and classification sc…
Word-level handwritten optical character recognition (OCR) remains a challenge for morphologically rich languages like Bangla. The complexity arises from the existence of a large number of alphabets, the presence of several diacritic forms, and the appearance of complex conjuncts. The difficulty is exacerbated by the fact that some graphemes occur infrequently but remain indispensable, so addressing the class imbalance is required for satisfactory results. This paper addresses this issue by introducing two knowledge distillation methods: Leveraging Isolated Letter Accumulations By Ordering Tea…
Rochester Institute of Technology（罗切斯特理工学院）；Kodak Alaris（柯达乐芮）
Handwritten text recognition is challenging because of the virtually infinite ways a human can write the same message. Our fully convolutional handwriting model takes in a handwriting sample of unknown length and outputs an arbitrary stream of symbols. Our dual stream architecture uses both local and global context and mitigates the need for heavy preprocessing steps such as symbol alignment correction as well as complex post processing steps such as connectionist temporal classification, dictionary matching or language models. Using over 100 unique symbols, our model is agnostic to Latin-base…
University of Maryland（马里兰大学）；GE Global Research（通用电气全球研究中心）；Avitas Systems（阿维塔斯系统公司）
Visual Domain Adaptation is a problem of immense importance in computer vision. Previous approaches showcase the inability of even deep neural networks to learn informative representations across domain shift. This problem is more severe for tasks where acquiring hand labeled data is extremely hard and tedious. In this work, we focus on adapting the representations learned by segmentation networks across synthetic and real domains. Contrary to previous approaches that use a simple adversarial objective or superpixel information to aid the process, we propose an approach based on Generative Adv…
