We summarize the tasks, protocol, and outcome for the 6th Competition on Recognition of Handwritten Mathematical Expressions (CROHME), which includes a new formula detection in document images task (+ TFD). For CROHME + TFD 2019, participants chose between two tasks for recognizing handwritten formulas from 1) online stroke data, or 2) images generated from the handwritten strokes. To compare LATEX strings and the labeled directed trees over strokes (label graphs) used in previous CROHMEs, we convert LATEX and stroke-based label graphs to label graphs defined over symbols (symbol-level label graphs, or symLG). More than thirty (33) participants registered for the competition, with nineteen (19) teams submitting results. The strongest formula recognition results were produced by the USTC-iFLYTEK research team, for both stroke-based (81%) and image-based (77%) input. For the new typeset formula detection task, the Samsung R&D Institute Ukraine (Team 2) obtained a very strong F-score (93%). System performance has improved since the last CROHME - still, the competition results suggest that recognition of handwritten formulae remains a difficult structural pattern recognition task.

In this paper a multi-classifier method for early recognition of handwritten gesture is presented. Unlike the other works which study the early recognition problem related to the time, we propose to make the recognition according to the quantity of incremental drawing of handwritten gestures. We train a segment length based multi-classifier for the task of recognizing the handwritten touch gesture as early as possible. To deal with potential similar parts at the beginning of different gestures, we introduce a reject option to postpone the decision until ambiguity persists. We report results on two freely available datasets: MGSet and ILG. These results demonstrate the improvement we obtained by using the proposed reject option for the early recognition of handwritten gestures.

In this study, we extend the chain-structured BLSTM to tree structure topology and apply this new network model for online math expression recognition. The proposed system addresses the recognition task as a graph building problem. The input expression is a sequence of strokes from which an intermediate graph is derived using temporal and spatial relations among strokes. In this graph, a node corresponds to a stroke and an edge denotes the relationship between a pair of strokes. Then several trees are derived from the graph and labeled with Tree-based BLSTM. The last step is to merge these labeled trees to build an admissible label graph (LG) modeling 2-D formulas uniquely. The proposed system achieves competitive results in online math expression recognition domain.

Dynamic Time Warping (DTW) is a famous distance to compare two mono-stroke symbols. It obeys the boundary and continuity constraints. The extension to multi-stroke symbols raises specific problems. A naïve solution is to convert the multi-stroke symbol into a single one by a direct concatenation respecting the handwriting order. However, people may write a symbol with different stroke orders and different stroke directions. Applying a brute force method by searching all the possible directions and orders leads to prohibitive calculation times. To reduce the searching complexity, we propose the DTW-A* algorithm that keeps the continuity constraint during each partial matching. This DTW-A* distance achieves the best recognition rate and the best stability in cross-validation when comparing three distances (DTW-A*, DTW, Modified Hausdorff Distance) on a flowchart dataset which mainly contains multi-stroke symbols.

We report on the third international Competition on Handwritten Mathematical Expression Recognition (CROHME), in which eight teams from academia and industry took part. For the third CROHME, the training dataset was expanded to over 8000 expressions, and new tools were developed for evaluating performance at the level of strokes as well as expressions and symbols. As an informal measure of progress, the performance of the participating systems on the CROHME 2012 data set is also reported. Data and tools used for the competition will be made publicly available.

In this paper, we study the problem of symbol knowledge extraction. We assume that some unknown symbols are used to compose a handwritten message, and from a dataset of handwritten samples, we would like to recover the symbol set used in the corresponding language. We applied our approach on online handwriting, and select the domain of numerical expressions, mixing digits and operators, to test the ability to retrieve the corresponding symbol classes. The proposed method is based on three steps: a quantization of the stroke space, a description of the layout of strokes with a relational graph, and the extraction of an optimal lexicon using a minimum description length algorithm. At the symbol level, a recall rate of 74% is obtained on the test dataset produced by 100 writers.

Considerable efforts are being done within the scientific community to make as easier as possible the way that the human being converses with its machine. Handwriting and speech are two common ways used to achieve this goal and are probably among those which attracted much interest. In mathematical content recognition tasks, these two modalities are used with a certain success. This paper presents an architecture based on a speech handwriting data fusion for isolated mathematical symbol recognition. Different fusion methods are explored. The results are very encouraging since recognition rates are increased comparatively to mono modality approaches.

A competition on recognition of online handwritten mathematical expressions is organized. Recognition of mathematical expressions has been an attractive problem for the pattern recognition community because of the presence of enormous uncertainties and ambiguities as encountered during parsing of the two-dimensional structure of expressions. The goal of this competition is to bring out a state of the art for the related research. Three labs come together to organize the event and six other research groups participated the competition. The competition defines a standard format for presenting information, provides a training set of 921 expressions and supplies the underlying grammar for understanding the content of the training data. Participants were invited to submit their recognizers which were tested with a new set of 348 expressions. Systems are evaluated based on four different aspects of the recognition problem. However, the final rating of the systems is done based on their correct expression recognition accuracies. The best expression level recognition accuracy (on the test data) shown by the competing systems is 19.83% whereas a baseline system developed by one of the organizing groups reports an accuracy 22.41% on the same data set.

In this paper, we present HAMEX, a new public dataset that contains mathematical expressions available in their on-line handwritten form and in their audio spoken form. We have designed this dataset so that, given a mathematical expression, its handwritten signal and its audio signal can be used jointly to design multimodal recognition systems. Here, we describe the different steps that allowed us to acquire this dataset, from the creation of the mathematical expression corpora (including expressions from Wikipedia pages) to the segmentation and the transcription of the collected data, via the data collection process itself. Currently, the dataset contains 4 350 on-line handwritten mathematical expressions written by 58 writers, and the corresponding audio expressions (in French) spoken by 58 speakers. The ground truth is also provided both for the handwritten expressions (as INKML files with the digital ink, the symbol segmentation, and the MATHML structure) and for the audio expressions (as XML files with the transcriptions of the spoken expressions).

Evaluating mathematical expression recognition involves a complex interaction of input primitives (e.g. pen/finger strokes), recognized symbols, and recognized spatial structure. Existing performance metrics simplify this problem by separating the assessment of spatial structure from the assessment of symbol segmentation and classification. These metrics do not characterize the overall accuracy of a pen-based mathematics recognition, making it difficult to compare math recognition algorithms, and preventing the use of machine learning algorithms requiring a criterion function characterizing overall system performance. To address this problem, we introduce performance metrics that bridge the gap from handwritten strokes to spatial structure. Our metrics are computed using bipartite graphs that represent classification, segmentation and spatial structure at the stroke level. Overall correctness of an expression is measured by counting the number of relabelings of nodes and edges needed to make the bipartite graph for a recognition result match the bipartite graph for ground truth. This metric may also be used with other primitive types (e.g. image pixels).

In this paper, we propose a new framework for online handwritten mathematical expression recognition. The proposed architecture aims at handling mathematical expression recognition as a simultaneous optimization of symbol segmentation, symbol recognition, and 2D structure recognition under the restriction of a mathematical expression grammar. To achieve this goal, we consider a hypothesis generation mechanism supporting a 2D grouping of elementary strokes, a cost function defining the global likelihood of a solution, and a dynamic programming scheme giving at the end the best global solution according to a 2D grammar and a classifier. As a classifier, a neural network architecture is used; it is trained within the overall architecture allowing rejecting incorrect segmented patterns. The proposed system is trained with a set of synthetic online handwritten mathematical expressions. When tested on a set of real complex expressions, the system achieves promising results at both symbol and expression interpretation levels.

We present an automatic on-line adaptation mechanism to the writer's handwriting style for the recognition of isolated handwritten characters. The classifier is based on a fuzzy inference system (FIS). This FIS is composed of fuzzy prototypes which represent the intrinsic properties of the classes and it uses numeric conclusions. The proposed adaptation mechanism affects both the conclusions of the rules and the fuzzy prototypes of the premises by re-centering and re-shaping them. Doing so, the FIS is automatically fitted to the handwriting style of the writer that is currently using the system. This adaptation mechanism has been tested with 8 different writers. The results show the adaptation mechanism is able to improve the recognition rate from 88% to 98.2% in average for the 26 Latin letters.