Transformer-based encoder-decoder networks have recently achieved impressive results in handwritten text recognition, partly thanks to their auto-regressive decoder which implicitly learns a language model. However, such networks suffer from a large performance drop when evaluated on a target corpus whose language distribution is shifted from the source text seen during training. To retain recognition accuracy despite this language shift, we propose an external n-gram injection (NGI) for dynamic adaptation of the network's language modeling at inference time. Our method allows switching to an n-gram language model estimated on a corpus close to the target distribution, therefore mitigating bias without any extra training on target image-text pairs. We opt for an early injection of the n-gram into the transformer decoder so that the network learns to fully leverage text-only data at the low additional cost of n-gram inference. Experiments on three handwritten datasets demonstrate that the proposed NGI significantly reduces the performance gap between source and target corpora.

Kaligo-based Intelligent Handwriting Teacher (KIHT) is a bi-nationally funded research project. The aim of this joint project is to develop an intelligent learning device for automated handwriting, composed of existing components, which can be made available to as many students as possible. With KIHT, we specifically address the challenging task of using inertial sensors to retrace the trajectory of a pen without relying on external reference systems. The nearly unlimited freedom to let the pen glide over the paper has not yet provided a satisfactory solution to this challenge in the state-of-the-art methods, even with sophisticated algorithms and AI approaches. The final phase of the project is now being launched and together with partners from industry and academia, we are taking a holistic approach by considering the entire chain of components, from the pen to the embedded processing system, the algorithms and the app.

State-of-the-art methods for handwriting text recognition are based on deep learning approaches and language modeling that require large data sets during training. In practice, there are some applications where the system processes mono-writer documents, and would thus benefit from being trained on examples from that writer. However, this is not common to have numerous examples coming from just one writer. In this paper, we propose an approach to adapt both the optical model and the language model to a particular writer, from a generic system trained on large data sets with a variety of examples. We show the benefits of the optical and language model writer adaptation. Our approach reaches competitive results on the READ 2018 data set, which is dedicated to model adaptation to particular writers.

Augmented feedback has been shown to improve interaction in virtual environments and to facilitate motor learning. Recent studies proposed this type of feedback to guide users, to highlight specific areas or to help them to perform a specific task. They can follow a path, pass through specific waypoints or even mimic an avatar. However these approaches do not show the gap between learners' performance and the desired one. Our hypothesis is that by revealing this gap to the users, they will reduce it step by step and tend to the required performance. Thus, in this paper, we propose a new visual metaphor to guide trainees' gestures by showing trajectory errors instead of showing the path to follow. In a first study we evaluated trainees' improvement by measuring the mentioned gap. First results indicate that our approach allows an enhanced task performance.

We introduce a novel handwriting recognition approach based on sub-lexical units known as multigrams of characters, that are variable lengths characters sequences. A Hidden Semi Markov model is used to model the multigrams occurrences within the target language corpus. Decoding the training language corpus with this model provides an optimized multigram lexicon of reduced size with high coverage rate of OOV compared to the traditional word modeling approach. The handwriting recognition system is composed of two components: the optical model and the statistical n-grams of multigrams language model. The two models are combined together during the recognition process using a decoding technique based on Weighted Finite State Transducers (WFST). We experiment the approach on two Latin language datasets (the French RIMES and English IAM datasets) and we show that it outperforms words and character models language models for high Out Of Vocabulary (OOV) words rates, and that it performs similarly to these traditional models for low OOV rates, with the advantage of a reduced complexity.

Pedagogical Action Selection (PAS) is a major issue for intelligent tutoring and training systems. Expert knowledge provides useful insights to build strategies that relate students representation to PAS, but it can be difficult to collect. Furthermore, the influence of a specific action may vary across students, which is rarely reflected in expert knowledge. As part of an automatic gesture training system, we propose to model the co-evolution between a student and a training environment in order to provide personalized action selection. The proposed approach is based on three models representing the student, the environment, and the interactions between these two entities. The latter model sees the PAS as a multi-armed bandit problem, each arm representing a possible action. Thus, PAS personalization only relies on the interactions between the student and the learning environment, without any prior knowledge. Two experiments, one in a simulated environment and a second in a calligraphy training environment, highlight the model ability to personalize action selection, and the benefits of this ability on students skill acquisition.