Current multimodal large language models (MLLMs) face significant challenges in visual document understanding (VDU) tasks due to the high resolution, dense text, and complex layouts typical of document images. These characteristics demand a high level of detail perception ability from MLLMs. While increasing input resolution improves detail perception capability, it also leads to longer sequences of visual tokens, increasing computational costs and straining the models' ability to handle long contexts. To address these challenges, we introduce DocKylin, a document-centric MLLM that performs visual content slimming at both the pixel and token levels, thereby reducing token sequence length in VDU scenarios. We introduce an Adaptive Pixel Slimming (APS) preprocessing module to perform pixel-level slimming, increasing the proportion of informative pixels. Moreover, we propose a novel Dynamic Token Slimming (DTS) module to conduct token-level slimming, filtering essential tokens and removing others to adaptively create a more compact visual sequence. Experiments demonstrate DocKylin's promising performance across various VDU benchmarks and the effectiveness of each component.

Handwriting verification has stood as a steadfast identity authentication method for decades. However, this technique risks potential privacy breaches due to the inclusion of personal information in handwritten biometrics such as signatures. To address this concern, we propose using the Random Digit String (RDS) for privacy-preserving handwriting verification. This approach allows users to authenticate themselves by writing an arbitrary digit sequence, effectively ensuring privacy protection. To evaluate the effectiveness of RDS, we construct a new HRDS4BV dataset composed of online naturally handwritten RDS. Unlike conventional handwriting, RDS encompasses unconstrained and variable content, posing significant challenges for modeling consistent personal writing style. To surmount this, we propose the Pattern Attentive VErification Network (PAVENet), along with a Discriminative Pattern Mining (DPM) module. DPM adaptively enhances the recognition of consistent and discriminative writing patterns, thus refining handwriting style representation. Through comprehensive evaluations, we scrutinize the applicability of online RDS verification and showcase a pronounced outperformance of our model over existing methods. Furthermore, we discover a noteworthy forgery phenomenon that deviates from prior findings and discuss its positive impact in countering malicious impostor attacks. Substantially, our work underscores the feasibility of privacy-preserving biometric verification and propels the prospects of its broader acceptance and application.

Existing scene text spotting (i.e., end-to-end text detection and recognition) methods rely on costly bounding box annotations (e.g., text-line, word-level, or character-level bounding boxes). For the first time, we demonstrate that training scene text spotting models can be achieved with an extremely low-cost annotation of a single-point for each instance. We propose an end-to-end scene text spotting method that tackles scene text spotting as a sequence prediction task. Given an image as input, we formulate the desired detection and recognition results as a sequence of discrete tokens and use an auto-regressive Transformer to predict the sequence. The proposed method is simple yet effective, which can achieve state-of-the-art results on widely used benchmarks. Most significantly, we show that the performance is not very sensitive to the positions of the point annotation, meaning that it can be much easier to be annotated or even be automatically generated than the bounding box that requires precise positions. We believe that such a pioneer attempt indicates a significant opportunity for scene text spotting applications of a much larger scale than previously possible. The code is available at https://github.com/shannanyinxiang/SPTS.

Handwritten signature verification is a challenging task because signatures of a writer may be skillfully imitated by a forger. As skilled forgeries are generally difficult to acquire for training, in this paper, we propose a deep learning-based dynamic signature verification framework, SynSig2Vec, to address the skilled forgery attack without training with any skilled forgeries. Specifically, SynSig2Vec consists of a novel learning-by-synthesis method for training and a 1D convolutional neural network model, called Sig2Vec, for signature representation extraction. The learning-by-synthesis method first applies the Sigma Lognormal model to synthesize signatures with different distortion levels for genuine template signatures, and then learns to rank these synthesized samples in a learnable representation space based on average precision optimization. The representation space is achieved by the proposed Sig2Vec model, which is designed to extract fixed-length representations from dynamic signatures of arbitrary lengths. Through this training method, the Sig2Vec model can extract extremely effective signature representations for verification. Our SynSig2Vec framework requires only genuine signatures for training, yet achieves state-of-the-art performance on the largest dynamic signature database to date, DeepSignDB, in both skilled forgery and random forgery scenarios. Source codes of SynSig2Vec will be available at https://github.com/LaiSongxuan/SynSig2Vec.

This article presents SVC-onGoing1, an on-going competition for on-line signature verification where researchers can easily benchmark their systems against the state of the art in an open common platform using large-scale public databases, such as DeepSignDB2 and SVC2021_EvalDB3, and standard experimental protocols. SVC-onGoing is based on the ICDAR 2021 Competition on On-Line Signature Verification (SVC 2021), which has been extended to allow participants anytime. The goal of SVC-onGoing is to evaluate the limits of on-line signature verification systems on popular scenarios (office/mobile) and writing inputs (stylus/finger) through large-scale public databases. Three different tasks are considered in the competition, simulating realistic scenarios as both random and skilled forgeries are simultaneously considered on each task. The results obtained in SVC-onGoing prove the high potential of deep learning methods in comparison with traditional methods. In particular, the best signature verification system has obtained Equal Error Rate (EER) values of 3.33% (Task 1), 7.41% (Task 2), and 6.04% (Task 3). Future studies in the field should be oriented to improve the performance of signature verification systems on the challenging mobile scenarios of SVC-onGoing in which several mobile devices and the finger are used during the signature acquisition.

Dynamic time warping (DTW) is a popular technique for sequence alignment, and is the de facto standard for dynamic signature verification. In this paper, we go a significant step further to enhance DTW with the capability of deep representation learning, and propose an end-to-end trainable Deep soft-DTW (DsDTW) model for dynamic signature verification. Specifically, we design a convolutional recurrent adaptive network (CRAN) to process dynamic signatures, and utilize it to provide robust and discriminative local representations as inputs for DTW. As DTW is not fully differentiable with regard to its inputs, we introduce its smoothed formulation, soft-DTW, and incorporate the soft-DTW distances of signature pairs into the loss function for optimization. Because soft-DTW is differentiable, the proposed DsDTW is end-to-end trainable, and achieves an elegant integration of CRAN deep learning model and traditional DTW mechanism. Our method achieves state-of-the-art performance on several public benchmarks, and has won first place in the ICDAR 2021 competition for online signature verification. Source codes of DsDTW is available at https://github.com/KAKAFEI123/DsDTW.

An open research problem in automatic signature verification is the skilled forgery attacks. However, the skilled forgeries are very difficult to acquire for representation learning. To tackle this issue, this paper proposes to learn dynamic signature representations through ranking synthesized signatures. First, a neuromotor inspired signature synthesis method is proposed to synthesize signatures with different distortion levels for any template signature. Then, given the templates, we construct a lightweight one-dimensional convolutional network to learn to rank the synthesized samples, and directly optimize the average precision of the ranking to exploit relative and fine-grained signature similarities. Finally, after training, fixed-length representations can be extracted from dynamic signatures of variable lengths for verification. One highlight of our method is that it requires neither skilled nor random forgeries for training, yet it surpasses the state-of-the-art by a large margin on two public benchmarks.

Handwritten mathematical expression recognition (HMER) is an important research direction in handwriting recognition. The performance of HMER suffers from the two-dimensional structure of mathematical expressions (MEs). To address this issue, in this paper, we propose a high-performance HMER model with scale augmentation and drop attention. Specifically, tackling ME with unstable scale in both horizontal and vertical directions, scale augmentation improves the performance of the model on MEs of various scales. An attention-based encoder-decoder network is used for extracting features and generating predictions. In addition, drop attention is proposed to further improve performance when the attention distribution of the decoder is not precise. Compared with previous methods, our method achieves state-of-the-art performance on two public datasets of CROHME 2014 and CROHME 2016.

Offline signature verification is a challenging task, where the scarcity of the signature data per writer makes it a few-shot problem. We found that previous deep metric learning based methods, whether in pairs or triplets, are unaware of intra-writer variations and have low training efficiency because only point-to-point (P2P) distances are considered. To address this issue, we present a novel point-to-set (P2S) metric for offline signature verification in this paper. By dividing a training batch into a support set and a query set, our optimization goal is to pull each query to its belonging support set. To further strengthen the P2S metric, a hard mining scheme and a margin strategy are introduced. Experiments conducted on three datasets show the effectiveness of our proposed method.

Offline writer identification plays an important role in forensic document examination and historical document analysis. Today, challenges still exist in historical writer identification (WI), where documents may present very complex handwriting styles. In this paper, we propose novel techniques for a detailed description and accurate identification of handwriting in historical documents. Because handwriting contours are one of the most salient components to characterize one's handwriting style, a novel pathlet feature is proposed to describe their rich properties beyond slant and curvature in a principled way; these properties can be exploited in a VLAD-like encoding framework for fine-grained handwriting description. Besides the pathlet feature, we extract unidirectional SIFT feature to describe handwriting corners and junctions. To effectively encode the pathlet and SIFT features, a novel encoding method, named bagged VLAD, is further proposed to address the problem that a large codebook sparsely spreads out the data points and leads to a degraded performance, allowing a much larger codebook for improved encoding performance. Our proposed method achieves state-of-the-art performance on ICDAR2017 Historical-WI database and ICDAR2019 HDRC-IR database, and has won the first place in ICDAR2019 HDRC-IR competition.

A new method is proposed for removing text from natural images. The challenge is to first accurately localize text on the stroke-level and then replace it with a visually plausible background. Unlike previous methods that require image patches to erase scene text, our method, namely ensconce network (EnsNet), can operate end-to-end on a single image without any prior knowledge. The overall structure is an end-to-end trainable FCN-ResNet-18 network with a conditional generative adversarial network (cGAN). The feature of the former is first enhanced by a novel lateral connection structure and then refined by four carefully designed losses: multiscale regression loss and content loss, which capture the global discrepancy of different level features; texture loss and total variation loss, which primarily target filling the text region and preserving the reality of the background. The latter is a novel local-sensitive GAN, which attentively assesses the local consistency of the text erased regions. Both qualitative and quantitative sensitivity experiments on synthetic images and the ICDAR 2013 dataset demonstrate that each component of the EnsNet is essential to achieve a good performance. Moreover, our EnsNet can significantly outperform previous state-of-the-art methods in terms of all metrics. In addition, a qualitative experiment conducted on the SBMNet dataset further demonstrates that the proposed method can also preform well on general object (such as pedestrians) removal tasks. EnsNet is extremely fast, which can preform at 333 fps on an i5-8600 CPU device.

In this paper, we propose a novel set of features for offline writer identification based on the path signature approach, which provides a principled way to express information contained in a path. By extracting local pathlets from handwriting contours, the path signature can also characterize the offline handwriting style. A codebook method based on the log path signature-a more compact way to express the path signature-is used in this work and shows competitive results on several benchmark offline writer identification datasets, namely the IAM, Firemaker, CVL and ICDAR2013 writer identification contest dataset.

Online signature verification remains a challenging task owing to large intra-individual variability. To tackle this problem, in this paper, we propose to use recurrent neural networks (RNN) for representation learning in the dynamic time warping framework. Metric-based loss functions are designed explicitly to minimize intra-individual variability and enhance inter-individual variability and to guide the RNN in learning discriminative representations for online signatures. An RNN variant-gated auto regressive units-is proposed and shows a better generalization performance in our framework. Furthermore, we interpret the online signature verification problem as a meta-learning problem: one client is considered as one task, therefore, different clients compose the task space. Based on this formulation, we design an end-to-end trainable meta-layer that learns to adapt to different clients, allowing fast adaptation to new clients in the test stage. In addition, a new descriptor-the length-normalized path signature-is proposed to describe online signatures. Our proposed system achieves a state-of-the-art performance on three benchmark datasets, namely, MCYT-100, Mobisig, and e-BioSign.

Handwritten signature is the most widespread and socially accepted method for personal authentication. Therefore automatic handwritten signature verification (HSV) is a task of realistic significance and a popular research topic in the field of pattern recognition. In this paper, we propose to learn discriminative feature hierarchies using supervised convolutional neural networks (CNN) to improve the off-line HSV performance. The feature space is modeled at both the global and local levels, combines clues from both shallow and deep representations, and is expected to capture intrinsic properties of handwritten signatures. Writer-dependent support vector machines (SVM) are trained based on the learned features for verification. Experimental results show that our method achieves competitive performance on two benchmark data sets, namely the MCYT-75 data set and the CEDAR data set.

Handwritten Chinese character recognition (HCCR) is a major research area in pattern recognition. Although recent years have seen tremendous improvement in the use of deep neural networks in handwriting recognition, unconstrained handwriting recognition remains an open problem. Considering the problem of rotated handwritten Chinese character recognition, we propose a two-stage convolutional neural network combined with path signature features to achieve high-accuracy rotation-free HCCR. For handwritten Chinese characters whose rotation angle is up to ±45°, the recognition rate can reach 97.38% on the ICDAR-2013 online HCCR competition dataset, which is comparable to the accuracy achieved by several state-of-the-art methods for non-rotated characters, thus showing the effectiveness of the proposed method.

Inspired by the great success of recurrent neural networks (RNNs) in sequential modeling, we introduce a novel RNN system to improve the performance of online signature verification. The training objective is to directly minimize intra-class variations and to push the distances between skilled forgeries and genuine samples above a given threshold. By back-propagating the training signals, our RNN network produced discriminative features with desired metrics. Additionally, we propose a novel descriptor, called the length-normalized path signature (LNPS), and apply it to online signature verification. LNPS has interesting properties, such as scale invariance and rotation invariance after linear combination, and shows promising results in online signature verification. Experiments on the publicly available SVC-2004 dataset yielded state-of-the-art performance of 2.37% equal error rate (EER).