Sequence recognition models under a fully supervised paradigm require large-scale training data, incurring substantial annotation costs. Pseudo-labeling is one of the most effective techniques in semi-supervised learning, which leverages predicted confidence to filter pseudo-labels on unlabeled data for model training. However, recent studies indicate that the performance of semi-supervised learning is compromised by overconfident models, as the predicted unreliable confidences will filter noisy samples into training. In this work, we discover that the overconfidence in sequence recognition models is influenced by the linguistic properties of a sequence, where the tail character classes are prone to be mispredicted as the head ones that frequently appear in the language with high confidence. And this overconfidence continuously intensifies throughout the semi-supervised training process. To address this limitation, we propose a Dynamic Sequential Class-Aware Smoothing (DSCS) method that calibrates the overconfidence of the head class to alleviate the inaccurate pseudo-labeling caused by overconfident misprediction to improve the quality of pseudo-labels. Specifically, we design a sequential class-aware smoothing module that incorporates token class frequency information to regularize the model and prevent it from becoming overconfident toward the head class. Meanwhile, to address the overconfidence problem intensifying throughout the semi-supervised learning processes, we introduce a dynamic regularization module to adjust the calibration strength dynamically for the coordination between the calibration and semi-supervised learning processes. Extensive experiments demonstrate the effectiveness and generality of our method, which significantly reduces annotation efforts while maintaining competitive recognition performance.

Despite notable advancements across various tasks, deep sequence recognition models are shown to grapple with the dilemma of over-confidence, leading to unreliable predicted confidence, necessitating the need for calibration. Current efforts predominantly focus on classification model calibration, leaving the sequence recognition model calibration analysis underexplored and challenging. In this work, we discover that the primary reason for over-confidence in sequence recognition models stems from the one-hot encoding target sequence training paradigm and identify two distinct manifestations of over-confidence: perception and semantic context over-confidence. To address these challenges, we propose a heterogeneous correlation aware sequence regularization (HCSR) method that adaptively incorporates correlated sequences into training alongside the target sequence as additional supervision to regularize the probability of the target sequence from arbitrarily escalating. Specifically, a correlated sequence mining (CSM) model is designed, capable of efficiently mining heterogeneous correlated sequences, which can be flexibly customized to search for specific types of correlated sequences in demand to facilitate the calibration of corresponding types of over-confidence in the calibrating model, thereby achieving fine-grained calibration. Meanwhile, an adaptive calibration module is introduced to adaptively coordinate the optimization weights between the target sequence and correlated sequences, enabling the co-calibration among different samples. Comprehensive experiments conducted on several widely employed sequence recognition tasks demonstrate that the proposed method outperforms the current competing methods by a substantial margin.

Deep sequence recognition (DSR) models receive increasing attention due to their superior application to various applications. Most DSR models use merely the target sequences as supervision without considering other related sequences, leading to over-confidence in their predictions. The DSR models trained with label smoothing regularize labels by equally and independently smoothing each token, reallocating a small value to other tokens for mitigating overconfidence. However, they do not consider tokens/sequences correlations that may provide more effective information to regularize training and thus lead to sub-optimal performance. In this work, we find tokens/sequences with high perception and semantic correlations with the target ones contain more correlated and effective information and thus facilitate more effective regularization. To this end, we propose a Perception and Semantic aware Sequence Regularization framework, which explore perceptively and semantically correlated tokens/sequences as regularization. Specifically, we introduce a semantic context-free recognition and a language model to acquire similar sequences with high perceptive similarities and semantic correlation, respectively. Moreover, over-confidence degree varies across samples according to their difficulties. Thus, we further design an adaptive calibration intensity module to compute a difficulty score for each samples to obtain finer-grained regularization. Extensive experiments on canonical sequence recognition tasks, including scene text and speech recognition, demonstrate that our method sets novel state-of-the-art results. Code is available at https://github.com/husterpzh/PSSR.