Sequence recognition models under a fully supervised paradigm require large-scale training data, incurring substantial annotation costs. Pseudo-labeling is one of the most effective techniques in semi-supervised learning, which leverages predicted confidence to filter pseudo-labels on unlabeled data for model training. However, recent studies indicate that the performance of semi-supervised learning is compromised by overconfident models, as the predicted unreliable confidences will filter noisy samples into training. In this work, we discover that the overconfidence in sequence recognition models is influenced by the linguistic properties of a sequence, where the tail character classes are prone to be mispredicted as the head ones that frequently appear in the language with high confidence. And this overconfidence continuously intensifies throughout the semi-supervised training process. To address this limitation, we propose a Dynamic Sequential Class-Aware Smoothing (DSCS) method that calibrates the overconfidence of the head class to alleviate the inaccurate pseudo-labeling caused by overconfident misprediction to improve the quality of pseudo-labels. Specifically, we design a sequential class-aware smoothing module that incorporates token class frequency information to regularize the model and prevent it from becoming overconfident toward the head class. Meanwhile, to address the overconfidence problem intensifying throughout the semi-supervised learning processes, we introduce a dynamic regularization module to adjust the calibration strength dynamically for the coordination between the calibration and semi-supervised learning processes. Extensive experiments demonstrate the effectiveness and generality of our method, which significantly reduces annotation efforts while maintaining competitive recognition performance.

In this work, we introduce Monocular and Generalizable Gaussian Talking Head Animation (MGGTalk), which requires monocular datasets and generalizes to unseen identities without personalized re-training. Compared with previous 3D Gaussian Splatting (3DGS) methods that requires elusive multi-view datasets or tedious personalized learning/inference, MGGtalk enables more practical and broader applications. However, in the absence of multi-view and personalized training data, the incompleteness of geometric and appearance information poses a significant challenge. To address these challenges, MGGTalk explores depth information to enhance geometric and facial symmetry characteristics to supplement both geometric and appearance features. Initially, based on the pixel-wise geometric information obtained from depth estimation, we incorporate symmetry operations and point cloud filtering techniques to ensure a complete and precise position parameter for 3DGS. Subsequently, we adopt a two-stage strategy with symmetric priors for predicting the remaining 3DGS parameters. We begin by predicting Gaussian parameters for the visible facial regions of the source image. These parameters are subsequently utilized to improve the prediction of Gaussian parameters for the non-visible regions. Extensive experiments demonstrate that MGGTalk surpasses previous state-of-the-art methods, achieving superior performance across various metrics. Project page: https://scut-mmpr.github.io/MGGTalk-Homepage/.

Autonomous driving visual question answering (AD-VQA) aims to answer questions related to perception, prediction, and planning based on given driving scene images, heavily relying on the model’s spatial understanding capabilities. Prior works typically express spatial information through textual representations of coordinates, resulting in semantic gaps between visual coordinate representations and textual descriptions. This oversight hinders the accurate transmission of spatial information and increases the expressive burden. To address this, we propose a novel Marker-based Prompt learning framework (MPDrive), which represents spatial coordinates by concise visual markers, ensuring linguistic expressive consistency and enhancing the accuracy of both visual perception and spatial expression in AD-VQA. Specifically, we create marker images by employing a detection expert to overlay object regions with numerical labels, converting complex textual coordinate generation into straightforward text-based visual marker predictions. Moreover, we fuse original and marker images as scene-level features and integrate them with detection priors to derive instance-level features. By combining these features, we construct dual-granularity visual prompts that stimulate the LLM’s spatial perception capabilities. Extensive experiments on the DriveLM and CODA-LM datasets show that MPDrive achieves state-of-the-art performance, particularly in cases requiring sophisticated spatial understanding.

Continual Anomaly Detection (CAD) enables anomaly detection models in learning new classes while preserving knowledge of historical classes. CAD faces two key challenges: catastrophic forgetting and segmentation of small anomalous regions. Existing CAD methods store image distributions or patch features to mitigate catastrophic forgetting, but they fail to preserve pixel-level detailed features for accurate segmentation. To overcome this limitation, we propose ReplayCAD, a novel diffusion-driven generative replay framework that replay high-quality historical data, thus effectively preserving pixel-level detailed features. Specifically, we compress historical data by searching for a class semantic embedding in the conditional space of the pre-trained diffusion model, which can guide the model to replay data with fine-grained pixel details, thus improving the segmentation performance. However, relying solely on semantic features results in limited spatial diversity. Hence, we further use spatial features to guide data compression, achieving precise control of sample space, thereby generating more diverse data. Our method achieves state-of-the-art performance in both classification and segmentation, with notable improvements in segmentation: 11.5% on VisA and 8.1% on MVTec. Our source code is available at https://github.com/HULEI7/ReplayCAD.

Document Large Vision Language Models excel in document-centric tasks and have become a key focus of research. Existing frameworks embed features from a lightweight, document-specific encoder into the first layer of a general-purpose Vision Language Model (VLM). However, this introduces a feature mismatch problem. VLMs typically consist of many stacked layers, with the feature hierarchy becoming increasingly abstract at higher layers. Specifically, the first-layer feature in a VLM is token-level, whereas the feature from the encoder is task-level, resulting in a mismatch. Consequently, it is crucial to identify an optimal layer within the VLM for embedding the encoder's features. Inspired by physics, we reformulate the search for the optimal embedding as a problem of finding the shortest time curve. Leveraging the properties of the shortest time curve, we theoretically derive a task-agnostic proxy score that requires only partial training and propose our searching framework, Brac4VLM. Our theoretical derivation shows that Brac4VLM reduces search time by 97.8% compared to brute-force methods. Experimental results further demonstrate that Brac4VLM identifies embedding points that closely align with the true optima. Moreover, the DocVLM with the optimal embedding position identified achieves state-of-the-art performance across various document-centric tasks. Codes: https://github.com/MaxKinny/Brac4VLM.

In this paper, we explore an important yet previously neglected question: Do context aggregation patterns across Language Models (LMs) share commonalities? While some works have investigated context aggregation or attention weights in LMs, they typically focus on individual models or attention heads, lacking a systematic analysis across multiple LMs to explore their commonalities. In contrast, we focus on the commonalities among LMs, which can deepen our understanding of LMs and even facilitate cross-model knowledge transfer. In this work, we introduce the Order-Level Attention (OLA) derived from the order-wise decomposition of Attention Rollout and reveal that the OLA at the same order across LMs exhibits significant similarities. Furthermore, we discover an implicit mapping between OLA and syntactic knowledge. Based on these two findings, we propose the Transferable OLA Adapter (TOA), a training-free cross-LM adapter transfer method. Specifically, we treat the OLA as a unified syntactic feature representation and train an adapter that takes OLA as input. Due to the similarities in OLA across LMs, the adapter generalizes to unseen LMs without requiring any parameter updates. Extensive experiments demonstrate that TOA's cross-LM generalization effectively enhances the performance of unseen LMs. Code is available at \url{https://github.com/jinglin-liang/OLAS}.

Despite notable advancements across various tasks, deep sequence recognition models are shown to grapple with the dilemma of over-confidence, leading to unreliable predicted confidence, necessitating the need for calibration. Current efforts predominantly focus on classification model calibration, leaving the sequence recognition model calibration analysis underexplored and challenging. In this work, we discover that the primary reason for over-confidence in sequence recognition models stems from the one-hot encoding target sequence training paradigm and identify two distinct manifestations of over-confidence: perception and semantic context over-confidence. To address these challenges, we propose a heterogeneous correlation aware sequence regularization (HCSR) method that adaptively incorporates correlated sequences into training alongside the target sequence as additional supervision to regularize the probability of the target sequence from arbitrarily escalating. Specifically, a correlated sequence mining (CSM) model is designed, capable of efficiently mining heterogeneous correlated sequences, which can be flexibly customized to search for specific types of correlated sequences in demand to facilitate the calibration of corresponding types of over-confidence in the calibrating model, thereby achieving fine-grained calibration. Meanwhile, an adaptive calibration module is introduced to adaptively coordinate the optimization weights between the target sequence and correlated sequences, enabling the co-calibration among different samples. Comprehensive experiments conducted on several widely employed sequence recognition tasks demonstrate that the proposed method outperforms the current competing methods by a substantial margin.

Maintaining stylistic consistency is crucial for the cohesion and aesthetic appeal of images, a fundamental requirement in effective image editing and inpainting. However, existing methods primarily focus on the semantic control of generated content, often neglecting the critical task of preserving this consistency. In this work, we introduce the Neural Scene Designer (NSD), a novel framework that enables photo-realistic manipulation of user-specified scene regions while ensuring both semantic alignment with user intent and stylistic consistency with the surrounding environment. NSD leverages an advanced diffusion model, incorporating two parallel cross-attention mechanisms that separately process text and style information to achieve the dual objectives of semantic control and style consistency. To capture fine-grained style representations, we propose the Progressive Self-style Representational Learning (PSRL) module. This module is predicated on the intuitive premise that different regions within a single image share a consistent style, whereas regions from different images exhibit distinct styles. The PSRL module employs a style contrastive loss that encourages high similarity between representations from the same image while enforcing dissimilarity between those from different images. Furthermore, to address the lack of standardized evaluation protocols for this task, we establish a comprehensive benchmark. This benchmark includes competing algorithms, dedicated style-related metrics, and diverse datasets and settings to facilitate fair comparisons. Extensive experiments conducted on our benchmark demonstrate the effectiveness of the proposed framework.

Audio-driven talking head synthesis aims to generate lifelike facial animations synchronized with audio. Current approaches primarily focus on lip motion information in 2D visual space for lip-audio synchronization and expressive lip dynamic, often neglecting 3D geometric motion representations of the lips that can more accurately capture lip movements in real-world scenarios. This oversight can result in suboptimal lip dynamic authenticity. In this work, we introduce a novel 3D Temporal Representation Learning (3D-TRL) algorithm that models 3D lip temporal information as latent representations and utilizes these representations as additional supervision to enhance dynamic authenticity. To achieve this, we leverage the geometric mesh constructed from the 3D Morphable Model (3DMM) as our 3D information of the lip and explore two self-supervised strategies to learn temporal representation in 3D geometric space. First, we propose a Reconstruction-Oriented 3D-TRL algorithm that reconstructs the input to obtain motion tokens in hidden space, encapsulating content while capturing richer contextual representations of the sequence. Second, we develop a Contrastive-Based 3D-TRL algorithm that utilizes contrastive learning to extract hidden 3D motion representations. This algorithm employs data augmentation strategies appropriate specifically for the 3D temporal sequences of the lips. Extensive experiments demonstrate that our approach, as a versatile and adaptable supervisory, can be integrated into various state-of-the-art network frameworks, leading to substantial enhancements in lip dynamic authenticity.

Document image dewarping aims to reconstruct the flat document image from distorted inputs. Previous methods often use geometric or text-line priors to guide the dewarping process. However, document images contain diverse contents, including figures, tables, or paragraph structures, image dewarping without considering the layout structure may fail to obtain global optimization. This paper proposes an encoder-decoder neural network, called DocTLNet, to achieve document image dewarping, which uses both 3D geometry and layout as constraints to refine the content details. To further enhance the layout details, a layout-aug loss is also proposed to explicitly guides the network to handle the distorted layout boundaries. Qualitative and quantitative experiments were conducted on DocUNet benchmark, and the results indicate that our DocTLNet is superior to related methods.

Text detection is a task of great significance in different scenarios, which has wide applications for downstream tasks such as text recognition and text retrieval. Varieties of text detection methods have been proposed to solve this problem in natural scenes and have achieved good results. However, these methods cannot get satisfied performance in industrial scenes for various interferences caused by the industrial environment like the image noise, background material texture and low contrast. To deal with these challenges, we first propose a contour modeling algorithm based on graph Fourier transform mapping to represent arbitrary shaped text contours. A refined fast Fourier convolutional network module with ability of spectral-domain sensing is also intruduced to enhance text feature and suppress interference. Based on these two components, we construct a novel network to achieve accurate industrial scene text detection. Quantitative evaluations are conducted on benchmark datasets MPSC and IcText, experimental results show that our method obtains the state-of-the-art detection accuracy.

Mesh Denoising has become a popular area of research, many traditional and learning-based methods have been proposed to remove noise from meshes. However, most approaches only focus on denoising meshes with low levels of noise. When the noise is high-frequency, it can be difficult to recover the original shape. In this paper, we present a mesh denoising approach that utilizes graph convolution representations to enhance the understanding of the mesh characteristics. It integrates precisely designed graphs to explore the inherent compositional structure of the mesh. When analyzing meshes under the influence of various noises, we extract information about the original features of the mesh by capturing spatial geometric features through graph convolution calculations. Our method is based on Guided Normal Filtering (GNF) to design a Graphical Representation Module (GRM) and a GCN-Based Normal Prediction Module (NPM). It can adaptively obtain the optimal guided normal vectors for noisy meshes. We have compared and analyzed the various methods to produce state-of-the-art results.

Training machines to synthesize diverse handwritings is an intriguing task. Recently, RNN-based methods have been proposed to generate stylized online Chinese characters. However, these methods mainly focus on capturing a person's overall writing style, neglecting subtle style inconsistencies between characters written by the same person. For example, while a person's handwriting typically exhibits general uniformity (e.g., glyph slant and aspect ratios), there are still small style variations in finer details (e.g., stroke length and curvature) of characters. In light of this, we propose to disentangle the style representations at both writer and character levels from individual handwritings to synthesize realistic stylized online handwritten characters. Specifically, we present the style-disentangled Transformer (SDT), which employs two complementary contrastive objectives to extract the style commonalities of reference samples and capture the detailed style patterns of each sample, respectively. Extensive experiments on various language scripts demonstrate the effectiveness of SDT. Notably, our empirical findings reveal that the two learned style representations provide information at different frequency magnitudes, underscoring the importance of separate style extraction. Our source code is public at: https://github.com/dailenson/SDT.

Deep sequence recognition (DSR) models receive increasing attention due to their superior application to various applications. Most DSR models use merely the target sequences as supervision without considering other related sequences, leading to over-confidence in their predictions. The DSR models trained with label smoothing regularize labels by equally and independently smoothing each token, reallocating a small value to other tokens for mitigating overconfidence. However, they do not consider tokens/sequences correlations that may provide more effective information to regularize training and thus lead to sub-optimal performance. In this work, we find tokens/sequences with high perception and semantic correlations with the target ones contain more correlated and effective information and thus facilitate more effective regularization. To this end, we propose a Perception and Semantic aware Sequence Regularization framework, which explore perceptively and semantically correlated tokens/sequences as regularization. Specifically, we introduce a semantic context-free recognition and a language model to acquire similar sequences with high perceptive similarities and semantic correlation, respectively. Moreover, over-confidence degree varies across samples according to their difficulties. Thus, we further design an adaptive calibration intensity module to compute a difficulty score for each samples to obtain finer-grained regularization. Extensive experiments on canonical sequence recognition tasks, including scene text and speech recognition, demonstrate that our method sets novel state-of-the-art results. Code is available at https://github.com/husterpzh/PSSR.

In a two-round auction, a subset of bidders is selected (probabilistically), according to their bids in the first round, for the second round, where they can increase their bids. We formalize the two-round auction model, restricting the second round to a dominant strategy incentive compatible (DSIC) auction for the selected bidders. It turns out that, however, such two-round auctions are not directly DSIC, even if the probability of each bidder being selected for the second round is monotonic to its first bid, which is surprisingly counter-intuitive. We also illustrate the necessary and sufficient conditions of two-round auctions being DSIC. Besides, we characterize the Nash equilibria for untruthful two-round auctions. One can achieve better revenue performance by setting proper probability for selecting bidders for the second round compared with single-round auctions.

The study of ancient writings has great value for archaeology and philology. Essential forms of material are photographic characters, but manual photographic character recognition is extremely time-consuming and expertise-dependent. Automatic classification is therefore greatly desired. However, the current performance is limited due to the lack of annotated data. Data generation is an inexpensive but useful solution to data scarcity. Nevertheless, the diverse glyph shapes and complex background textures of photographic ancient characters make the generation task difficult, leading to unsatisfactory results of existing methods. To this end, we propose an unsupervised generative adversarial network called AGTGAN in this paper. By explicitly modeling global and local glyph shape styles, followed by a stroke-aware texture transfer and an associate adversarial learning mechanism, our method can generate characters with diverse glyphs and realistic textures. We evaluate our method on photographic ancient character datasets, e.g., OBC306 and CSDD. Our method outperforms other state-of-the-art methods in terms of various metrics and performs much better in terms of the diversity and authenticity of generated samples. With our generated images, experiments on the largest photographic oracle bone character dataset show that our method can achieve a significant increase in classification accuracy, up to 16.34%. The source code is available at https://github.com/Hellomystery/AGTGAN.

Despite the success of deep neural network (DNN) on sequential data (i.e., scene text and speech) recognition, it suffers from the over-confidence problem mainly due to overfitting in training with the cross-entropy loss, which may make the decision-making less reliable. Confidence calibration has been recently proposed as one effective solution to this problem. Nevertheless, the majority of existing confidence calibration methods aims at non-sequential data, which is limited if directly applied to sequential data since the intrinsic contextual dependency in sequences or the class-specific statistical prior is seldom exploited. To the end, we propose a Context-Aware Selective Label Smoothing (CASLS) method for calibrating sequential data. The proposed CASLS fully leverages the contextual dependency in sequences to construct confusion matrices of contextual prediction statistics over different classes. Class-specific error rates are then used to adjust the weights of smoothing strength in order to achieve adaptive calibration. Experimental results on sequence recognition tasks, including scene text recognition and speech recognition, demonstrate that our method can achieve the state-of-the-art performance.

The oracle bone script from ancient China is among the world's most famous ancient writing systems. Identifying and deciphering oracle bone scripts is one of the most important topics in oracle bone study and requires a deep familiarity with the culture of ancient China. This task remains very challenging for two reasons. The first is that it is executed mainly by humans and requires a high level of experience, aptitude, and commitment. The second is due to the scarcity of domain-specific data, which hinders the advancement of automatic recognition research. A collection of well-labeled oracle-bone data is necessary to bridge the oracle bone and information processing fields; however, such a dataset has not yet been presented. Hence, in this paper, we construct a new large-scale dataset of oracle bone characters called OBC306. We also present the standard deep convolutional neural network-based evaluation for this dataset to serve as a benchmark. Through statistical and visual analyses, we describe the inherent difficulties of oracle bone recognition and propose future challenges for and extensions of oracle bone study using information processing. This dataset contains more than 300,000 character-level samples cropped from oracle-bone rubbings or images. It covers 306 glyph classes and is the largest existing raw oracle-bone character set, to the best of our knowledge. It is anticipated the publication of this dataset will facilitate the development of oracle bone research and lead to optimal algorithmic solutions.

In this work, we present a new conceptually simple and flexible network for accurate scene text extraction, which handle text detection and text recognition concurrently. Apart from solving feature sharing by designing a unified network, we implement end-to-end training of the whole system by a set of new optimization strategies. More importantly, this method highlights its novel parallel detection-recognition structure, which constructs a loose connection between both detection and recognition. This loose connection is embodied in the definition of overall loss and derivative back propagation for the model parameter updating, which automatically balances the contribution of the two branches to the system performance. It is different from the existing end-to-end methods where two subtasks are connected serially and thus yielding heavy dependence of the predecessor text detection task on the follow-up text recognition task and sensitivity of recognition to detection noise. In addition, a simple Mask-Rectifier mechanism is applied to easily adapt our system to incidental text recognition with arbitrary orientation and shape. Experiment results on Incidental Scene Text ICDAR2015 dataset surpass the current state-of-the-art FOTS method, as suggests the effectiveness of the proposed approach.

Scene text detection has been attracting increasing interests in recent years and a rich body of approaches has been proposed. These previous works of detecting scene text have been dominated by region proposals based approaches, which always generate too many text candidates relative to the number of ground truth bounding boxes. Only a few of those candidates are output as true predictions, and most of the other is fruitlessly involved in regression or classification predictions that consume a great amount of time and storage. Thus emerges the problem of low efficiency of generating text candidates. To address the issue, we propose a method for focusing on scene text gradually guided by an active model. The model allows an agent to take the whole image as the only region proposal in each episode when locating text and therefore significantly reduces the region proposals needed. The agent is trained by deep reinforcement strategy to learn how to estimate future returns of given states and sequentially make decisions to find scene text. Considering the characteristics of scene text, we additionally propose a flexible action scheme and a new reward scheme together with lazy punishment. The experiments on the ICDAR 2013 dataset shows that the proposed method achieve a promising performance while using region proposals as few as the ground truth bounding boxes.

In this paper, we develop a new approach called DeepText for text region proposal generation and text detection in natural images via a fully convolutional neural network (CNN). First, we propose the novel inception region proposal network (Inception-RPN), which slides an inception network with multi-scale windows over the top of convolutional feature maps and associates a set of text characteristic prior bounding boxes with each sliding position to generate high recall word region proposals. Next, we present a powerful text detection network that embeds ambiguous text category (ATC) information and multi-level region-of-interest pooling (MLRP) for text and non-text classification and accurate localization refinement. Our approach achieves an F-measure of 0.83 and 0.85 on the ICDAR 2011 and 2013 robust text detection benchmarks, outperforming previous state-of-the-art results.

Recent studies have demonstrated that Biologically Inspired Model features (BIM) are effective for object or scene categorization. However, according to BIM's forming mechanism, it may not hit the typical pattern due to its blind feature selection. In order to provide more informative pattern information for different visual object classes, a large number of prototypes have to be used in describing images. This leads to huge redundancy which may decrease categorization accuracy. Thus, improving BIM feature's performance by just increasing the number of prototypes is not adequate. In this paper, we propose an integrated approach to address this problem. In our approach, some simple non-biological features such as color histogram and Edge Orientation Histogram (EOH) are incorporated into BIM for discriminative image representation. Experimental results have shown that combination of BIM and simple features can improve visual categorization performances significantly.

Classifying natural scenes into semantic categories has always been a challenging task. So far, many works in this field are primarily intended for single label classification, where each scene example is represented as a single instance vector. The multi-instance multi-label (MIML) learning framework proposed by Z.H. Zhou et al. provides a new solution to the problem of scene classification in a different way. In this paper, we propose a novel scene classification method based on pLSA-based semantic bag generator and MIML learning framework. Under the framework of MIML learning, we introduce the mechanism that transfers an image into a set of instances through the pLSA-based bag generator. Experiments show that our approach achieves better classification performance comparing with the previous work.