In the field of scene text spotting, previous OCR methods primarily relied on image encoders and pre-trained text information, but they often overlooked the advantages of incorporating human language instructions. To address this gap, we propose InstructOCR, an innovative instruction-based scene text spotting model that leverages human language instructions to enhance the understanding of text within images. Our framework employs both text and image encoders during training and inference, along with instructions meticulously designed based on text attributes. This approach enables the model to interpret text more accurately and flexibly. Extensive experiments demonstrate the effectiveness of our model and we achieve state-of-the-art results on widely used benchmarks. Furthermore, the proposed framework can be seamlessly applied to scene text VQA tasks. By leveraging instruction strategies during pre-training, the performance on downstream VQA tasks can be significantly improved, with a 2.6% increase on the TextVQA dataset and a 2.1% increase on the ST-VQA dataset. These experimental results provide insights into the benefits of incorporating human language instructions for OCR-related tasks.

Multi-modal Large Language Models (MLLMs) have introduced a novel dimension to document understanding, i.e., they endow large language models with visual comprehension capabilities; however, how to design a suitable image-text pre-training task for bridging the visual and language modality in document-level MLLMs remains underexplored. In this study, we introduce a novel visuallanguage alignment method that casts the key issue as a Visual Question Answering with Mask generation (VQA-Mask) task, optimizing two tasks simultaneously: VQA-based text parsing and mask generation. The former allows the model to implicitly align images and text at the semantic level. The latter introduces an additional mask generator (discarded during inference) to explicitly ensure alignment between visual texts within images and their corresponding image regions at a spatially-aware level. Together, they can prevent model hallucinations when parsing visual text and effectively promote spatially-aware feature representation learning. To support the proposed VQAMask task, we construct a comprehensive image-mask generation pipeline and provide a large-scale dataset with 6M data (MTMask6M). Subsequently, we demonstrate that introducing the proposed mask generation task yields competitive document-level understanding performance. Leveraging the proposed VQAMask, we introduce Marten, a trainingefficient MLLM tailored for document-level understanding. Extensive experiments show that our Marten consistently achieves significant improvements among 8B-MLLMs in document-centric tasks. Code and datasets are available at https://github.com/PriNing/Marten.

In recent years, text-image joint pre-training techniques have shown promising results in various tasks. However, in Optical Character Recognition (OCR) tasks, aligning text instances with their corresponding text regions in images poses a challenge, as it requires effective alignment between text and OCR-Text (referring to the text in images as OCR-Text to distinguish from the text in natural language) rather than a holistic understanding of the overall image content. In this paper, we propose a new pre-training method called OCR-Text Destylization Modeling (ODM) that transfers Diverse styles of text found in images to a uniform style based on the text prompt. With ODM, we achieve better alignment between text and OCR-Text and enable pre-trained models to adapt to the complex and diverse styles of scene text detection and spotting tasks. Additionally, we have designed a new labeling generation method specifically for ODM and combined it with our proposed Text-Controller module to address the challenge of annotation costs in OCR tasks, allowing a larger amount of unlabeled data to participate in pre-training. Extensive experiments on multiple public datasets demonstrate that our method significantly improves performance and outperforms current pre-training methods in scene text detection and spotting tasks. Code is available at ODM.

Abstract Deep neural networks (DNNs) have been applied in many pattern recognition or object detection applications. DNNs generally consist of millions or even billions of parameters. These demanding computational storage and requirements impede deployments of DNNs in resource-limited devices, such as mobile devices, micro-controllers. Simplification techniques such as pruning have commonly been used to slim DNN sizes. Pruning approaches generally quantify the importance of each component such as network weight. Weight values or weight gradients in training are commonly used as the importance metric. Small weights are pruned and large weights are kept. However, small weights are possible to be connected with significant weights which have impact to DNN outputs. DNN accuracy can be degraded significantly after the pruning process. This paper proposes a roulette wheel-like pruning algorithm, in order to simplify a trained DNN while keeping the DNN accuracy. The proposed algorithm generates a branch of pruned DNNs which are generated by a roulette wheel operator. Similar to the roulette wheel selection in genetic algorithms, small weights are more likely to be pruned but they can be kept; large weights are more likely to be kept but they can be pruned. The slimmest DNN with the best accuracy is selected from the branch. The performance of the proposed pruning algorithm is evaluated by two deterministic datasets and four non-deterministic datasets. Experimental results show that the proposed pruning algorithm generates simpler DNNs while DNN accuracy can be kept, compared to several existing pruning approaches.

Speech is a natural user interface for the Internet of Things system. However, the presence of noise affects severely the performance of such system. With the deployment of smart devices with microphones, one can form a powerful acoustic sensor network to enhance the speech via beamforming techniques. On the other hand, reliability of data transmission also determines the beamforming performance, since faulty data will drift the beamformer steering location randomly. Currently, there is no protection scheme for acoustic data transmitted over the wireless network in order to keep steady beamforming performance. In this article, we design a compound distributed beamformer, where nodes are grouped and the system is embedded with blockchain technology to protect the data integrity during transmission. It attempts to provide more possible reliable connections between groups. Simulated experiments show that the distributed beamformer with blockchain protection is able to maintain steady beamforming performance.

This paper proposes a hierarchical decision-making and control algorithm for the shepherd game, the seventh mission in the International Aerial Robotics Competition (IARC). In this game, the agent (a multirotor aerial robot) is required to contact targets (ground vehicles) sequentially and drive them to a certain boundary to earn score. During the game of 10 min, the agent should be fully autonomous without any human interference. Regarding the lower-level controller and dynamics of the agent, each action takes a duration of time to accomplish. Denoted as an action delay, in this paper, this action duration is nonconstant and is related to the final reward. Therefore, the challenging point is making the agent "aware of time" when applying a certain action. We solve this problem by two approaches: deep Q-networks and lookup table. The action delay predictor in the decision-level is fitted by a lower-level controller. Through simulations by the example of the shepherd game, the effectiveness and efficiency of this approach are validated. This paper helps our team winning the first prize in IARC 2017, and keeps the best record of this mission since it was released in 2013.

Hyperspectral remote sensing exploits the fact that all material reflects, absorb, and emit electromagnetic energy, at specific wavelengths, in distinctive patterns related to their molecular composition. Hyperspectral algorithms for the estimation of the concentrations of chlorophyll A and carotenoids can be developed using statistical approaches. Some algorithms for the estimation of the concentrations of chlorophyll A and carotenoids in rice leaves from airborne hyperspectral data were developed in this research. Algorithms based on reflectance band ratios and first derivative have been developed for the estimation of chlorophyll A and carotenoid content of rice leaves by using airborne hyperspectral data acquainted by Pushbroom Hyperspectral Imager (PHI). There was a strong R680/R825 and chlorophyll A relationship with a linear relationship between the ratio of reflectance at 680 nm and 825 nm. The first derivative at 686 nm and 601 nm correlated best with carotenoid. The relationship between the ratio of R680/R825 and chlorophyll A relationship, the first derivative at 686 nm and carotenoid concentration were used to develop predictive regression equations for the estimation of canopy chlorophyll A and carotenoid concentration respectively. The relationship was applied to the imagery and a chlorophyll A concentration map was generated.

The International Organization for Standardization (ISO) defines standards as documented agreements containing technical specifications or other precise criteria to be used consistently as rules, guidelines, or definition characteristics, to ensure that materials, products, procedures, and services are fit for their purpose. Further, standards contribute to making life simpler, and to increasing the reliability and effectiveness of the goods and services we use. The Scientific Database (SDB) of the Chinese Academy of Sciences (CAS) is comprehensive scientific information service system. It has been built by different disciplines of many institutes of CAS in the past ten years. The Chinese Resources and Environment Remote Sensing Database is one of subsidiary database of SDB. The purpose of data standardization for the Chinese Resources and Environment Remote Sensing Database is to facilitate data sharing and increase interoperability among resources and environment remote sensing applications. The primary objective of this proposed standard is to define the content for remote sensing data, thereby providing a solid basis from which to develop interoperable data formats for this common form of remote sensing data.