Chinese spell checkers are more difficult to develop because of two language features: 1) there are no word boundaries, and a character may function as a word or a word morpheme; and 2) the Chinese character set contains more than ten thousand characters. The former makes it difficult for a spell checker to detect spelling errors, and the latter makes it difficult for a spell checker to construct error models. We develop a word lattice decoding model for a Chinese spell checker that addresses these difficulties. The model performs word segmentation and error correction simultaneously, thereby solving the word boundary problem. The model corrects nonword errors as well as real-word errors. In order to better estimate the error distribution of large character sets for error models, we also propose a methodology to extract spelling error samples automatically from the Google web 1T corpus. Due to the large quantity of data in the Google web 1T corpus, many spelling error samples can be extracted, better reflecting spelling error distributions in the real world. Finally, in order to improve the spell checker for real applications, we produce n-best suggestions for spelling error corrections. We test our proposed approach with the Bakeoff 2013 CSC Datasets; the results show that the proposed methods with the error model significantly outperform the performance of Chinese spell checkers that do not use error models.

The syntactic ambiguity of a transitive verb (Vt) followed by a noun (N) has long been a problem in Chinese parsing. In this paper, we propose a classifier to resolve the ambiguity of Vt-N structures. The design of the classifier is based on three important guidelines, namely, adopting linguistically motivated features, using all available resources, and easy in-tegration into a parsing model. The lin-guistically motivated features include semantic relations, context, and morpho-logical structures; and the available re-sources are treebank, thesaurus, affix da-tabase, and large corpora. We also pro-pose two learning approaches that resolve the problem of data sparseness by auto-parsing and extracting relative knowledge from large-scale unlabeled data. Our experiment results show that the Vt-N classifier outperforms the cur-rent PCFG parser. Furthermore, it can be easily and effectively integrated into the PCFG parser and general statistical pars-ing models. Evaluation of the learning approaches indicates that world knowledge facilitates Vt-N disambigua-tion through data selection and error cor-rection. 1

introduction Share on Introduction to the Special Issue on Chinese Language Processing Authors: Keh-Jiann Chen Institute of Information Science, Academia Sinica Institute of Information Science, Academia SinicaView Profile , Qun Liu Institute of Computing Technology, Chinese Academy of Sciences Institute of Computing Technology, Chinese Academy of SciencesView Profile , Nianwen Xue Brandeis University Brandeis UniversityView Profile , Le Sun Institute of Software, Chinese Academy of Sciences Institute of Software, Chinese Academy of SciencesView Profile Authors Info & Claims ACM Transactions on Asian Language Information ProcessingVolume 10Issue 3September 2011 Article No.: 11pp 1–3https://doi.org/10.1145/2002980.2002981Published:01 September 2011Publication History 0citation284DownloadsMetricsTotal Citations0Total Downloads284Last 12 Months3Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

In order to pursue high performance of Chinese information access on the Internet, this paper presents an attractive approach with a successful integration of efficient speech recognition and information retrieval techniques. A working system based on the proposed approach for speech retrieval of real-time Chinese netnews services has been implemented and tested. Very exciting performance has been achieved.

This paper presents a new Chinese text-to-speech system that produces very natural and intelligible synthetic Mandarin speech based on part-of-speech analysis, prosodic modeling and non-uniform units. The distinguishing features and key technology for the system can be summarized as follows. (1) A text analysis module for word identification and tagging was developed based on part-of-speech modeling and using heuristic rules to achieve very high accuracy. (2) The required prosodic parameters for the synthetic speech are derived from a two-stage procedure. The prosodic structures of the input texts are first derived from a statistical model trained by a large speech database, and the prosodic parameters are then determined according to the structures. (3) A specially designed speech segments inventory constructed with non-uniform and pitch dependent units is used to improve the fluency and intelligibility of the system.

To retrieve a Chinese word from a Chinese dictionary, it needs the user to know exactly the first character of the desired word. Because there is more than 10,000 Chinese characters, this makes the Chinese dictionary relatively difficult to be used. To reduce the problem, this paper presents intelligent retrieval techniques for very large Chinese dictionaries with speech queries. The proposed techniques properly integrate the technologies of Mandarin speech recognition and Chinese information retrieval with a syllable-based approach utilizing the mono-syllabic structure of the language. Moreover, it is very nice to provide the function of retrieving all relevant word entries from the dictionaries using speech queries describing “general concepts” of the desired words. To achieve the challenging function, the techniques of relevance feedback are also included. Based on these techniques, a retrieval system was implemented successfully on a Pentium PC for a very large Chinese dictionary which includes 160,000 word entries and the total length of the lexical information under the word entries exceeds 20,000,000 words.

This paper presents an accurate and efficient voice retrieval system for very-large-vocabulary Chinese textual databases with a specially-designed clustered language model. To reduce the problems resulted from the complexity of unconstrained speech-input queries for retrieval, the system is completely syllable-based in both speech recognition and database retrieval by properly utilizing the mono-syllabic structure of Chinese language. In addition, it partitions the records in the database into clusters and trains the clustered language model using the clustering results. The proposed clustered language model with its augmented search algorithm are very useful to improve accuracy and speed of the speech retrieval system. In the preliminary tests using an experimental database with about 30,000 bibliographical records, it was found that the present system can accept unconstrained speech-input queries and achieve very good performance.

This paper presents a prototype prosodic-segment-based Mandarin dictation machine for the Chinese language with very large vocabulary. It accepts utterances continuous within a prosodic segment which is composed of one or a few word(s). It also possesses various on-line learning capabilities for fast adaptation to a new user in acoustic, lexical and linguistic levels. The overall system is implemented on an IBM/PC with an additional DSP card including a Motorola DSP 96002 chip. The word accuracy can achieve nearly 90% for a new user after he produces about 10 minutes of speech to train the system, and the accuracy can be further improved with the on-line learning functions.

A language processing model is proposed in which the grammatical approach of unification grammar and the statistical approach of Markov language models are properly integrated in a word lattice chart parsing algorithm with different best-first parsing strategies. This model has been successfully implemented in experiments on Mandarin speech recognition although it is language-independent. Test results show that significant improvements in both correct rate of recognition and computation speed can be achieved. A correct rate of 93.8% and 5 s per sentence on an IBM PC/AT, as compared with 73.8% and 25 s using unification grammar alone and 82.2% and 3 s using a Markov language model alone, was achieved. This high performance is due to the effective rejection of noisy word hypothesis interferences; that is, the unification-based grammatical analysis eliminates all illegal combinations, while the Markovian probabilities of constituents combined with the considerations on constituent length indicate the correct direction of processing.>

Suppose LC 1 and LC 2 are two machine learning classes each based on a criterion of success. Suppose, for every machine which learns a class of functions according to the LC 2 criterion of success, there is a machine which learns this class according to the LC 2 criterion. In the case where the converse does not hold LC, is said to be separated from LC 2. It is shown that for many such separated learning classes from the literature a much stronger separation holds: (∀𝒞∈LC 1) (∃𝒞' ∈LC 2 - LC 1(( [' ⊃𝒞] It is also shown that there is a pair of separated learning classes from the literature for which the stronger separation above does not hold. A philosophical heuristic toward the design of artificially intelligent learning programs is presented with each strong separation result.

A language processor is to find out a most promising sentence hypothesis for a given word lattice obtained from acoustic signal recognition. In this paper a new language processor is proposed, in which unification grammar and Markov language model are integrated in a word lattice parsing algorithm based on an augmented chart, and the island-driven parsing concept is combined with various preference-first parsing strategies defined by different construction principles and decision rules. Test results show that significant improvements in both correct rate of recognition and computation speed can be achieved.

An efficient algorithm is developed to handle the difficulties in parsing noise word lattices (sets of word hypotheses obtained in continuous-speech recognition) which include problems such as word boundary overlapping, homonyms, lexical ambiguities, recognition uncertainty and errors, etc. An augmented chart is proposed, and the algorithms is then derived on this chart. This algorithm properly integrates the global structural synthesis capabilities of the unification grammar and the local relation estimation capabilities of the Markov language model. The parsing algorithm is island driven and best first. In this way, the features of the grammatical and statistical approaches can be combined, and the effects of the two different approaches are reflected in a single algorithm such that the overall selectivity can be appropriately optimized.>

An experimental Mandarin dictation machine for inputting Mandarin speech (spoken Chinese language) into computers is described. Because of the special characteristics of the Chinese language, syllables are chosen as the basic units for dictation. The machine is designed based on a hierarchical language recognition approach in which acoustic signals are first recognized as a sequence of syllables, possible word hypotheses are then formed from the syllables, and the complete sentences are finally obtained. This approach is implemented by two subsystems. The first recognizes the syllables using speech signal processing techniques, the second subsystem then identifies the exact characters from the syllable and corrects the errors in syllable recognition. The detailed syllable recognition algorithms, word formation rules, parser, grammar, and the syntactic checking algorithms are described. With newspaper text in the form of isolated syllables as input, the preliminary test results indicate that such a dictation machine is not only practically attractive, but technically feasible.>

The target of this recognition system is the set of handwritten Chinese characters input from tablet devices with stroke-sequence and stroke-count being free but within the constraint of normal writing. A formalism based upon an initial stroke-sequence decision tree and position matching has been developed for recognizing handwritten Chinese characters. This formalism has the advantages of using the features of strokes, stroke-sequence, and geometric relations but avoids the disadvantages caused by the instability of all of the above features. With extensive training, it can be proven that this formalism may provide a very promising result even in handling erroneous writing such as missing a stroke, wrong writing sequence etc.