Abstract New statistical formulas were developed for identifying two‐ and three‐character words in Chinese text. The formulas were constructed by performing stepwise logistic regression using a sample of sentences that had been manually segmented. For identifying two‐character words, the relative frequency of the adjacent characters and the document frequency of the overlapping bigrams were found to be significant factors. These provide information about the immediate neighborhood or context of the character string. Contextual information was also found to be significant in predicting three‐character words. Local information (the number of times the bigram or trigram occurs in the document being segmented) and the position of the bigram/trigram in the sentence were not found to be useful in identifying words. The new formulas, called contextual information formulas, were found to be substantially better than the mutual information formula in identifying two‐ and three‐character words. Using the contextual information formulas for both two‐ and three‐character words gave significantly better results than using the formula for two‐character words alone. The method can also be used for identifying multiword terms in English text.

Article A new statistical formula for Chinese text segmentation incorporating contextual information Share on Authors: Yubin Dai Division of Information Studies, School of Applied Science, Nanyang Technological University, Singapore 639798 Division of Information Studies, School of Applied Science, Nanyang Technological University, Singapore 639798View Profile , Teck Ee Loh 10 Kent Ridge Crescent, Data Storage Institute, Singapore 119260 10 Kent Ridge Crescent, Data Storage Institute, Singapore 119260View Profile , Christopher S. G. Khoo Division of Information Studies, School of Applied Science, Nanyang Technological University, Singapore 639798 Division of Information Studies, School of Applied Science, Nanyang Technological University, Singapore 639798View Profile Authors Info & Claims SIGIR '99: Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrievalAugust 1999 Pages 82–89https://doi.org/10.1145/312624.312659Online:01 August 1999Publication History 17citation864DownloadsMetricsTotal Citations17Total Downloads864Last 12 Months10Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access