Design and Implementation of a New Chinese Word Segmentation Dictionary for the Personalized Mobile Search

Zhongmin Wang, Jingna Qi, Yan He

School of Computer Science and Technology, Xi’an University of Posts and Telecommunications, Xi’an, China.

DOI: 10.4236/cn.2013.51B019                

Abstract

Chinese word segmentation is the basis of natural language processing. The dictionary mechanism significantly influences the efficiency of word segmentation and the understanding of the user’s intention which is implied in the user’s query. As the traditional dictionary mechanisms can't meet the present situation of personalized mobile search, this paper presents a new dictionary mechanism which contains the word classification information. This paper, furthermore, puts forward an approach for improving the traditional word bank structure, and proposes an improved FMM segmentation algorithm. The results show that the new dictionary mechanism has made a significant increase on the query efficiency and met the user’s individual requirements better.

Keywords

Chinese Word Segmentation; Dictionary Mechanism; Natural Language Processing; Personalized Search; Word Classification Information

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	M.-S. Sun and Z.-P. Zuo, “An Experimental Study on Dictionary Mechanism for Chinese Word Segmentation,” Journal of Chinese Information Processing, Vol. 1, 2000, pp. 1-6.
[2]	W. Yang, L.-Y. Ren and R. Tang, “A Dictionary Mechanism for Chinese Word Segmentation Based on the Finite Automata,” 2010 International Conference on Asian Language Processing (IALP), pp. 39-42.
[3]	Z. X. Li, Z. P. Xu, W. Q. Tang and R. X. Tang, “Ambiguity Processing in Word Segmenting,” Computer Engineering and Applications, Vol. 38, No. 11, 2002, pp. 106-109.
[4]	Q. Y. Zhang and S. Chai, “Chinese Word Segmentation Dictionary using Two-level Index,” Computer Engineering and Applications, Vol. 19, 2009.
[5]	Q. H. Li, Y. J. Chen and J. G. Sun, “A New Dictionary Mechanism for Chinese Word Segmentation,” Journal of Chinese Information Processing, Vol. 17, 2003, pp. 13-18.
[6]	Y. Niu and L. L. Li, “An Improved Chinese Segmentation Algorithm Based on New Dictionary Construction,” International Conference on Computational Science and Engineering, Vol. 2, 2009, pp. 993-996.
[7]	A. Choi, C. H. Cheng and Y. L. Ko, “Word Extraction from Chinese Documents by Occurrence Counts,” 1988 International Conference on Computer Processing of Chinese and Oriental Languages, Toronto, Canada, pp. 488-491.
[8]	H. Y. Cui, “Research 0n an Improved Chinese Segmentation Algorithm based on Word Frequency Statistic,” Information Technology, Vol. 04, 2008.