摘要：本论文详细地介绍了中文分词技术。指出了中文分词技术的意义和重要性。着重介绍了几个常见的中文分词算法，包括TRIE树、DAG、HMM算法等。以及歧义和未登录词两大困难。重点是如何利用常见的中文分词算法设计并实现自己的分词器。包括如何将词典存储为TRIE树结构、如何根据待分词句子生成DAG、以及详细推导了如何使用Viterbi算法对未登录词进行词性标注和切分。然后给出了在Python环境下的具体实现和部分代码。此外，本论文还详细展示了该分词器的分词性能，并且将其与商用分词器进行对比，寻找它们之间性能的差异。
关键词 中文分词 HMM算法 未登录词 Python
毕业设计说明书外文摘要
Title Research on Chinese Word Segmentation
Abstract：This paper introduces the Chinese word segmentation technology in detail. The significance and importance of Chinese word segmentation are pointed out. This paper focuses on a number of common Chinese word segmentation algorithm, including TRIE tree, DAG, HMM algorithm, etc.. And the two miseries of ambiguity and unknown words are pointed out. The focus of this paper is how to design and implement the Chinese word segmentation device using the common Chinese word segmentation algorithm, including how to store the dictionary as the TRIE tree structure, how to generate DAG according to the word to be segmented, and how to use Viterbi algorithm to tag and segment the unknown words in detail. Then it gives the specific implementation and part of the code in the Python environment. In addition, this paper also shows the segmentation performance of the word segmentation and compares it with the commercial segmentation device to find the differences between them.
Keywords Chinese Word Segmentation HMM Algorithm Unknown Words Python
目 次
1 绪论 3
1.1 课题研究背景与意义 3
1.3 章节安排 4
2 中文分词的算法原理 5
2.1 中文分词器的原理和工作流程 5
2.2 中文分词涉及的主要算法 5
2.3 中文分词器面临的困难 6
2.4 本章小结 8
3 中文分词的实现 10
3.1 词典扫描与DAG生成 10
3.2 Viterbi算法识别未登录词 12
3.3 Python实现 21
3.4 本章小结 25
4 测试与分析 26
4.1 分词算法测试 26
4.2 与jieba分词的对比 29
4.3 与THULAC的对比 31
4.4 三种分词器的综合对比 35
4.5 本章小结 37
结论 38
致谢 Python中文分词技术研究:http://www.chuibin.com/tongxin/lunwen_205968.html