论文标题:文本过滤关键技术研究
Research on Negotiation and Related Technology in Electronic Commerce
论文作者 夏迎炬
论文导师 吴立德,论文学位 博士,论文专业 计算机软件与理论
论文单位 复旦大学,点击次数 122,论文页数 103页File Size4113k
2003-04-18论文网 http://www.lw23.com/lunwen_49798772/ 自适应文本过滤;文本特征抽取;向量空间模型;机器学习;自然语言处理
Adaptive Text Filtering,Text Feature Extraction,Vector Space Model,Machine Learning,Natural Language Processing
随着互联网的发展和存贮技术的提高，计算机可读的文本信息也越来越多。而对于特定的用户而言，所需要的信息往往只占其中极小的一部分。大规模的可用网络资源和特定用户对特定的信息的需求使得大规模文本信息处理软件成为信息用户的迫切需求。而要从大规模的网络信息中抽取有用的信息资源，对信息处理的智能性、速度和精度都将提出极为严格的要求。文本过滤所研究的内容就是如何准确地表达用户需求，进而在大规模的信息流中自动地筛选出满足用户需求的信息，使人们更有效地利用信息资源。 论文的工作主要集中在文本过滤尤其是自适应文本过滤中的关键技术上，设计并实现了大规模文本过滤的实验平台，参加了两届国际文本检索会议(TREC10、TREC11)并取得了好成绩。以此为基础，我们实现了多个实用系统，包括中文文本过滤系统、因特网话题信息检索系统等均达到了非常好的效果。 在文本表示方面，我们用向量空间模型来表示文本，以词汇、概念、术语等作为向量空间模型的特征项，并根据文本中的统计信息来计算项的权重。我们尝试将WordNet应用到英文过滤系统中，使用其中的语义信息，并对消歧进行了研究；对中文系统，我们设计开发了HowNet接口并首次将其应用到中文过滤系统中，在中文过滤系统中使用HowNet概念信息使得向量维数得到大幅削减并提高了系统性能。 自适应文本过滤中的机器学习方法包括模板学习和阈值学习。我们对自适应文本过滤中的阈值调整进行了深入的研究，并在TREC10中提出了一种全新的阈值调整学习算法，该算法能使用极少的正例对自适应过滤中阈值进行快速、有效的调整。 在主题描述信息的处理方面，我们在TREC11中针对用户描述信息提出了新颖的使用winnow分类器的算法，对用户描述信息中的词汇构造winnow分类器辅助自适应过滤使得系统性能得到大幅度的提高。
More and more information sources are now available in machine-readable form due to the rapid development of communication networks and inexpensive massive storage. For a special user, the information needed is relatively small. Since most of the data are massive in scale and diverse in subject areas, they make the information acquisition bottleneck more severe, thus greatly challenge the processing systems in speed, precision and robustness. In order to find useful information embedded in electronic form, efficient and effective techniques for large scale real text processing have become one of the most urgent demands.This dissertation focuses on the key techniques of adaptive text filtering. We have designed and developed an experiment platform. Based on that we took part in the Filtering track of Text REtrieval Conference (TREC10 and TREC11) and obtained very good results. In TREC11, We are selected for the first of only three speaking slots on Filtering track. We also have developed several systems, including "Chinese text filtering system" and "Web-Based Trend Search System".Vector Space Model (VSM) is used to represent text. There are two principle problems about VSM: term selection and term weighting. Words, concepts, and terminologies are selected as terms, while term weights are calculated with statistical information and heuristic rules. We have applied the WordNet to filtering system, used the semantic information and tried disambiguation. We have developed the interface of HowNet and applied it to Chinese filtering system by using its conception information. This method has enhanced the system"s performance while lower the dimension of vector greatly.Machine Learning in adaptive text filtering includes profile learning and threshold learning. We make research on threshold learning. In TREC10, we presented a novel threshold-adjusting algorithm. This algorithm adjusts the threshold fast and efficiently using a small number of samples.In TREC11, we presented a novel method that uses a winnow classifier building from these words to assist the text filtering system. This method can enhance the system"s performance greatly.
|
