Xie, H., Lu, X., Tang, Z., et al. (2016) Detection of Entity Mixture in Knowledge Bases Using Hierarchical Clustering. Proceedings of the National CCF Conference on Natural Language Processing and Chinese Computing, Kunming, 2-6 December 2016, 288-299.
https://doi.org/10.1007/978-3-319-50496-4_24

has been cited by the following article:

• TITLE: An Improvement of Data Cleaning Method for Grain Big Data Processing Using Task Merging

  AUTHORS: Feiyu Lian, Maixia Fu, Xingang Ju

  KEYWORDS: Grain Big Data, Data Cleaning, Task Merging, Hadoop, MapReduce

  JOURNAL NAME: Journal of Computer and Communications, Vol.8 No.3, March 5, 2020

  ABSTRACT: Data quality has exerted important influence over the application of grain big data, so data cleaning is a necessary and important work. In MapReduce frame, parallel technique is often used to execute data cleaning in high scalability mode, but due to the lack of effective design, there are amounts of computing redundancy in the process of data cleaning, which results in lower performance. In this research, we found that some tasks often are carried out multiple times on same input files, or require same operation results in the process of data cleaning. For this problem, we proposed a new optimization technique that is based on task merge. By merging simple or redundancy computations on same input files, the number of the loop computation in MapReduce can be reduced greatly. The experiment shows, by this means, the overall system runtime is significantly reduced, which proves that the process of data cleaning is optimized. In this paper, we optimized several modules of data cleaning such as entity identification, inconsistent data restoration, and missing value filling. Experimental results show that the proposed method in this paper can increase efficiency for grain big data cleaning.