Author(s)

Ibrahim Moukouop Nguena¹, Amolo-Makama Ophélie Carmen Richeline²

¹National Advanced School of Engineering, University of Yaounde I, Yaounde, Cameroon.
²Computer Sciences Department, University of Yaounde I, Yaounde, Cameroon.

Semantic duplicates in databases represent today an important data quality challenge which leads to bad decisions. In large databases, we sometimes find ourselves with tens of thousands of duplicates, which necessitates an automatic deduplication. For this, it is necessary to detect duplicates, with a fairly reliable method to find as many duplicates as possible and powerful enough to run in a reasonable time. This paper proposes and compares on real data effective duplicates detection methods for automatic deduplication of files based on names, working with French texts or English texts, and the names of people or places, in Africa or in the West. After conducting a more complete classification of semantic duplicates than the usual classifications, we introduce several methods for detecting duplicates whose average complexity observed is less than O(2n). Through a simple model, we highlight a global efficacy rate, combining precision and recall. We propose a new metric distance between records, as well as rules for automatic duplicate detection. Analyses made on a database containing real data for an administration in Central Africa, and on a known standard database containing names of restaurants in the USA, have shown better results than those of known methods, with a lesser complexity.

Semantic Duplicate, Detection Technique, Detection Capability, Automatic Deduplication, Detection Rates and Error Rates

Nguena, I. and Richeline, A. (2017) Fast Semantic Duplicate Detection Techniques in Databases. Journal of Software Engineering and Applications, 10, 529-545. doi: 10.4236/jsea.2017.106029.