| 摘要 |
In an approach to identifying duplicates in data, one or more computer processors receive a request from a user to identify duplicates in a data set. The one or more computer processors retrieve the data set utilizing data discovery. The one or more computer processors perform data profiling on the data set. The one or more computer processors determine one or more domain types of the data set, based, at least in part, on the performed data profiling. The one or more computer processors perform data standardization on the data set, based, at least in part, on the one or more determined domain types. Responsive to performing data standardization, the one or more computer processors perform probabilistic matching on the data set. The one or more computer processors to identify two or more duplicates in the data set, based, at least in part, on the probabilistic matching. |
| 主权项 |
1. A method for identifying duplicates in a data set, the method comprising:
receiving, by one or more computer processors, a request from a user to identify duplicates in a data set; retrieving, by the one or more computer processors, the data set utilizing data discovery; performing, by the one or more computer processors, data profiling on the data set; determining, by the one or more computer processors, one or more domain types of the data set, based, at least in part, on the performed data profiling; performing, by the one or more computer processors, data standardization on the data set, based, at least in part, on the one or more determined domain types; responsive to performing data standardization, performing, by the one or more computer processors, probabilistic matching on the data set; and identifying, by the one or more computer processors, two or more duplicates in the data set, based, at least in part, on the probabilistic matching. |
