Reducing false positives in data validation using statistical heuristics

摘要

To validate data, a plurality of strings that match a predetermined regular expression is extracted from the data. A validated subset of the strings is identified. To determine whether the validated subset has been falsely validated, it is determined whether the validated subset satisfies each of one or more predetermined criteria relative to the plurality of strings. In one embodiment, the subset is determined to be falsely validated if at least one of the criteria is satisfied. In another embodiment, the subset is determined to be falsely validated if all of the criteria are satisfied. The data are released only if the subset is determined to be falsely validated.

主权项

1. A method of validating data comprising the steps of:
(a) extracting from the data a plurality of strings that match a predetermined regular expression; (b) identifying a validated subset of said strings; (c) calculating a known probability for a statistical measure of the data based on said predetermined regular expression; (d) comparing said statistical measure of said validated subset to said known probability for said statistical measure; and (e) determining, based on said comparing, if said validated subset meets at least one predetermined criterion relative to all of said plurality of strings taken collectively.