DYNAMIC OUTLIER BIAS REDUCTION SYSTEM AND METHOD

摘要

A system and method is described herein for data filtering to reduce functional, and trend line outlier bias. Outliers are removed from the data set through an objective statistical method. Bias is determined based on absolute, relative error, or both. Error values are computed from the data, model coefficients, or trend line calculations. Outlier data records are removed when the error values are greater than or equal to the user-supplied criteria. For optimization methods or other iterative calculations, the removed data are re-applied each iteration to the model computing new results. Using model values for the complete dataset, new error values are computed and the outlier bias reduction procedure is re-applied. Overall error is minimized for model coefficients and outlier removed data in an iterative fashion until user defined error improvement limits are reached. The filtered data may be used for validation, outlier bias reduction and data quality operations.

主权项

1. A computer implemented method comprising the steps of:
electronically receiving, by a specially programmed computing system, at least one error threshold criteria and a data set; performing, by the specially programmed computing system, a first iteration of outlier bias reduction using a model that comprises at least one coefficient, wherein performing the first iteration of outlier bias reduction comprises the steps of:
determining a set of predicted values by applying the model to the data set;comparing the set of predicted values to the data set to produce at least one set of error values;removing one or more data values from the data set as data outliers to form an outlier filtered data set, wherein the data outliers are determined from the at least one set of error values and the at least one error threshold criteria; andconstructing an updated model comprising at least one updated coefficient using the outlier filtered data set; and performing, by the specially programmed computing system, a second iteration of outlier bias reduction when at least one termination criteria is not satisfied, wherein performing the second iteration of outlier bias reduction comprises determining a set of second predicted values by applying the updated model to the data set.