Evaluation of predictions in the absence of a known ground truth

摘要

Disclosed is a novel system, and method to evaluate a prediction of a possibly unknown outcome out of a plurality of predictions of that outcome. The method begins with accessing a particular prediction of an outcome out of a plurality of predictions of that outcome in which the outcome may be unknown. Next, a subsample of the plurality of predictions of the outcome is accessed. The subsample can possibly include the particular prediction. A consensus prediction of the outcome based on the subsample of the plurality of predictions is determined. A proximity of the particular prediction to the consensus prediction is determined. Each prediction is ranked out of the plurality of predictions in an order of a closest in proximity to the consensus prediction to a farthest in proximity to the consensus prediction.

主权项

1. A non-transitory computer program product for evaluating a prediction of an outcome out of a plurality of predictions of that outcome, the computer program product comprising instructions configured to perform a method comprising:
collecting a plurality of clinical data from each patient of a cohort of patients; providing a plurality of research modules, each of which having an input for receiving a clinical data of the plurality of clinical data from each patient of the cohort of patients, and each of which having an output for outputting predictions of an outcome that represents a prediction that a patient has a given disease; inputting each clinical data of the plurality of clinical data into a respective research module; outputting, from the plurality of research modules, a plurality of prediction of outcomes; accessing a particular prediction of an outcome out of a plurality of predictions of that outcome in which the outcome may be unknown; accessing a subsample of the plurality of predictions of the outcome, with the subsample possibly including the particular prediction; determining with a computer, a consensus prediction of the outcome based on the subsample of the plurality of predictions; determining a proximity of the particular prediction to the consensus prediction, wherein the consensus prediction is calculated using a mixture of two unimodal probability distributions; and ranking each prediction out of the plurality of predictions in an order of a closest in proximity to the consensus prediction to a farthest in proximity to the consensus prediction.