Predicting computer model accuracy

摘要

A social networking system receives messages from users that include hashtags. The social networking system may use a natural language model to identify terms in the hashtag corresponding to words or phrases of the hashtag. The words or phrases may be used to modify a string of the hashtag. The social networking system may also generate computer models to determine likely membership of a message with various hashtags. Prior to generating the computer models, the social networking system may filter certain hashtags from eligibility for computer modeling, particularly hashtags that are not frequently used or that more typically appear as normal text in a message instead of as a hashtag. The social networking system may also calibrate the computer model outputs by comparing a test message output with outputs of a calibration group that includes positive and negative examples with respect to the computer model output.

主权项

1. A method comprising:
accessing a trained computer model that predicts membership in a group; receiving a set of calibration data items, each calibration data item having a known membership in the group; applying a computer model to each of the calibration data items, the output of the computer model generating a calibration value for each of the calibration data items that represents a predicted probability of membership in the group; receiving a test data item having an unknown membership in the group; applying the computer model to the received test data item to generate a test output value, the test output value reflecting a predictive probability by the computer model of membership in the group; selecting a subset of the calibration data items that have a calibration value within a range of the test output value; and adjusting the test output value based on the membership in the group of the selected subset of calibration data items, wherein the adjusted test output value reflects an adjusted predictive probability for the computer model of membership in the group.