Collection of machine learning training data for expression recognition

摘要

Apparatus, methods, and articles of manufacture for implementing crowdsourcing pipelines that generate training examples for machine learning expression classifiers. Crowdsourcing providers actively generate images with expressions, according to cues or goals. The cues or goals may be to mimic an expression or appear in a certain way, or to “break” an existing expression recognizer. The images are collected and rated by same or different crowdsourcing providers, and the images that meet a first quality criterion are then vetted by expert(s). The vetted images are then used as positive or negative examples in training machine learning expression classifiers.

主权项

1. A computer-implemented method comprising steps of:
providing one or more cues to a first plurality of untrained providers for mimicking a predetermined expression; receiving from the first plurality of untrained providers images created in response to the step of providing one or more cues; sending requests to rate the images to a second plurality of untrained providers; receiving ratings of the images from the second plurality of untrained providers, in response to the step of sending requests; applying a first quality check to the images rated by the second plurality of untrained providers, the first quality check being based on the ratings of the second plurality of untrained providers, the step of applying the first quality check resulting in one or more images that passed the first quality check; sending the one or more images that passed the first quality check to one or more experts, for rating by the one or more experts; in response to the step of sending the one or more images that passed the first quality check, receiving one or more ratings from the one or more experts; and applying a second quality check to the images rated by the one or more experts, the second quality check being based on the one or more ratings of the one or more experts, the step of applying the second quality check resulting in one or more images that passed the second quality check.