SYSTEM FOR INTERACTIVELY VISUALIZING AND EVALUATING USER BEHAVIOR AND OUTPUT

摘要

The present invention discloses CrowdScape, a system that supports the human evaluation of complex crowd work through interactive visualization and mixed initiative machine learning. The system combines information about worker behavior with worker outputs and aggregate worker behavioral traces to allow the isolation of target worker clusters. This approach allows users to develop and test their mental models of tasks and worker behaviors, and then ground those models in worker outputs and majority or gold standard verifications.