USER FEEDBACK IN SEMI-AUTOMATIC QUESTION ANSWERING SYSTEMS

摘要

A system applies rules to a set of documents to generate codes, such as billing codes for use in medical billing. A human operator provides input specifying whether the generated codes are correct. Based on the input from the human operator, the system attempts to identify which clause(s) in the rules which were relied on to generate the particular code are correct and which such clause(s) are incorrect. The system then assigns praise to components of the system responsible for generating codes in the correct clauses, and assigns blame to components of the system responsible for generating codes in the incorrect clauses. Such blame and praise may then be used to determine whether particular code-generating components are insufficiently reliable. The system may disable, or take other remedial action in response to, insufficiently reliable code-generating components.