Utilizing failures in question and answer system responses to enhance the accuracy of question and answer systems

摘要

A computerized device for enhancing the accuracy of a question-answer system is disclosed. The computerized device comprises a question-answer system comprising software for performing a plurality of question answering processes. A receiver receives a question into the question-answer system. A processor that generates a plurality of candidate answers to the question is connected to the question-answer system. The processor determines a confidence score for each of the plurality of candidate answers. The processor evaluates sources of evidence used to generate the plurality of candidate answers. The processor identifies missing information from a corpus of data. The missing information comprises any information that improves a confidence score for a candidate answer. The processor generates at least one follow-on inquiry based on the missing information. A network interface outputs the at least one follow-on inquiry to external sources separate from the question-answer system.

主权项

1. A device, comprising:
a question-answer system comprising software for performing question answering processes; a receiver receiving a question into said question-answer system; a processor connected to said question-answer system, said processor:
generating candidate answers to said question,determining a confidence score for each of said candidate answers,evaluating sources of evidence used to generate said candidate answers,identifying missing information from a corpus of data, said missing information comprising any information that improves the ability of said question-answer system to understand and evaluate said sources of evidence associated with said candidate answers, said identifying said missing information comprising:
evaluating a piece of evidence and producing an evidence score according to a scoring process that reflects a degree to which said piece of evidence supports or refutes said candidate answer,classifying said piece of evidence as marginal evidence if said score is below a threshold value, andevaluating said marginal evidence to identify said missing information, andgenerating a follow-on inquiry based on said missing information; and a network interface outputting said follow-on inquiry to external sources separate from said question-answer system to obtain responses to said follow-on inquiry and receiving from said external sources a response to said follow-on inquiry, said processor inputting said response into said corpus of data.