Evaluating the ranking quality of a ranked list

摘要

The ranking quality of a ranked list may be evaluated. In an example embodiment, a method is implemented by a system to access log data, ascertain which entries of a ranked list are skipped, and determine a ranking quality metric from the skipped entries. More specifically, log data that reflects user interactions with a ranked list having multiple entries is accessed. The user interactions include at least indications of which of the multiple entries are selected entries. It is ascertained which entries of the multiple entries of the ranked list are skipped entries based on the selected entries. The ranking quality metric for the ranked list is determined responsive to the skipped entries.

主权项

1. One or more processor-accessible storage media comprising processor-executable instructions for evaluating a ranking quality of a ranked list, wherein the processor-executable instructions, when executed, direct a system to perform acts comprising:
accessing log data that reflects user interactions by a user with the ranked list, the ranked list having multiple entries; determining, from the log data, that the multiple entries of the ranked list include viewed entries that have been viewed by the user and unviewed entries that have not been viewed by the user; ascertaining, from the log data, that the viewed entries include skipped viewed entries that the user has skipped and selected viewed entries that the user has selected, wherein each of the skipped viewed entries and each of the selected viewed entries are ranked relatively higher in the ranked list than the unviewed entries and the skipped viewed entries are interspersed with the selected viewed entries in the ranked list; determining a first number of the skipped viewed entries and a second number of the viewed entries including both the skipped viewed entries and the selected viewed entries; determining a ranking quality metric for the ranked list using the first number of the skipped viewed entries and the second number of the viewed entries; and using the ranking quality metric to tune a search application, wherein the skipped viewed entries include partially skipped viewed entries and fully skipped viewed entries, and the ascertaining comprises using dwell times to identify the partially skipped viewed entries.