System, method and apparatus for computer aided association of relevant images with text

摘要

Methods, systems, and apparatus, including computer program products are disclosed for the association of relevant images with text and presentation of image recommendations to users. In an aspect, text, image and user/usage information is processed in order to extract features therefrom and create three-component feature vectors called triplets. One or more reference databases store the triplets and a modeling component builds a model to learn and recommend images based on the triplets stored. A reference database may be initially populated with information from publically available image/text for use by the modeling component. Using the model, an Illustration Index is calculated for each image in a collection for a given text. Images are ranked by their Illustration Index and provided as recommendations for use with the text. User interactions with images provides the system with personalized feedback, adding new associations/triplets to the reference database to further refine the model.

主权项

1. A computer implemented method, using a processor of a server, for recommending images, the method comprising:
receiving content comprising text, processing the content to extract data comprising text features, wherein the text features form a text feature vector; receiving a candidate set of image suggestions, the candidate set comprising images that may be submitted to a user taken from one or more collections of images, and for each image in the candidate set of image suggestions, processing the image to extract data comprising image features, wherein the image features form an image feature vector, receiving user information and processing the information to extract data comprising user features, wherein the user features form a user feature vector; storing the text feature vector, image feature vector, and user feature vector as a triplet in a reference database; and applying means for machine learning using said triplets to learn an association function to calculate an Illustration Index, comprising a score that measures the level of association of the image suggestion to the content; using the triplets stored in the reference database to generate one or more suggested images to be used with one or a plurality of subsequent text content input by a user in a user device; and displaying the one or more suggested images on a user device.