Method and apparatus for data storage and downloading

摘要

The present invention discloses a data storage method, a data downloading method, and a related apparatus. The data storage method includes: receiving a storage request containing a uniform resource locator (URL) for a storage operation; acquiring the URL in the storage request; determining whether a URL schema corresponding to the URL is in a storage, the URL schema corresponding to different URLs for a same file under a same domain name; when the URL schema is in the storage, completing the storage operation without storing the URL; when the URL schema is not in the storage, determining a hash value of a file corresponding to the URL and a hash value of the URL schema, and storing mapping relationship information between the file hash value and the URL schema hash value in the URL index database. The storage resource usage of the URL index database is reduced.

主权项

1. A method for data storage, comprising:
receiving a storage request containing a uniform resource locator (URL) for a storage operation; acquiring the URL in the storage request; determining whether a URL schema corresponding to the URL is in a storage, the URL schema corresponding to different URLs for a same file under a same domain name; when it is determined that the URL schema is in the storage, completing the storage operation without storing the URL in a URL index database; when it is determined that the URL schema is not in the storage, determining a hash value of a file corresponding to the URL and a hash value of the URL schema, and storing mapping relationship information between the hash value of the file and the hash value of the URL schema in the URL index database, wherein determining the hash value of the file corresponding to the URL includes:
if the storage request is a storage request sent from a downloading client terminal, acquiring a file hash value in the storage request as the hash value;if the storage request is a storage request sent from a resource crawler system, downloading the file corresponding to the URL in the storage request, and determining the hash value of the downloaded file.