System and method for collection, retrieval, and distribution of data

摘要

A system may include a data collector to collect data from at least one data source and to tag each group of collected data with a metadata tag. The metadata tag may include information about the collected data. The system may also include a metadata catalogue for containing a plurality of metadata tags and to allow a search for existence and a location on a network of any data corresponding to the search.

主权项

1. A system for collection, retrieval, and distribution of data, comprising:
a computing device comprising a processor and a memory; a data collector embodied in the computing device to collect data from at least one data source and to tag each group of collected data with an associated metadata tag, wherein the metadata tag includes information about the collected data; an object distribution logic module associated with the data collector, the object distribution logic module comprising one or more caches to store data and a distributed caching strategy structure, the distributed caching strategy structure including a cache collapsing module to only send updated portions of data rather than an entire data set in accordance with data consumer submitted information and a store and forward feature to dynamically reduce flow of the data to data consumers; at least one other computing device; at least one distributed metadata catalogue embodied in the at least one other computing device for containing a plurality of metadata tags, wherein the metadata tags without any associated groups of collected data are stored on the at least one distributed metadata catalogue, wherein only metadata indicating the existence and location of a requested group of data on a network is sent to a requester in response to a data request corresponding to the requested group of data associated with the metadata tag to avoid sending the requested group of data over the network until needed; and wherein the at least one distributed metadata catalogue comprises a metadata catalogue query services module configured to receive a query from the data consumer, wherein the query is received by a local metadata catalogue query services module of the at least one distributed metadata catalogue that is a closest distance from the data consumer, the local catalogue query services module is configured to determine if a scope of the query is set to local, the local catalogue query services module responding to the query in response to the scope of the query being set to local and the local catalogue query services module passing the query to another catalogue query services module of another distributed metadata catalogue capable of responding to the query.