Systems and methods for dynamic sharding of hierarchical data

摘要

A method for serving a request for hierarchical data includes receiving, at a processing circuit, a request for hierarchical data stored in one or more databases. The request includes an indication of a particular data hierarchy. The method further includes identifying, by the processing circuit, one or more entity count records corresponding to the particular data hierarchy. The entity count records indicate a distribution of the hierarchical data within the data hierarchy. The method further includes splitting the request for hierarchical data into a plurality of sharded requests based on the distribution of the hierarchical data indicated by the one or more entity count records, assigning the plurality of sharded requests to one or more processing sites for processing at least a subset of the sharded requests in parallel, and reporting a combined result of processing the plurality of sharded requests.

主权项

1. A method for serving a request for hierarchical data, the method comprising:
receiving, at a sharding service component of one or more data processing sites that each include one or more machines, a request for hierarchical data stored in one or more databases, the request including an indication of a particular data hierarchy; identifying, by the sharding service component, one or more entity count records corresponding to the particular data hierarchy, the entity count records indicating a distribution of the hierarchical data within the data hierarchy; splitting, by the sharding service component, the request for hierarchical data into a plurality of sharded requests based on the distribution of the hierarchical data indicated by the one or more entity count records, wherein splitting the request for hierarchical data into the plurality of sharded requests comprises recursively splitting a sharded request into a plurality of sub-sharded requests in response to a determination that a sharded request is associated with a shard that has a size exceeding a maximum threshold value; assigning the plurality of sharded requests to one or more of the one or more data processing sites for processing at least a subset of the sharded requests in parallel; and reporting a combined result of processing the plurality of sharded requests.