System and method providing hierarchical cache for big data applications

摘要

The embodiments herein develop a system for providing hierarchical cache for big data processing. The system comprises a caching layer, a plurality of actors in communication with the caching layer, a machine hosting the plurality of actors, a plurality of replication channels in communication with the plurality of actors, a predefined ring structure. The caching layer is a chain of memory and storage capacity elements, configured to store a data from the input stream. The plurality of actors is configured to replicate the input data stream and forward the replicated data to the caching layer. The replication channels are configured to forward the replicated data from a particular actor to another actor. The predefined ring structure maps the input data to the replica actors.

主权项

1. A system for providing a hierarchical cache for big data processing, the system comprising:
a caching layer configured to store a data from an input stream, and wherein the caching layer is a chain of memory and storage capacity elements, wherein the caching layer comprises:
a stack comprising a plurality of storage elements, and wherein each storage element is configured to store an input stream chunk, and wherein the plurality of storage elements is hierarchically sorted based on an access performance of a main memory operations and cost;a top element of the stack, and wherein the top element of the stack is an active memory available to the actor; andan adapter operated on the top element of the stack and configured to push a received data to the top element of the stack;a plurality of actors, in communication with the caching layer, configured to replicate the input data stream and forward the replicated data to the caching layer;a machine configured to host the plurality of actors;a plurality of replication channels, in communication with the plurality of actors, configured to forward the replicated data from a selected actor to another actor; anda predefined ring structure to map an input data to replicate the plurality of actors.