Twitter has unveiled Heron, a real-time analytics system that will better support analytics for the scale and activity of its social stream. Heron, which replaces Storm, will also be able to take the load of increased activities during any popular event. The micro-blogging site has designed Heron to enhance developers’ productivity and provide better performance predictability.
Faster than Storm
Heron is integrated with a data flow pressure mechanism for handling any increase in traffic but does not compromise on data accuracy. Heron has already been deployed by Twitter, and the company said that it is now seeing a threefold reduction in hardware, resulting in substantial improvements in the infrastructure’s efficiency.
Various tests reveal that the new data analytics software is a step ahead of Storm in performance. Heron will help Twitter process billions of events per minute, and the company claims it is 15-times faster than Storm. Twitter rolled out Heron as the main stream processing engine, according to Ramasamy, but it is backward-compatible with Storm, which was widely used across Twitter previously.
Over the years, various competing services such as Google Cloud platform, Microsoft Azure and Amazon Web Services have used Storm, and Heron is apparently capable of replacing its predecessor at these companies. Ramasamy did not reveal if the company would release Heron under an open-source license, although he talked about publishing an academic paper on Heron at the 2015 ACM SIGMOD International Conference on Management of Data, which would be a first step toward sharing lessons learned “in order to further develop these programs.”
Why Twitter designed Heron
Twitter, in a blog post, mentioned that there are certain systemic qualities of a large-scale, real-time streaming system. It needs to process billions of events per minute and have a high accuracy level in handling data, strength to handle any increase in the traffic and pipeline congestion, and easy debugging and be simple to deploy.
Twitter mentioned in its blog post that Heron packs the strength to handle large-scale topologies with high throughput and low latency requirements. Twitter stated that in order to meet the required needs, it analyzed several options such as enhancing Storm, deploying an alternative open source system, developing a brand new one and so on. Since it was required to change the core architecture of Storm in order to develop longer development cycles, Twitter went with the plan of creating a new tool.
