Big Stream Processing Inside and Outside Red Hat: Research Day Summary

Mar 24, 2021 | Blog, Research Day

The first Tel Aviv Red Hat Research Day event took place on March 2nd. During this session, Dr. Ilya Kolchinsky outlined the most critical challenges faced by the currently available data processing technologies, presented a new paradigm for large-scale data processing called Stream Processing, and discussed how can this paradigm be employed both inside and outside of Red Hat to address the aforementioned challenges and bring additional value to the customers.

As we enter the era of Big Data, a large number of data-driven systems and applications have become an integral part of our daily lives. this trend is accelerating dramatically. It is estimated that 1.7MB of data is created every second for every person on Earth, for a total of over 2.5 quintillion bytes of new data every day, reaching 163 zettabytes by 2025. In addition to the growing volume, velocity, and variety of continuously generated data, novel technological trends such as edge processing, IoT, 5G, and federated AI bring new requirements for faster processing and deeper, more computationally heavy data analysis. 

Meeting these requirements in modern applications by relying on the “old school” data processing mechanisms is nearly impossible, however. On the one hand, the high latency and I/O overhead of the traditional database systems prevent the required computation result to be available immediately in real-time. On the other hand, by simply dropping the DB we severely limit the complexity of the supported operations due to the scarcity of the processing resources. To overcome this situation, a new solution is needed.

Stream processing comprises a variety of methods for scalable and efficient data processing that do not rely on traditional databases for storing and processing the data. Instead, the main focus of these methods is on performing highly complex computations on high-rate data streams while only using minimal resources. This makes stream processing a perfect choice for implementing intensive data processing operations in real-time applications and on edge devices.

During the talk, we identified a number of tools and technologies closely related to Red Hat products which could greatly benefit from integrating stream processing solutions into their core, achieving an orders-of-magnitude performance boost. Among these examples are Kubernetes, Ceph, Elasticsearch, and Prometheus. We believe that this list could be continued and are looking forward to collaborating on these and many other initiatives.

For more information click here.

Related Stories

Encouraging mentees to thrive: How to be a good mentor

Encouraging mentees to thrive: How to be a good mentor

An internship is a great opportunity for a company to evaluate candidates in a time-limited job role, but it is also a chance for interns to learn, gain work experience, and evaluate that company as a potential future employer; this is an equally important part of the position.

Beyond – The 3rd round for the Open Source academy course

Beyond – The 3rd round for the Open Source academy course

During the summer semester of 2020, Red Hat’s Beyond platform offered a class on open source development in conjunction with Efi Arazi School of Computer Science at the Interdisciplinary Center (IDC) of Herzliya, Israel’s only private university.

Glimpses into Future Tech: Red Hat Research Days 2020

Glimpses into Future Tech: Red Hat Research Days 2020

In this series of Research Days, Red Hat experts, including Red Hat’s Chief Technology Officer, Chris Wright, and researchers from Harvard, Yale, Boston University, and other leading universities discussed their ongoing research on technology to improve privacy and security, make experimentation and system execution more reproducible, and enhance the performance and reliability of cloud systems.

Bringing technical concepts to life through digital illustration

Bringing technical concepts to life through digital illustration

Red Hat Research Days 2020 was an experiment in recreating face to face interaction between researchers, Red Hat engineers, customers and partners in the virtual space. The goal; moving research into open source communities. And the event was immensely successful --...

Open Source at the Turing

Open Source at the Turing

In July last year, Red Hat visited the Turing Institute to deliver a two-day workshop on Open Source and why it's a great choice for academic software. The majority of the schedule was available to members of Turing Institute, however the first day was closed with a...

Jonathan Cameron: Getting Started on Linux Kernel

Jonathan Cameron: Getting Started on Linux Kernel

Jonathan Cameron is a rising senior at Boston University studying Computer Engineering. He spoke with the Red Hat Research (RHR) team about his internship project and why this project is important for future interns working on the Linux kernel.  RHR: Jonathan,...

Where there’s a will there’s a way: Honors graduate and blind programmer Vojtěch Polásek joins the Red Hat Security Compliance team.

Where there’s a will there’s a way: Honors graduate and blind programmer Vojtěch Polásek joins the Red Hat Security Compliance team.

Vojta Polásek graduated with honors at Masaryk University Brno, receiving the Dean’s award for his Bachelor and Diploma theses. His diploma thesis was nominated by his faculty to prestigious Czech and Slovak IT SPY competition evaluating IT diploma theses and was among the 8 best finalists. Recently, Vojta joined Red Hat and kick-started his career in the Security Compliance team. While Vojta’s record is impressive for any young engineer, it’s made all the more remarkable by the fact that he was born visually impaired and lost his sight entirely during his teenage years.