Each quarter, Red Hat Research Quarterly highlights new and ongoing research collaborations from around the world.
This quarter we highlight collaborative projects in Israel at The Technion, The Ben Gurion University of The Negev, Ariel University, Reichman University, and The Hebrew University. Contact academic@redhat.com for more information on any project described here, or explore more research projects at research.redhat.com.
PROJECT: SpotOS
ACADEMIC INVESTIGATOR: Prof. Assaf Schuster (Technion)
Red Hat investigator: Ilya Kolchinsky
The SpotOS project aims to devise a distributed cloud-based operating system that uses unreliable or temporarily available resources to provide a reliable and scalable execution experience with a high quality of service. The proposed system will achieve this vision by harnessing the power of spot instances, resources representing the currently unused cloud capacity. While spot instances are considerably cheaper than “regular” instances, they can be unexpectedly reclaimed by the cloud provider at any time. In this case, a very limited time window is given to the running application to back up its current state. SpotOS aims to overcome this limitation by providing a reliable, adaptive, self-healing, user-transparent layer with spot instances serving as the underlying unreliable building blocks.
The low-level design of SpotOS is complete. The next major step is implementing a prototype of the component representing the key innovation: the EDM (external distributed memory). With EDM, the application state is split among multiple storage units (that could be hosted on regular instances, spot instances, or a mix of both) in sufficiently small chunks to complete the evacuation on time. Designing and implementing the EDM will pave the way for the rest of the SpotOS framework.
PROJECT: Kubernetes optimized service discovery across clusters
ACADEMIC INVESTIGATORS: Prof. Anat Bremler-Barr and Daniel Bachar (Reichman University)
Red Hat investigator: Vishal Thapar
This research project aims to provide better and more balanced service discovery capabilities for Kubernetes multicluster deployments. Currently, the service discovery in this space is very basic. The project aims to investigate and assess different approaches to improve it by making it more balanced, reducing bottlenecks, and improving latency.
After several iterations on the design and implementation, code for simple round robin and weighted round robin is now available as part of the Lighthouse plugin in Submariner. Details on the design are available on GitHub. This project was also featured in the article “Optimizing Kubernetes service selection” in RHRQ 3:4 (February 2022).
PROJECT: CCO: cloud cost optimizer
ACADEMIC INVESTIGATOR: Prof. Assaf Schuster (Technion)
Red Hat investigator: Ilya Kolchinsky
Cost optimization is one of the core challenges for users of cloud computing platforms: given a workload, how can we minimize the monetary cost of its deployment and execution over the cloud? Accurately answering this fundamental question for arbitrarily complex real-life workloads turns out to be an exceedingly hard problem. The goal of this project is to develop a tool, which we call the cloud cost optimizer (CCO), capable of doing exactly that. CCO retrieves information regarding the prices and the capacities of all instance types, regions, availability zones, and so forth as advertised by the cloud provider. Based on this information, CCO calculates the optimal allocation of workload components to minimize the overall cost.
An early alpha version of the optimizer, limited to AWS and only supporting applications with a small-to-medium number of distinct components, is currently available for use. Our ongoing efforts focus on several areas, most importantly:
- Lifting the above limitations by introducing more sophisticated optimization algorithms and techniques
- Extending CCO to take into account additional operational aspects, such as performance and reliability, during the decision-making process
- Introducing a machine learning component for predicting future instance prices to improve the accuracy of the optimization process
PROJECT: OpenCEP: an advanced open source complex event processing engine
ACADEMIC INVESTIGATOR: Prof. Assaf Schuster (Technion)
Red Hat investigator: Ilya Kolchinsky
In RHRQ 2:1, investigators described their plan to build a scalable, real-time, cloud-based CEP engine capable of efficiently detecting arbitrarily complex patterns in high-volume data streams. The engine was designed to be implemented on top of the Red Hat® OpenShift® Container Platform and to be applicable to any domain where event-based streaming data is present. Researchers aim to advance the state of the art in complex event processing and combine academic research with the implementation and deployment of novel CEP mechanisms and techniques.
The beta version of OpenCEP is currently available for use. The project team is now mainly tasked with solving the open issues and bugs, addressing user feedback, introducing minor features and modifications, and improving the project documentation.
PROJECT: Advanced proactive caching for heterogeneous storage systems
ACADEMIC INVESTIGATORS: Dr. Gabriel Scalosub and Dr. Gil Einziger (Ben Gurion University of The Negev)
Red Hat investigators: Josh Salomon, Gabriel BenHanokh, Orit Wasserman, and Mark Kogan
Caching is a classic optimization technique ubiquitous both within single hosts and in large distributed systems. While the cache algorithms currently employed in real-life systems are highly efficient and yield significant performance improvements, their reactive behavior severely limits their ability to exploit various naturally arising patterns.
This project aims to overcome this limitation by developing a novel caching framework that performs proactive caching decisions, considers the various heterogeneous characteristics of the workload, and exploits recurring access patterns to speculatively prefetch items into the cache. In addition to the purely algorithmic contribution, we intend to pursue upstreaming our suggested solutions within the open source community, most notably within the Ceph architecture, for example, by extending the D4N framework.
The project’s research phase is underway, with many promising approaches, insights, and results available. According to the current timeline estimations, the first prototype version of the proactive caching mechanism will be available mid-2023.
PROJECT: PHYSICS: oPtimized HYbrid Space-time servIce Continuum in faaS
Red Hat Investigators: Luis Tomas Bolivar and Josh Salomon
The PHYSICS project started in January 2021, with funding from the Horizon Europe research and innovation program and participation from fourteen international partners. It is scheduled to end in December 2023. The goal of PHYSICS is to unlock the potential of the Function-as-a-Service (FaaS) paradigm for cloud service providers and application developers. Specifically, it will enable application developers to design, implement, and deploy advanced FaaS applications in the scope of advanced cloud application design environments, leveraging proven design patterns and existing libraries of cloud/FaaS components. PHYSICS will offer a novel Global Continuum Layer that will deploy functions to optimize multiple application objectives simultaneously, such as performance, latency, and cost. PHYSICS will validate the benefits of its Global Continuum Layer and tools in the scope of user-driven application scenarios in three important sectors: healthcare, agriculture, and industry. Visit physics-faas.eu for more information.
The main architecture and components have already been defined, as well as the first prototype of them. The main architecture is based on several projects Red Hat is working on, such as OCM and Submariner for multicluster environments, MicroShift for the edge, and OperatorSDK for building the needed operators to glue different components together. Current work is focused on integrating the various components towards a first end-to-end demonstration.
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101017047.
PROJECT: Ceph: Wire-level compression: efficient object storage daemon communication for the cloud
ACADEMIC INVESTIGATORS: Prof. Anat Bremler-Barr and Maya Gilad (Reichman University)
Red Hat investigator: Josh Salomon
This project’s purpose is to reduce storage network traffic (object, block, etc.) for the following cases: between the failure domains in cost-sensitive environments such as public clouds, and between nodes in cases where the network bandwidth is the bottleneck of the node performance.
The project was completed successfully. The code was merged to Ceph version 17 (Quincy) and will be part of RHCS v6 and ODF versions based on RHCS 6 (probably ODF 12 and on). More information can be found in the pull request.
PROJECT: Secured API in hybrid cloud
ACADEMIC INVESTIGATOR: Dr. Amit Davir (Ariel University)
Red Hat investigators: Avihu Goren and Noa Haim, mentored by Anna Sandler and Luiza Nacshon
The project was submitted successfully last year and has attracted much interest at the university and in the Red Hat products security group. That project led to creating a RapidPT tool for testing API security.
A new student has taken the lead in continuing last year’s project, and we are now focusing on improving the PT tool to support a broader range of APIs and attack vectors. We plan to implement a user-friendly dashboard and a Firebase database to store results and user data in a more optimal way.