by Anna Sandler – Software Quality Engineer, OpenShift Storage at Red Hat
and Luiza Nacshon – Senior Security Engineer, Managed Services at Red Hat
We began this project just over a year ago with a vision of combining knowledge in security, OpenShift, and automation into a product that will contribute to Red Hat security. The goal of the research was to explore the security aspects of the cloud world in greater depth and develop a tool that will help security researchers in the company test cloud security more effectively.
In the research phase, we started with some basic questions:
- What is a cloud?
- What is a hybrid cloud?
- What cloud solutions are developed by Red Hat?
- What are the main security risks in the cloud?
- What rules can be defined to protect hybrid cloud communications?
- What tool can we develop?
Research process
The initial step in our research was to understand and explore the possible attacks on hybrid clouds, what is currently covered, and what may not be covered yet by the community.
While exploring various security fields, we found that API security in a hybrid cloud lacks research and solutions. Therefore we decided to focus on this area.
Why cloud APIs?
For a simple server with one service (e.g., SMTP, file server, video streaming server, etc.), an API is created specifically for the type of data stored on this server. When using a cloud, the amount and variety of data stored on it can replace all the servers. Therefore, a cloud API should be able to handle data of all kinds — SQL, photos, applications, and more.
APIs are one of the main ways to communicate with the cloud, both from outside of the cloud and for inside communication between the different entities of a hybrid cloud. An API can function as the main and only door for data I/O in the cloud, and this door must be secured from attacks, data loss, and even authorized users without the right permissions.
Our research questions were:
- What rules can be defined to let the developers know if the API in the hybrid cloud communication is secured or vulnerable to attacks?
- Where should those rules be located?
- What are the common attacks that we should focus on in our research?
As a solution, we decided to focus on the following possible attacks on API in a hybrid cloud:
- Injections
By using injection flaws, such as JSON, XSS, SQL, NoSQL, or Command Injection, the attacker’s malicious data can trick the interpreter into executing unintended commands or accessing data without proper authorization. The cloud must be secured not from one but from all of these attacks. For example, when securing API from SQL injection, data can be retrieved using XSS and vice versa.
- Denial of Service
APIs are meant to deal with a large number of requests at any moment from a variety of users and for different purposes. That makes APIs a good target for Denial of Service (DoS) attacks: They are meant to deal with massive workloads, but when is a massive workload classified as normal or as an anomaly?
- Broken user authentication
Authentication mechanisms are often implemented incorrectly, allowing attackers to compromise authentication tokens or to exploit implementation flaws. An API must not only check the request but also validate the user requesting the data—their role, their permissions, and any limitations on those permissions (e.g., time limit, volume limit).
Proof of Concept and development
Our next research step was finding the best rules patterns to identify malicious activities in the API calls. The rules are based on a reversed proxy: the rapid penetration testing (PT) API test proxy. Any connections from private clusters to public clusters and vice versa go through the proxy.
On any incoming API call, the proxy executes the rules on the API calls. If malicious activity is found in the call, the request will be denied and we will see an error message in the dashboard with the possible vulnerability.
After the research phase, we developed a proof of concept (POC), a reverse proxy that helps Red Hat security engineers test the API in Red Hat cloud products in an easy way. We developed a rapid PT tool that works as a proxy between a hybrid cloud and an API. We wrote simple rules for the POC focused on checking four different attacks: JSON injection, DoS, XSS, and authentication.
POC Architecture
We have deployed two separate OpenShift clusters: one on a Red Hat lab and one on a private host. On each of them, we ran a SQL pod and loaded some data for the dummy API that we built.
The system core is the process that runs the tests. It executes approximately fourteen tests that are creating and sending API queries with different attack vectors: SQL injections, JSON injection, and a large number of queries at one time to simulate a DoS attack.
Each test waits for a response from the API and analyzes it:
- If the API returned data even though the query was injected, the test would fail.
- If the API blocked the malicious query, the test passed.
All the results are collected into a JSON file.
A Web dashboard reads the result from the file and displays them in an interactive and simple-to-read view.
This project is unique in that it works as a reverse proxy to run live penetration tests on API classes. It is a cross-platform system that does not depend on any cloud structure or API, which gives security engineers flexibility.
Technologies used in the POC
Development tools :
- Dashboard: Flask with bootstrap + Jinja + JSON
- API: Flask app to SQL DB pod
- Tests: Python 3.7
Development environment :
- Private cloud: Minishift on localhost
- Public cloud: OpenShift Container Platform (OCP) cluster in QE lab
Challenges
Working on this project was not easy. We faced a lot of challenges during the research and development phases.
No previous research
The first problem we encountered was the near-total lack of research on this subject. We found cloud security research, hybrid cloud research, cloud API security, or just web API security, but no research or a software solution focused on hybrid cloud API security, especially cloud-to-cloud channel API.
Lack of resources
When we started working on the POC, we had to deploy an environment to test our hypotheses. We tried several work frames:
- IBM cloud: We contacted Avi Vizel, the university relations manager at IBM Israel, who gave us free usage of the IBM cloud. Unfortunately, we didn’t have enough resources to run an OpenShift cluster, and this option has dropped.
- MOC: The Massachusetts Open Cloud gave us some machines deployed on OpenStack. Deploying OpenShift on OpenStack in this environment was a bit tricky, and we decided to pass.
- Red Hat lab: We requested access to Red Hat’s training lab shared clusters, which were blocked to SSH connections
- VMware: Deploying OCP cluster in the QE lab was the easiest way to run a public cloud and was used for the final POC
- Minishift: This was the best match for running a private cloud
Deployment issues
After picking up the deployments that fitted us the most, we encountered problems such as:
- How can we create a connection between those deployments?
- How do we choose the right operator to test data I/O?
- Can we run the proxy on the system?
Summary
Working on this project was a unique experience, as we were looking into a new, almost unresearched subject. We had to match facts and concepts from different resources in academia and industry, talking to professional people from Red Hat Research, R & D, security, QE, and even IBM, to create a new product that will be developed in the future.
Given the importance of security in the cloud computing world, we found that there is a lack of solutions for securing API in the hybrid cloud, especially in live communication. In the future, we are hoping that the tool developed from this research will be part of the Red Hat rapid PT tool.
Any comments and suggestions are very welcome. Reach out to Anna or Luiza if you have any questions or comments.
About the authors
Anna Sandler joined Red Hat as an intern in her second year of university. This research and development project was her final project for her degree. Her mentor on the project was Luiza Nacshon from the managed services security team at Red Hat.
The motivation behind the project was combining the things Anna learned during her cyber specialization in her BSc studies with the OpenShift and cloud world she discovered while working as an intern at Red Hat.
