Virtualization is an amazing technology that has become a popular solution for sharing resources among members of an organization. However, some organizations need to harness the capabilities of an entire machine, without a layer of virtualization between the code and the hardware. Is it possible to share hardware between projects with the same ease as sharing virtual resources? 

Research into this question made clear that the flexibility and security necessary for sharing hardware created a host of additional complexities. The Massachusetts Open Cloud created an initiative to simplify those complexities, with a goal of enabling a multi-tenant, bare metal cloud where users can lease bare metal servers and create private networks to form their own isolated bare metal clusters. Initial work centered around the HIL (open.bu.edu/handle/2144/19198) and BMI/M2 (ieeexplore.ieee.org/abstract/document/8360313) projects. Those efforts have morphed into the Elastic Secure Infrastructure (ESI) project, which will build a solution on top of an established open source cloud platform: OpenStack.

The OpenStack service that manages bare metal is called Ironic. When we looked at Ironic, we discovered that it included many features we needed. However, it was missing a key element: bare metal node multi-tenancy. What does that mean? While most OpenStack services allow each project to own their own resources, Ironic provides an all-or-nothing view of bare metal hardware: either you’re an administrator and have access to everything, or you’re not and see nothing.

A framework for Ironic multi-tenancy

Node multi-tenancy has long been a nice-to-have goal for Ironic: a feature acknowledged to be useful, but which slips behind other important requirements in release after release. Fortunately OpenStack is an open source project, so we approached the upstream Ironic community and asked them how we might roll up our sleeves and implement the feature ourselves. They helped us document a plan that consisted of the following steps.

1. Make the Owner field operationally significant.
Ironic added an Owner field for nodes back in 2018, but it was strictly for informational purposes. The first step in bringing multi-tenancy to Ironic was to allow the Owner field to be used for access control.

2. Add a Lessee field to Ironic nodes.
Many of the workflows we envision involve a hardware owner temporarily lending their hardware to another project. This requires a mechanism for recording this secondary operator: the lessee.

3. Create new policy rules for owners and lessees.
Like most OpenStack services, access control in Ironic is managed through policy rules attached to various actions. Adding policy rules related to node owners and lessees allows administrators to grant node API access to non-admins.

These updates allow each Ironic node to specify a project as an owner (a project that will always be able to control the hardware) and/or a lessee (a project that will be able to control the hardware for the duration of a lease). An Ironic administrator can then expose individual node API calls to node owners and lessees by updating a policy file. For example, you can configure Ironic policy to allow:

• owners and lessees to control an individual node’s power state 
• owners (and not lessees) the ability to modify node attributes

These changes provide the foundation for a shared bare metal cluster, but they aren’t a complete solution. For example, provisioning an Ironic node requires the ability to update some node attributes, but granting a lessee the ability to update any node attribute would allow them to steal control of the node from the owner. What’s the solution? Simple: we created new policy rules with granular access control for specific node attributes. Lessees can then have restricted update access to these attributes, while owners still enjoy the ability to modify any node attribute they want.

The upstream Ironic community also pointed out additional Ironic resources that would be affected by node multi-tenancy. With their guidance, we added multi-tenant support for managing network ports and allocations (objects used for scheduling bare metal nodes when operating Ironic in standalone mode).

After all this was done, we attempted to provision a bare metal node using Metalsmith (metalsmith.readthedocs.io/en/latest), an upstream tool created expressly for this purpose, to see what further gaps we might have to close in order for non-admins to use the tool successfully. To our surprise and gratification, Metalsmith worked seamlessly with our changes, with no additional work necessary—a pretty good validation of the implementation!

What’s next for ESI?

The code for all this multi-tenant functionality is done and merged, and should be part of the OpenStack Ussuri release (opendev.org/openstack/ironic). Furthermore, during the 2020 OpenCloud Workshop we held a positive Birds of a Feather discussion (research.redhat.com/wp-content/uploads/2020/05/ESI_Ironic-Presentation.pdf) on multi-tenant Ironic with both ESI and OpenStack contributors participating. What comes next? We’re starting research into the networking and storage needs of ESI. We’re also planning a new leasing service that sits on top of all of this: a service that makes it easy for owners to offer up nodes for lease, and for lessees to claim those nodes.

Additional initiatives promise to expand the scope of ESI in exciting ways. Last summer, Boston University students collaborated on FLOCX (youtube.com/watch?v=goDpCRLhCao): a marketplace for bare metal with an early proof of concept that was demonstrated at DevConf.US 2019. Other researchers are investigating increased bare metal security through non-intrusive software introspection (research.redhat.com/wp-content/uploads/2020/04/Mohan_NISI.pdf) and secure allocation of resources (usenix.org/system/files/conference/hotcloud18/hotcloud18-paper-mosayyebzadeh.pdf).

ESI is a complex project with a lot of moving parts, but we’re excited to see it start to come together! If you’d like more information, browse through our Git repository at github.com/CCI-MOC/esi, or join the #moc IRC channel over on FreeNode.