The recent past has seen the adoption of multi-cloud deployments by enterprises due to availability, features and regulatory requirements. A typical deployment involves parts of an application/workloads running inside a private cloud with the other parts spread across multiple on-prem/public clouds. Typical cluster-to-cluster networking in such deployments involve the establishment of site-to-site encrypted tunnels to connect the workloads.
We systematically evaluate popular overlay tunnelling and encryption mechanisms over different setups (inter-DC VMs over WAN, intra-DC VMs within a data-center and an on-prem intra-rack bare-metal setup) to provide insights on their performance (throughput & latency) and the overhead (CPU utilization) incurred. We performed the experiments using standard open-source performance benchmarking tools. Based on the various experiments conducted on three different testbeds, we present quantifiable data which can be leveraged by operators and cloud providers tasked with design and development decisions.