Ceph: Wire-Level Compression-Efficient Object Storage Daemon Communication for the Cloud

IMPORTANT: as of September 2022, this project is completed with no ongoing or planned work.

The project’s purpose is to reduce storage network traffic (object, block, etc.) for the following cases:

• Public cloud communications can be costly, for example between Availability Zones in AWS..

• In environments where the performance bottleneck of the node is the network and not the storage devices. This becomes very common as storage devices performance (such as NVMe SSDs and persistent memory devices)  improves much faster than network performance.

We have divided the project into 3 milestones:

• Applying compression for data transfer between different data centers, or different data nodes.

• Enable/disable compression given hints from the client

• Minimize compression efforts when data is non-compressible

The project aims to minimize the network traffic for the dashed red lines (the traffic for replicating the data in order to ensure availability and durability) while not wasting too much resources when this effort is hopeless (i.e. when the data is not compressible).