HYPERSONIC: A Hybrid Parallelization Approach for Scalable Complex Event Processing
Authors
Maor Yankovitch, Technion, Israel Institute of Technology, Haifa, Israel; Ilya Kolchinsky, Red Hat and Technion, Israel Institute of Technology, Haifa, Israel; and Assaf Schuster, Technion, Israel Institute of Technology, Haifa, Israel
Abstract
The ability to promptly and efficiently detect arbitrarily complex patterns in massive real-time data streams is a crucial requirement in many modern applications. The ever-growing scale of these applications and the sophistication of the patterns involved make it imperative to employ advanced solutions that can optimize pattern detection. One of the most prominent and well-established ways to achieve the above goal is to apply complex event processing (CEP) in a parallel manner, using a multi-core machine and/or a distributed environment. However, the inherent tightly coupled nature of CEP severely limits the scalability of the parallelization methods currently available. In this paper, we introduce a novel parallelization mechanism for efficient complex event processing over data streams. This mechanism is based on a hybrid two-tier model combining multiple layers of parallelism. By employing a fine-grained load balancing model, this multi-layered approach leads to a substantial increase in event detection throughput, while at the same time reducing the latency and the memory consumption. An extensive experimental evaluation on multiple real-life datasets shows that our approach consistently outperforms state-of-the-art CEP parallelization methods by a factor of two to three orders of magnitude.
From
SIGMOD ’22: Proceedings of the 2022 International Conference on Management of Data, June 2022, Pages 1093–1107, https://doi.org/10.1145/3514221.3517829