Open CEP – An Advanced Open Source Complex Event Processing Engine
IMPORTANT: as of September 2022, this project is completed with no ongoing or planned work.
The aim of this project is to create a scalable open-source complex event processing (CEP) framework. CEP is a prominent technology for robust and high-performance real-time detection of arbitrarily complex patterns in massive data streams. It is widely employed in many areas where extremely large amounts of streaming data are continuously generated and need to be promptly and efficiently analyzed on-the-fly. Online finance, network security monitoring, credit card fraud detection, sensor networks, traffic monitoring, healthcare industry, and IoT applications are among the many examples.
CEP systems treat data items as primitive events arriving from event sources. As new primitive events are observed, they are assembled into higher-level complex events matching the specified patterns. The process of complex event detection generally consists of collecting primitive events and combining them into potential (partial) matches using some type of detection model. As more events are added to a partial match, a full pattern match is eventually formed and reported.
The patterns detected by CEP engines are often of exceedingly high complexity and nesting level, and may include multiple complex operators and conditions on the data items. Moreover, these systems are typically required to operate under tight constraints on response time and detection precision, and to process multiple patterns and streams in parallel. Therefore, advanced algorithmic solutions and sophisticated optimizations must be utilized by CEP implementations to achieve an acceptable level of service quality.
The figure above illustrates the general structure of a CEP system. Incoming data streams are analyzed on-the-fly and useful statistics and data characteristics are extracted to facilitate the optimizer in applying the aforementioned optimization techniques and maximize the performance of the evaluation mechanism – a component in charge of the actual pattern matching.
This project aims at creating an open-source CEP project and a supporting community of users and contributors. Our fully operational CEP framework will feature a set of generic and intuitive APIs, making it easily applicable to any domain where event-based streaming data is present. In collaboration with top CEP experts from the Technion, Israel Institute of Technology, we will develop an engine that utilizes the latest breakthrough results of ongoing research. By incorporating a multitude of state-of-the-art methods and algorithms for scalable event processing, our system will be able to adequately satisfy the requirements of modern event-driven domains and outperform existing alternatives, both in terms of the actual performance and the provided functionality.
The engine will be implemented as a standalone library, with advanced execution modes including OpenShift deployment and integration with other stream processing frameworks.
Link to the repository.
For more information click here.
Publications
Lazy Evaluation Methods for Detecting Complex Events
Join Query Optimization Techniques for Complex Event Processing Applications