SYS-CON MEDIA Authors: Stackify Blog, Zakia Bouachraoui, Elizabeth White, Pat Romanski, Liz McMillan

Blog Feed Post

What Exactly is Complex Event Processing Today?

As much as I disagree with much of what Curt Monash writes, he did actually ask a good question recently in his post, “Renaming CEP… or not

Without getting into a rehash of the hash over there, let’s look at things a bit differently.  Let’s talk about what CEP is not.

I left trading to join a firm called NEON.  I was an early investor in this company, my mentor had started the firm, and to me it looked like the cat’s meow.  It was a great time, a lot of people made a ton of money, and I was introduced into the world of software via Enterprise Application Integration (EAI).  Using EAI, one could centralize the business logic associated with plugging different systems into each other, convert the format of one system into another, and pass messages around seamlessly between these applications.  There weren’t a lot of competitors at that time, we were arguable #1 in the space, and the technology became MQ Series Integrator (think Websphere, like I said, we made a lot of money.)

Well, CEP isn’t EAI because there’s no concept of format libraries – sure CEP engines use input/output adapters but sure does every program ever written (I’m waiting for the first salesperson to licence the keyboard/screen adapter set – available in different languages soon!).  We’re going to come back to EAI in a moment.

Throughout my career, the teams I’ve worked with have used a variety of 4th generation languages.  Stuff like Powerbuilder, SQL Windows, Paradox, etc.  Each of those environments had some common elements, screen designers, a domain specific language designed to make bizapp dev faster, and abstractions for common data stores.  Often times, our groups wrote servers that integrated with these front ends via RPC, a streaming connection, or databases.

CEP isn’t a 4th generation bizapp dev environment – there’s no facility for building gui’s.  Although some of the CEP platforms out there do have DSL’s, some also use SQL derivations.  I’ve used the SQL derivations (I’ve worked at two of those co’s) and guess what?  The people in those firms hated using the language themselves.  ”Yes, you could do a covariance matrix with <insert proprietary get-me-sued-for-naming-it-here> but I could do it faster and easier in a different language.

I’ve also used many databases.  But you don’t use CEP to store data – you only process the data in flight.

So, CEP isn’t EAI, it’s not a database, and it’s not an application development environment.  Where, then, did CEP come from?  Let’s look at a couple.

The work out of Berkeley and the work out of Brown, Brandeis and MIT focused on event stream processing.  Here’s a blurb about Berkeley’s Telegraph:

Telegraph is an adaptive data-flow system, which allows individuals and institutions to access, combine, analyze, and otherwise benefit from this data wherever it resides.  As a data-flow system, Telegraph can tap into pooled data stored on the network, and harness streams of live data coming out of networked sensors, software, and smart devices.  In order to operate robustly in this volatile, inter-networked world, Telegraph is adaptive – it uses new data-flow technologies to route unpredictable and bursty data-flows through computing resources on a network, resulting in manageable streams of useful information.

And here’s one about Aurora (Brown, Brandeis, & MIT):

Aurora addresses three broad application types in a single, unique framework:

  1. Real-time monitoring applications continuously monitor the present state of the world and are, thus, interested in the most current data as it arrives from the environment. In these applications, there is little or no need (or time) to store such data.
  2. Archival applications are typically interested in the past. They are primarily concerned with processing large amounts of finite data stored in a time-series repository.
  3. Spanning applications involve both the present and past states of the world, requiring combining and comparing incoming live data and stored historical data. These applications are the most demanding as there is a need to balance real-time requirements with efficient processing of large amounts of disk-resident data.

Hmm.  I’ve worked with both of those packages – no mention of Complex Event Processing in there at all.  So where did that phrase even come from?  Well, that’s the title of David Luckham’s book, “The Power of the Event” in which the good professor describes not so much an implementation, but a set of processes designed to help us all run our businesses and missions more effectively.  In the book though, David references a language that deals with streaming data.  Oh oh….

Around 2005-2006, a couple of firms were struggling trying to describe what Event Stream Processing was and why it was important and more importantly, why you should be spending money on it.  I was the CTO of one of those firms.  We competed mostly against Streambase at the time.  Somewhere during that time frame, the phrase Complex Event Processing was adapted in an effort to differentiate.  At that time, Aleri wasn’t CEP – they were OLAP.  Streambase, formerly Grassy Brook, probably choose that name in homage to Stream Processing.  Kaskad is Swedish for waterfall, or where a bunch of rivers and/or streams collide.  I don’t think Apama ever used the phrase ESP, they were focused on trading from the start.  Starting to get the picture?

So, if CEP isn’t EAI, and it’s not a 4th generation bizapp tool, what is it?  I’ve probably kicked this dead horse enough, but one more time and it’s not going to notice.  CEP needs 4 things to be called CEP (or ESP…):

  1. Domain Specific Language
  2. Continuous Query
  3. Time or Length Windows
  4. Temporal Pattern Matching

These 4 things, in my opinion, don’t make up a separate space, let alone a market.  What they describe is Event Stream Processing.  What they describe are features found in larger, more complete event processing environments from IBM, SAP, TIBCO, and Progress.  And TIBCO, for example, just added the missing features described above to their Business Events Platform, and had instant CEP (sarcasm mine).  Those offerings look a lot like the traditional EAI platforms – or where all of this began roughly 20 years ago.

So, I don’t think what a couple of vendors sell as Complex Event Processing is really CEP at all.  If you want an idea of what CEP is really all about, read David’s book to get started.  Then take a look at Tim Bass’s blog thecepblog.com.

In my next post, I’ll describe what CEP means to me and talk about some of the current offerings in that space.  But for now, let’s just drop the phrase CEP (because it’s mostly just Stream Processing) because it means so little to so many and fails to impart any meaningful message to the people who actually write checks for this stuff.

Thanks for reading!

Read the original blog entry...

More Stories By Colin Clark

Colin Clark is the CTO for Cloud Event Processing, Inc. and is widely regarded as a thought leader and pioneer in both Complex Event Processing and its application within Capital Markets.

Follow Colin on Twitter at http:\\twitter.com\EventCloudPro to learn more about cloud based event processing using map/reduce, complex event processing, and event driven pattern matching agents. You can also send topic suggestions or questions to [email protected]

Latest Stories
In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). According to the survey, a quarter of the respondents have already deployed Docker containers and nearly as many (23 percent) are employing the AWS Lambda serverless computing framework. It's clear: serverless is here to stay. The adoption does come with some needed changes, within both application development and operations. Th...
The benefits of automated cloud deployments for speed, reliability and security are undeniable. The cornerstone of this approach, immutable deployment, promotes the idea of continuously rolling safe, stable images instead of trying to keep up with managing a fixed pool of virtual or physical machines. In this talk, we'll explore the immutable infrastructure pattern and how to use continuous deployment and continuous integration (CI/CD) process to build and manage server images for any platform....
AI and machine learning disruption for Enterprises started happening in the areas such as IT operations management (ITOPs) and Cloud management and SaaS apps. In 2019 CIOs will see disruptive solutions for Cloud & Devops, AI/ML driven IT Ops and Cloud Ops. Customers want AI-driven multi-cloud operations for monitoring, detection, prevention of disruptions. Disruptions cause revenue loss, unhappy users, impacts brand reputation etc.
Atmosera delivers modern cloud services that maximize the advantages of cloud-based infrastructures. Offering private, hybrid, and public cloud solutions, Atmosera works closely with customers to engineer, deploy, and operate cloud architectures with advanced services that deliver strategic business outcomes. Atmosera's expertise simplifies the process of cloud transformation and our 20+ years of experience managing complex IT environments provides our customers with the confidence and trust tha...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
Enterprises are adopting Kubernetes to accelerate the development and the delivery of cloud-native applications. However, sharing a Kubernetes cluster between members of the same team can be challenging. And, sharing clusters across multiple teams is even harder. Kubernetes offers several constructs to help implement segmentation and isolation. However, these primitives can be complex to understand and apply. As a result, it’s becoming common for enterprises to end up with several clusters. This...
Is advanced scheduling in Kubernetes achievable?Yes, however, how do you properly accommodate every real-life scenario that a Kubernetes user might encounter? How do you leverage advanced scheduling techniques to shape and describe each scenario in easy-to-use rules and configurations? In his session at @DevOpsSummit at 21st Cloud Expo, Oleg Chunikhin, CTO at Kublr, answered these questions and demonstrated techniques for implementing advanced scheduling. For example, using spot instances and co...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Public clouds dominate IT conversations but the next phase of cloud evolutions are "multi" hybrid cloud environments. The winners in the cloud services industry will be those organizations that understand how to leverage these technologies as complete service solutions for specific customer verticals. In turn, both business and IT actors throughout the enterprise will need to increase their engagement with multi-cloud deployments today while planning a technology strategy that will constitute a ...
GCP Marketplace is based on a multi-cloud and hybrid-first philosophy, focused on giving Google Cloud partners and enterprise customers flexibility without lock-in. It also helps customers innovate by easily adopting new technologies from ISV partners, such as commercial Kubernetes applications, and allows companies to oversee the full lifecycle of a solution, from discovery through management.
Using serverless computing has a number of obvious benefits over traditional application infrastructure - you pay only for what you use, scale up or down immediately to match supply with demand, and avoid operating any server infrastructure at all. However, implementing maintainable and scalable applications using serverless computing services like AWS Lambda poses a number of challenges. The absence of long-lived, user-managed servers means that states cannot be maintained by the service. Lo...
Today most companies are adopting or evaluating container technology - Docker in particular - to speed up application deployment, drive down cost, ease management and make application delivery more flexible overall. As with most new architectures, this dream takes significant work to become a reality. Even when you do get your application componentized enough and packaged properly, there are still challenges for DevOps teams to making the shift to continuous delivery and achieving that reducti...
Using serverless computing has a number of obvious benefits over traditional application infrastructure - you pay only for what you use, scale up or down immediately to match supply with demand, and avoid operating any server infrastructure at all. However, implementing maintainable and scalable applications using serverless computing services like AWS Lambda poses a number of challenges. The absence of long-lived, user-managed servers means that states cannot be maintained by the service. Lo...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Signs of a shift in the usage of public clouds are everywhere. Previously, as organizations outgrew old IT methods, the natural answer was to try the public cloud approach; however, the public platform alone is not a complete solution. Complaints include unpredictable/escalating costs and mounting security concerns in the public cloud. Ultimately, public cloud adoption can ultimately mean a shift of IT pains instead of a resolution. That's why the move to hybrid, custom, and multi-cloud will ...