In summary:

• Anything that happens, happens.
• Anything that, in happening, causes something else to happen, causes something else to happen.
• Anything that, in happening, causes itself to happen again, happens again.
• It doesn’t necessarily do it in chronological order, though.

One of the features which are used to sell event processing systems are their ability to keep track of which event caused something to happen.

This might sound simple. But it’s actually something which is more or less impossible to do if you are not treating events as events. Events could just be data which is called events to make things sound more impressive. Data processing is old, but event processing is new and cool.

Events have an id. Data does not.

For example, If I write two SQL queries which both return the value 12, if I then store these both values, then I would later on have no way of knowing which value came from which query. Data is simply anonymous. Events on the other hand have distinct and unique identities. Two stored events can always be identified.

This is one of the fundamental differences between data processing and event processing. Continuous and real-time processing of data streams can be easily confused with event processing. Although a great number of problems require data-driven algorithms which are executed continuously in real-time - That’s not event processing.

This is one of the first and fundamental issue that one should sort out before starting to build event processing systems. Is it really event processing you need or is it data driven processing?

One of the best ways to try to understand your requirements is to ask yourself if your basic processing entity has an id or not. If it has an id, you might need and event processing solution. If not, go for data processing.

For example. A stream of price information for iron ore could contain values like 65.0, 73.5, 80.0, 70.2, 63.0 and so on. If you need to react immediately when the price goes below 44.5, then you would probably be doing data-driven processing. Not event-processing. Each price update is totally anonymous.

If on the other hand you are monitoring the iron ore market to see when three suppliers can offer iron ore for less than 55.0/whatever iron ore is sold as, then you might have an event stream with events like PriceOffer(55.0, SupplierX, TermZ, ValidUntil(2222-01-01)). This would be an unique offer with an id. This is event processing.

Why is this important? Currently the event processing market consists of tools which can do data processing (but sold as event processing tools), a mix of event processing and data processing and pure event processing solutions.  Pick the wrong one and you get the same type of problems as you would when you try to do hard core data analytics with an OLTP database. Simply the wrong tool in a shiny box.

This is also important for the event processing community. If customers try to use event processing for the wrong kind of problems they will just be disappointed and event processing can easily be regarded as nothing more than hype. There is great value in true event processing, the vendors know it. But we need to make sure that the customers get some real business value of our systems and not just another toy.

Apparently there are a increasing number of international companies which consider Sweden the cheapest place to develop software.

I had a interesting lunch conversation with a client and he told me that a number of companies are developing their software in Sweden in order to reduce cost. If you know a bit about the system over here this might come as a surprise. It is totally unintuitive when looking at the facts:
Total tax ratio as percentage of GDP is over 50%. Some highlights:

• VAT (sales tax) 25% on most items.
• 70% of the price of fuel is tax. Diesel is now almost at 15kr/litre ($9.4/gallon). With the nice twist that we pay sales tax in the price including tax. Tax on tax must be a Swedish innovation.
• about 33%-52% of your salary goes to income tax
• A company pays an additional tax of about 30% on a employees salary.
• A company pays normally an additional 5% of the salary into a retirement fund
• 28% of a company’s profits goes to tax.
• Six weeks of summer vacation is normal
• You get payed to stay home with sick children.
• You can stay at home, also payed time, with your children for over a year.

[A common bumber sticker: Born free, taxed to death]

So how could anyone think a country with this kind of structure to be the most cost effective place on the earth to develop hi-tech software?

In order to understand why Sweden might actually be the worlds best place to develop software you have to look beyond hard facts. It’s a result of the Swedish mentality and the way our system works.

First of all, senior software developers are very inexpensive here. The difference in pay between a newbie and someone with 10+ years of experience is rather small. A fact of our socialist history where all swedes were thought that everyone is equal and nobody should receive special treatment, including high salary. The tax on work is also progressive so you will keep less and less of your earnings if you increase your salary. So there is more or less not point in asking for a raise when 50% of that goes to tax. Highly skilled inexpensive developers are the foundation.

The second interesting observation is that the Swedish labor laws makes it more or less impossible to fire someone. This sounds like a bad thing. But consider how software is created. It is a creative process where everyone must be able to argue against a design proposal, even if it comes from your managers manager, and criticizes all bad decisions. When you can’t get fired, this is easy to do and comes naturally. So we have a way of working where everyone feels that they can say whatever they like and still have a job to go to the next day. This creates extremely efficient software development teams.

A consequence of that is that every software engineer, independent of their formal position, can normally make their own decisions in many matters without consulting a manager. The managers would not be able to make an informed decision anyway. Our style is more that we inform our managers what we are up to. This is also a good thing in a software project where every single staff member have to make dozens of decisions every day. Clearing them all with a manager just takes too long time. Also a very efficient way of working.

Also, the high taxes of Sweden forces us to be really good at things where you can create lots of value per hour. Meaning high-tech. Sweden has two car companies, two truck manufacturers. Produces the worlds deadliest sub-marine, have created third generation fighter jet, have a world leading telco, home of IKEA, ABBA, inventor of the tetra pack and so on… All done with a population where the work force consists of five million people. A country of the size of Bangalore and half of the size of New York City.

My own international experience supports all this. A common frustration when we deal with other countries is that everything is so slow and bureaucratic. Think of Sweden as a whole country where the modus operandi of most companies are the one of the innovative and fast moving start-up. Add to that the fact the almost everyone speaks good English.

So if you are a international company with a well defined project and you ask how much it costs to produce a piece of software to a fixed price. Then amazingly we are the world leaders in this. A given specification built to match given quality standards is inexpensive to build in Sweden IF the project is hi-tech and requires substantial amount of creative thinking.

What has this to do with CEP? Probably not a thing, just some personal reflections…

But when dealing with swedes, be aware that we still have some pre-Christian celebration going around here from time to time. So for example, don’t expect anyone to be accessible tomorrow as it’s a public holiday spent in celebrating Midsummer’s Eve. Valborg is another one. Apart from those we are pretty normal.

Not a revolutionary piece of news, but anyway:

Progress Software Corporation (NASDAQ: PRGS), a global supplier of application infrastructure software used to develop, deploy, integrate and manage business applications today announced a technology development collaboration in which the Progress Apama complex event processing (CEP) platform has been ported to the Sun x64 system, running the Solaris 10 Operating System. 

It seems to be CEP release season, a new one from both Progress and StreamBase

What is interesting that the three major points highlighted in the StreamBase release are all ease-of-use features. No mention of new event processing functionality in the release. And no stupid looking performance claims. One interesting point is:

“As the CEP market continues to grow, business analysts and developers demand more sophisticated tools.  StreamBase has always had a singular focus on ease-of-use for business analysts and IT developers to graphically build, deploy, and evolve applications.”

Event processing is difficult to understand so this is a great move for the users. It also seems that StreamBase thinks that business analysts, in addition to developers, are a typical user of CEP tools. Interesting. Great move StreamBase!
So now I’m just waiting for 6.0 of Coral8 and some activity from IBM to see what they have done with AptSoft… And why not something new from Syndera while we are at it… (We have already released ruleCore 9 preview, we just did not tell anybody outside our client base about it… )

I’m just waiting to see amazing GUI tools from all vendors this year. For the sake of the customers, I hope great GUI tools will be the next race. Instead of claims of performance, which Moore will solve for us anyway soon, so anyone trying to live purely by adding performance to the race will loose badly.

Aleri is preparing to release a new version of their streaming data processing platform. There has been lot of discussion in the CEP community if the SQL based approaches (Coral8, StreamBase etc) are suitable for event processing. One of the key features highlighted by Aleri is:

Enhanced scripting – Aleri’s procedural scripting language, SPLASH, which overcomes the limitations of SQL, has been enhanced with new data structures and greater control and is the industry’s only hybrid CEP platform that combines SQL with relational operations. Here

Apparently they feel that SQL is not enough so they will introduce a new scripting language to improve upon the SQL based approach. Sounds like a good idea to me, I hope I can get my hands on this new version and play around a bit with SPLASH (who came up with this name?). It will be interesting to see how this new language feels and what approach they are using.

Visit the Event Processing Technical Society - Here

How error handling is done is one of the most defining aspects of any development language/platform/system.

For example, one of the benefits C++ has over C is the structured (well, more structured than C) error handling with exceptions. Exception handling is a common error handling concept in many of the traditional programming languages. It fits nicely into the call/response cycle. Just call any number of functions/methods and exceptions are raised in case something goes wrong:

try:

function1(1,2,3)

function2(a,v,b)

function3("hello")

except FatalError:

print "Error"

This is an improvement for the old method where the programmer had to check for errors after each call to a function, otherwise the errors were not noticed. At last not directly.

Although clever, this style of error management don’t fit into the asynchronous event driven world of CEP where loosely coupled components acomplish a task by using the publish/subscribe way of indirect communication.

If you are used to error management in a traditional request/response world, be prepared for an unpleasant surprise. Detecting and handling errors are not that simple in the complex world of CEP. If done properly it’s even simpler. If you get your event architecture wrong - It’s a complete nightmare.

The basic processing model of an event driven architecture is:

1. receive event
2. process it
3. optionally, publish event

It’s that simple. Now, things might get FUBAR in completely new ways to a traditional distributed programmer used to web services, (xml)-rpc and other synchronous ways of achieving something:

Missing Events - When creating your processing logic you use more or less implicit assumptions that you will actually receive the particular type of events needed by your logic. What if that kind of event never shows up? Is it ok? Is it annoying? Is it completely wrong? Is it fatal for your continued processing?

Unexpected events - What if you receive an event you don’t expect? Just ignore it? Note it and then ignore it? Just throw it away? Events that you did not anticipate in your initial design might or might not be a problem.

Assumptions not shared by others - If you publish an event and expect that event to lead to some kind of processing and eventually resulting in an event you are waiting for, you might be in trouble. Other components might not at all care about the event you publish. There might not be a single node anywhere expecting the kind of events you publish (see above).  This kind of  (lack of) coordination requires carefully designed event driven systems in order to work properly.

Timing assumptions - Event if you manage to get everything logically connected in the right way, there is another dimension in event processing, and that is time. Do you assume that processing is done in reasonable time as part of your design? Bad idea when it comes to loosely couple components.

Loosing track of state -  If your processing is state dependent, say you like to drive your state forward for each event you receive, then there’s the possibility that you get out of sync with the flow of events. Say you like to detect a sequence of 10 events. When you have seen 8 of them someone kills your application for just a second. During that second you miss event number 9 and will this be waiting for event number 10 forever.

So new failure modes is just one of the things you have to re-learn when getting into event processing. But still, it’s worth it. Event processing is more or less the only way to build really scalable, huge (and I mean huge) system with insane amounts of complexity and processing power.What we see in CEP today is just the first step. If you were around in the early days of databases, think this as the pre-SQL era.

I have been watching the marketing efforts of a number of tech companies for the past years. What strikes me is that I commonly have problems in understanding what their products actually do. Most marketing material seem to be written on a really high level so it could be re-used basically for any tech company. The Dilbert mission statement generator is not too far off.

When there’s some substance, most marketing messages tries to convince me as a reader that their amazing software can do just about anything. Just plug it in and you start making more money. Not likely.
When the marketing is good, and actually manages to convey some usable information about the features and benefits of a product there’s always something missing.

What doesn’t the product do?

When designing software, you always have to draw the line somewhere.

This is probably the number one no-no of marketing. Never tell anyone about the limits of your solution. But I never attended marketing school, instead I studied systems programming. So…

Here’s what ruleCore don’t do, I might be the worlds worst marketing guy:

• No event pull - There’s no way ruleCore can go out and fetch an event. It just sits there and waits for events to arrive. It’s event driven. Also, there is no way to pull an event out of ruleCore. You just have to sit there and wait for a event to arrive from ruleCore.
• No event routing - It’s a black box accepting, processing and producing events. It has no knowledge  of where the events come from or who wants them. We just assume that ruleCore will not be used in a vacuum. Tools like message brokers, ESBs or other MOM infrastructure is much better at dispatching events so we’ll just let them do that instead of re-inventing parts of their functionality inside ruleCore.
• Not a programming language. - Using ruleCore should not be a programming exercise. Actually you can’t write anything that looks like procedural code. The concepts are on a much higher level. Avoiding programming also make things simpler, not as generic as a full blown turing complete programming environment. But more suited for the task for which it was designed for, namely looking for complex event patterns. The drawback is that it’s hard to do things with ruleCore for which it was not designed for. Like brewing coffee.
• Machines create rules. Rules are created by other machines. Albeit in response (sometimes) to human action. But the concept of detecting complex event patterns is just too complex to require end-user to understand it, they should have their mind on their business and not on technicalities of event processing. It’s better left specialists who create small machines which actually create the rules. Rule creation and management should also be very dynamic process. So there could possibly be thousands of new rules created, changed and deleted each day. A task not suitable for humans. So the rule management interfaces in ruleCore are created to be machine friendly, not for humans (programmers love them but they are not the average humans anyway). This means XML with XSD and allowing every possible rule management task to be done remotely using events.

I’m a strong believer that a software solution should stick to doing one thing really good and be designed to nicely fit into a larger picture with other tools which all solve their part of the whole problem. This allows customers to create powerful enterprise architectures by mixing different kind of components. We strive to make ruleCore one of those components focused on tracking and detecting complex event patterns.

This is the kind of job I spent hour daydreaming of when at the university:
“This position requires hands-on development expertise, developing innovative solutions for scalable, high performance distributed systems.” - found here

Now, 15 years later of server side coding I start to realize how hard this stuff is to build…

So if you like to work with CEP and like a challange, apply at StreamBase. Now. My guess is that this is one of the most interesting jobs a software developer could have.

“We are extremely pleased with Coral8 Engine’s agility and flexibility, as well as the responsiveness of Coral8’s engineering organization and technical support team,” said Dan Zulu, head of IT at Nedbank Capital. “During the evaluation period, we developed and deployed in just three months an initial set of market-making applications based on Coral8’s CEP platform. Our next step is migrating existing strategies and algorithms from our legacy arbitrage and basket trading applications to the new architecture. This daunting task would typically take five man-years to complete but can be done in approximately three months on Coral8’s platform. The new CEP infrastructure reinforces our customer-centric business model, allowing us to provide a superior customer experience by delivering more new products more often.”

Nedbank Capital evaluated many US- and UK-based CEP vendors to address a number of business challenges. For instance, having to custom code applications led to costly, long development cycles and slowed time-to-market of new services. Also, Nedbank Capital lacked a shared infrastructure for responding to evolving market dynamics and needed to address the stovepipe applications that were costly to maintain. Finally, limited availability of programmers skilled in both programming and industry-domain knowledge in South Africa made it difficult to staff projects.

Nedbank Capital clearly needed a new platform that would support the development of next-generation market-making applications and the bank’s expansion.

Coral8 Engine was selected after a rigorous evaluation because it meets Nedbank Capital’s requirements:

• Offers a flexible, dynamic server architecture with an agile infrastructure capable of responding immediately to changing business requirements
• Easily integrates with multiple data sources and different trading applications, in a matter of one to two days
• Is powered by familiar, SQL-based Continuous Computation Language (CCL) that reduces the learning curve and speeds development cycles for even the most sophisticated market-making strategies and algorithms
• The CCL syntax relieves the staffing pain by enabling the bank to hire developers with business domain expertise but who may have only high-level programming skills

Equally important to the bank was Coral8’s open evaluation and development business model, giving businesses access – even during the trial phase – to production-grade software and to the technical support team.

“We are delighted to be a strategic part of Nedbank Capital’s expansion and provide the CEP platform that is helping to transform the bank’s trading environment,” said Terry Cunningham, Coral8 CEO. “With Nedbank Capital, Coral8 now serves customers on every continent. Our global deployment gives us insight into a wide range of business and technology issues enterprises are facing worldwide. Ultimately, the knowledge we are building will benefit all our customers.”

Nedbank Group Limited is one of the four largest banking groups in South Africa offering a wide range of business, corporate and retail banking, property finance, investment banking, private banking, foreign exchange and securities trading services. Nedbank Capital comprises the group’s investment banking businesses that together manage the structuring, lending, underwriting and trading businesses.

About Coral8

Based in Mountain View, Calif., Coral8, Inc. is a leading provider of software for Complex Event Processing (CEP). Bringing together high-performance, innovative SQL-based programming language and enterprise-class features, Coral8 Engine is the fastest, lowest cost way to build and deploy powerful, sophisticated CEP applications. Coral8 is speeding the delivery of critical business information for customers worldwide, including Fortune 500 companies and global leaders within financial services, e-commerce, telecommunications, transportation, government and other rapidly growing vertical applications. For more information, visit www.coral8.com or call (650) 210-3810.

Coral8, Inc.
Anna Luo, 650-210-3819
Director of Marketing
anna@coral8.com
or
Schwartz Communications
Jill Reed or Katy Garlinghouse, 415-512-0770
coral8@schwartz-pr.com

In an article in Wall Street Technology, Iwy Schmerken writes about end user’s perception of CEP:

“One of the main myths is that the technology is mature. According to panelists, CEP technology is immature. Another myth is that it’s a sell-side only technology. A third myth is that it’s out-of-the box and easy to use.”

It seems that the end-users are not swallowing the marketing hype. Which is good. Realistic expectations on CEP technology is the best thing that can happen to the CEP vendors. Even if some marketing efforts wants the customer to think that CEP is the solution to basically all problems, except perhaps carbon dioxide emissions.

We have a slightly different marketing culture here in Europe. So marketing messages from US companies are also many times perceived to be full of, eh, a bit unrealistic… A hint to CEP vendors, marketing is a bit different over here so use a local firm to adjust your message. And no, EU is not an unified entity. All countries are different and need local representation. You need to support local languages to be really good over here. There are actually laws in some countries that local language must be used in official documents.
Back to CEP - CEP marketing of some of the leading vendors says that their technology is mature. Using corporate marketing math that might be true. But customers tend to favor the more traditional arithmetic methods. So when a company founded some five years* ago talks about mature technology most customers sees a problem with that message…

Also you might want to consider that the Internet generation is now in charge for many decisions. Easy access to evaluation software and fairly transparent companies gets high scores in evaluations. We are used to open source and total insight into everything around a product. Like future plans, bug databases and access to developers. Don’t miss this one. It’s more important than you think.

Enough marketing. Back to more interesting technical issues like how to scale ruleCore on a 100 server grid…
* I have a personal theory that it takes ten year to build a really good (market leading) piece of software.

By way of my trusted friend Google Alerts I just found this article:
http://www.securitiesindustry.com/news/22370-1.html:

“Sybase RAP Trading Edition streams real-time market data with submillisecond latency, while transaction information moves at speeds ranging from sub-seconds to seconds and analytical data on disks can take minutes to reach applications. Sybase expects the platform will be used for high-performance quantitative analysis, real-time trade analytics, intraday risk analysis and regulatory-driven analytics.”

Sounds like CEP to me. I see two options. Either they built something themselves or did something smarter like OEMed a CEP engine under the hood and just added some user visible parts. My best guess would be Coral8. That’s the only engine in the marketplace that I know of that would be a good solution to build something like this on.

Even if I’m totally into CEP technology and like CEP partially for the cool technology I like when the benefits of CEP is visible in new software offerings, but presented in a more customer friendly way. Customers tend to favor spending € on solutions to problems and not just another piece of cool technology.

Another totally CEP unrelated reflection is that this is available for Linux and Solaris. Where did Windows go? It seems not even be an option anymore for new software offerings. When I think of it, I can’t remember a single startup building their future on Windows.

If you think current CEP products are fast, think again:

“The PL10000 can handle up to 5 million subscribers and can track 48 million real-time data flows.” - here

ok, so the thing is built to inspect network packets. But that’s pretty close to event processing. Would it not be nice if this thing could be updated with new software for algo trading or some other high performance CEP task:

“And because the PL10000 doesn’t use custom ASICs to get the job done, protocol identifications can be updated in software just as fast as Procera can issue them”

Suddenly all the existing CEP products seems veeery veeery slow. When Procera realizes that they can slap a new sticker on this beast saying “Monster Event Processor” and write some new software they have a really cool CEP product.

http://www.proceranetworks.com/products/pl10000.html

Or maybe they already have half of that in place:

“DRDL™ (Datastream Recognition Definition Language) is at the core of what makes PacketLogic stand out from the competition. This state-of-the-art traffic identification and classification engine drives all PacketLogic platforms from the PL10000 to the PL5600.”

Maybe I should OEM this thing and turn it into a nice cool CEP product myself…

It seems that the ‘ol Mighty Gartner started to look at this side of the great pond too:

“Thanks to its fascinating BI real-time solution Gartner Inc. has just voted Austrian software company SENACTIVE most convincing and innovative solutions provider for business intelligence and performance management in their Cool Vendor Report 2008.” - Senactive Press release

It’s nice to see that European companies also get some attention even though the center of gravity still seems to be on the west coast of US.

Dennis Byron from ebizQ have also found us Europeans, he has written a small summary of event processing and open source and you can find his findings here. Annoyingly you have to register to read that article. Short summary would be:

“As with many open source activities, there is a strong European flavor with Paremus and Seewhy based in the UK, ruleCore based in Sweden, and Senactive in Austria. On the other hand, you cannot get more American-based than EsperTech of Wayne, NJ, USA.”

But it’s worth registering to read the complete article.

One positive trend that I have spotted when reading articles like these from analysts is that they are at last starting to get it right! Now the writers own conclusions are more or less consistent with reality and the way vendors, products and the CEP field is presented starts to sound really good with less misunderstandings about what we actually do. A great improvement from a couple of years back.

I just got an e-mail from Dr. Jürgen Krämer asking their company to be included in my listing of CEP vendors. Which I gladly do. It’s nice to see that the CEP market is growing especially here in Europe. Their product seems to be using the SQL approach, like Coral8, Esper, BEA, StreamBase and Aleri. I have not checked in detail how their offering looks like but it’s starts to be interesting to see that there are so many SQL based products now so that it would actually be a good idea to compare them all. Here’s an short description of their company - I wish them best of luck.

We are currently looking for a person to join us to take charge of our sales and marketing effort.

We have a really great product and powerful concept. Now we need great marketing and sales.

The right person would know a lot of the CEP world and would have lots of ideas on how to market and sell  something like ruleCore. We need someone who can work both on our strategic goals and is really good at closing deals. You probably know a bit about ruleCore already and have tons of ideas on how to turn this into a profitable company.
What we can offer is a partnership into a company well positioned to be part of what Gartner says will be
the Next Big Thing. We have a great product and there’s more exciting things developed as we speak.

We like to think in new creative ways - after all its 2008, and marketing and sales can be done in many creative ways these days. New ideas are very welcome, we appreciate new creative minds. For example your geographical location is not very critical and this might also be an opportunity for companies willing to take to role of our sales/marketing partner.

Interested? Contact me at marco@rulecore.com

As I wrote earlier I have been playing around with some ideas around the Semantic Web and how to turn these into Semantic CEP. I got a comment from Jack Rusher from Aleri who also seems to have some Sementic CEP ideas.

My first idea how to use this is to be able to categorize events dynamically as they happen. For instance assigning dynamic roles to entities from which the events are notifications from. This could allow us to create semantically related groups of events which are dynamically updated as new inbound events arrive. (in the ruleCore world time is events too so we don’t have to think about temporal aspects separately).

This is one of the simplest uses I have in mind for adding “Semantic CEP” into ruleCore. What I’m experimenting, still a napkin product so don’t hold your breath, with it to find a simple model where the user don’t have to care about the details of all the semantic technology beneath.

It will be interesting to see if Tibco, BEA, Apama or IBM will produce something in this area soon, which I predict they will. Or perhaps the smaller vendors - Aleri, Coral8, StreamBase, Syndera - will get there first as innovation is not what the big ones are good at. But there’s lot of thinking in the semantic web community so for this there would possible by less innovation to do and more engineering, which at the big companies are good at.

Was Apt-Soft, now  WebSphere Business Events
http://www-306.ibm.com/software/integration/wbe/

Another proof that, some some reason, innovation seems impossible to achieve in big companies. It’s easier to buy a small innovative company than to build something yourself, even if you have great research (IBM Labs Haifa) in the area…

Sometimes it’s nice to get verification from third parties that we are on track:
http://www.ebizq.net/blogs/bethgb/archives/2008/04/bi_meets_event.php
As you might have noticed our marketing message is centred around operational BI. We designed ruleCore just for this particular area* so it’s nice to see that others agree with us that operational BI is one interesting area for CEP.

Many of the offerings from different vendors are complete event processing solutions usable for almost any event processing task. We have on the contrary designed ruleCore to target operational BI developers.

The drawback is that ruleCore is not too good at, for example, algo trading, but the benefit is that we try to be really, really good at supporting operational situation tracking and instant detection of exceptional conditions.

* Actually it was called BAM - Business Activity Monitoring  - back then. But the concepts are pretty much the same. Perhaps with a shift from real time dashboards (who has the time to watch those anyway?) to automatic tracking and tracing of interesting things.

Company blogs tends to be rather boring. Nothing much than poorly camouflaged attempts at marketing. Lately I have been reading Aleri’s blog which actually contain a lot of interesting stuff.

For example, Jack Rusher writes about Semantic CEP, or whatever we should call it. I think this is the next cool thing from many CEP vendors. I’m pretty sure that there’s activity within this field at many of the CEP vendors as we speak.

It will be interesting to see who will get there first and announce support for true semantic processing. I have been having the same ideas on my drawing board for a some years now. I got a lot of ideas by taking part in the REWERSE, a “network of excellence” funded by the EU commission and Switzerland.

There’s lots of things happening in semantic technology in the scope of the web. I think we in the CEP community can use lots of this stuff in our world. Like OWL, RDF, SPARQL.

I think this is a interesting area and will give a lot of new interesting capabilities. For example it could use ontologies to process events related to certain areas, for example everything considered to be a threat to public health.

With Semantic CEP in place we could easily describe situations like “more than five events related to public health during the last hour”.

Semantic CEP will allow us to describe situations in a way that is more relevant to end-users.

I’ll continue looking into this field to see how we can put some of this stuff into ruleCore. The hard part is how to create usable abstractions which are not too hard to use without being an semantic technology expert. The terminology in this field can be a bit confusing and terrifying.

My first idea is to put some semantic technology into our event stream views, so one could maintain dynamic views into the inbound event stream based on semantic properties found in the event payload.

I just stumbled (well, Google Alerts helped out a bit) onto a job ad for a pretty cool job, just look at the spec:

1. Experience with Tibco BusinessEvents or other event process tool - 2+ years
2. Experience with JMS and Message Queues Architectures - 4+ years
3. Experience with Java and J2EE - 2+ years
4. Development experience in XML messaging, XLST, XSD and XPATH.
5. Experience of using and interpreting UML / Use-Case analysis and functional specifications to design solutions a plus - 2+ years
6. Experience in SOA framework architecture.

Two year CEP plus some additional pretty heavy stuff. This is for me typical senior consulting stuff. And what’s the pay? $50 (about €32)!!! I would expect closer to $200/h for this kind of position… Is the consulting business in the US this bad?

The job

But anyway, I’m glad to see that CEP experience found its way into wanted ads….

The last weeks have been very exiting with great response to our announcement of ruleCore CEP Server 2008. Hopefully all the work that’s been put into the product, which I personally think is really great, can solve some interesting problems at our clients.

It seems that many CEP vendors are pushing performance as their main selling point. The ruleCore as a Service or CEP as Saas as the marketing nightmare term is, tries to go the other way. Bring inexpensive and simple event processing to everyone. The only fear is that we will be seen as a simpler low cost alternative. Low cost is ok, but as for situation detection I actually think we are one of the leaders among the CEP vendors.
It will be very interesting to see how this is received by our customers. Just pay as you go and don’t care about servers, upgrades or anything. We host everything for them.

At 0.0001 euro/event I think it’s a pretty nice price too.

The most common question is about security and I think we have to create a ssh tunnel or something like that for those customers that likes to have some privacy. It’s been on the drawing board for some time so it might be the time to actually implement that feature now.

There’s more to come from us too…. We are working on a dot release due in just a month or so. A collection of the normal fixes and improvements that did not make it to the 8 release and a few minor (but usable) new features. It’s mainly a performance, stability and documentation improvement release as I see it.

On a unrelated note, SPRING is at last here in Sweden!!! Now it’s time to go out and enjoy the sun for a while

Göteborg, Sweden, March 20, 2008 - ruleCore, a leader in rule based complex event processing technology (CEP) for real-time operational intelligence, today announced availability of ruleCore CEP Server 2008.

In this latest version of its flagship product, ruleCore users gain the advantage of receiving actionable alerts immediately when any process, item or resource is behaving in an exceptional manner.

Because of the advanced and flexible design, the ruleCore CEP Server is uniquely suited to provide reactive capabilities into any Event Driven Architecture (EDA) and act as a critical building block for Event Driven SOA implementations.

“Competitive pressures are forcing organizations to react faster to multiple aspects of business process execution in a high paced operational environment. As a result, there is now a need automate operational business decisions in order to assist and optimize business operations on a daily basis and to support intraday decision making”, says Marco Seiriö, CEO of ruleCore.

The core capabilities of the ruleCore CEP Server allows developers of operational intelligence solutions to add a stable and powerful foundation for their real-time intelligence solutions, allowing to bring advanced business event driven solutions to horizontal markets in the shortest possible time.

The ruleCore CEP Server brings a number of important capabilities for the agile low latency enterprise:

• Business Event Fusion – Business event fusion is the process of combining events from disparate business systems. The resulting situational awareness and situational detection capabilities are superior compared to individually processing these events.
• Semantic Event Correlation – Tracking of the progress of business situations are executed in a semantic context maintained based on event semantics, temporal constraints, logical relationships and event sequencing. Giving unprecedented insight into complex operational activity.
• Temporally Aware – Any timing constraints that your operational activities are subject to can easily be expressed and tracked within a temporal context.
• Event Causality Tracking – The ruleCore CEP Server keeps automatically track of event causality and adds meta data information to each outbound event to indicate what event caused it.
• Situational Awareness – The ruleCore CEP Server can be monitored in real-time to provide insight into the situation detection process by tracking every situation as it develops. This allows for easy monitoring of the behavior or changes in a particular process, item or other business entity and the capability to provide early warnings as exceptional, threatening or profitable situations develop.
• Pluggable Connectivity – A flexible and powerful connectivity layer provides support for all popular event transport protocols. Customized event flows and distributed configurations can easily be created without any programming.
• Persistence for Long Lived Process Tracking – The state of tracking long lived situations is automatically managed by advanced persistence and recovery procedures which guarantees the integrity of all internal data structures. The automatic failure recovery procedures protects the integrity of long lived rule instances and ensures that all events and rule instances are automatically restored to the same state as before the failure. As a result, state or events are never lost.

About ruleCore

RuleCore is a leading provider of complex event processing software for operational business intelligence that delivers real-time behavior monitoring of automated processes. The ruleCore CEP Server allows users to monitor, analyze and act on operational situations as they happen to establish a responsive, competitive business. Visit the ruleCore web site at http://www.rulecore.com

Interested in the state of art in rule based event processing? So are lots of other people that you can meet at RuleML 2008:

http://2008.ruleml.org

So set aside October 30-31 for this event. Then boot your typewriter, submit a paper and create a demo for the RuleML 2008 Challenge.

Here’s an interesting comment on a comment: http://blog.aleri.com/?p=33

In true blog style, I’d like to make a comment on this comment on the comment.

For anyone building CEP engines the question makes sense. It’s so obvious that this might be a problem so there’s no explanation on why this is a problem in the first place. For an outside viewer (customer, user, interested techie) this sound really strange I think.

What? Can a CEP engine be indeterministic? Is it broken or what?

The thing with CEP engines is that they try do do lots of stuff at the same time using threads mostly. So two event streams producing outputs are commonly processed independently of each other. So when looking at the output from these two streams you see different results depending on how fast they do processing. To fix this you need to do expensive synchronization at the end but this slows things down so you end up doing all kinds of fancy stuff to keep things synchronized and fast.

That’s how the stream oriented folks at Coral8, StreamBase and Aleri do it. Here at ruleCore HQ we just avoid the problem by using a single unified event stream when we do processing. In our rule processing model we don’t have the stream abstraction. We just see the world as a single stream of events and ignore where they are coming from.

PS: The ruleCore CEP Server can be added to the 100% deterministic and consistent list. We can even manage temporal dependencies correctly when the events are played back at a different rate. That’s really cool btw…

http://blogs.progress.com/soa_infrastructure/2008/03/the-emergence-o.html

When I read something as positive as this about CEP in a blog about SOA I see it as a good indication that out small world is slowly starting to expand. It seems that CEP is starting to get attention very rapidly outside the small community of CEP vendors and believers.

Following on the Coral8 announcement I saw that EsperTech is announcing EsperHA.

This seems to be their commercial version of the normal Esper which is given away for free. It contains a number of really cool features.

The HA version seems to add a lot a features which has to do with backing storage and peristence. All needed for high availablility I suppose. The coolest feature in this advanced version is the ability to persist state and not loose query processing state in case of failure.

I know Coral8 have the same kind of feature too, but theirs do not guarantee complete recovery from a crash. I wonder if EsperHA does that? We have the same feature in ruleCore and it restores state completely. Very nice when tracking long lived process oriented situations where it is not an option to loose an event or state of the situation detection progress.

Also a cool feature of EsperHA seems to be that they can somehow use disk to back up an overflowing system into a database. I wonder which one they use. They are rather vague on that point “EsperHA is optionally extensible to accommodate with third parties event storage mechanism although market proven storage technology is included into it for out of the box capabilities.”. My guess is that is not any market leading SQL database or otherwise they would tell us about it. And what does “market proven” mean anyway? We are using (now Oracle) BerkeleyDB and maybe they are smart enough to use it too?
Nice work Esper team, I really like what I see and I’m pretty sure it will give Coral8, StreamBase and Aleri something to think about. (and soon WebSphere Business Events of whatever it will be called)

(To add to this flood of announcements, expect soon to see one from us at ruleCore… Previews are available for those interested…)

There’s a new Coral8 release coming on Monday. It have taken a quick look at it and it contains some really cool features.

I’ll just ignore the normal improvements in fancy graphical interfaces, charts, SDKs and so on.
Personally I think the improvements in the core product is more interesting to look at from a CEP perspective.

One thing I notice is that Coral8 talks more and more about events and less about streaming data. But still they mix “tuples” and “events” in the documentation. But their model works really good whichever view you choose. But CCL is described as a data stream processing language in the documentation so this might confuse you. Just think of a tuple as another name for an event.

#1 The cool feature numero uno is an addition to track the cause of a pattern triggering. It allows you to see which events caused an pattern to trigger. This is very much event thinking and less data stream thinking. It makes more sense to track causality when the inbound data is events as events are commonly seen as having an identity.

#2 Cool feature number two is that you can now query the contents of a window. Nice.

#3 Patterns that match only once for an event is third on my list of cool features.

#4 is a new window policy. It allows you to keep tuples with identical values in some columns in the window.

I’ll be back (If I have the time) with a closer look on some more interesting CCL features soon…

This blog post over at Progress Apama was really a nice boost for my ego

http://apama.typepad.com/my_weblog/2008/02/cep-what-about.html

I was just wondering if we would need to extend to action part of the ruleCore CEP Server. It seems that we share the views on how an action must be done. *great relief*

The Apama team seems to know what they are doing so it’s nice that they write stuff that goes a bit along the ways we think up here in the north.

In the ruleCore CEP Server we are executing an XSLT transform in the context of all the contributing events and in the context of the current state of the event stream view (a windows into the stream of inbound events) as the action part of a rule. I think it more or less covers all the points raised in this post.

Also, ruleCore is event driven and the XSLT spits out a new event which in turn could create, delete or modify rules. Neat

Still we manage to stay completely declarative. Not being turing complete and not being able to do any kind of procedural processing is actually a design feature of ours. More on that later…

But I’m not sure if I agree that the SQL based approach is less powerful. If you for example look at the CCL of Coral8 you can basically do anything in it. It just looks a bit different. I’m sure the others are as powerful. I think it’s a mistake (or marketing) to equate the SQL based approaches with less powerful solutions. The are more or less generic programming environments based on SQL and retaining the look-and-feel or SQL but totally different beasts when you start looking at the details.

Basically the action part can be designed to be as complex as you like in a CEP rule processor such as ruleCore or Apama. But there is also a possibility to create a completely stand alone action execution engines. Just pick up the complete context of the rule’s evaluation and hand over that to the action evaluation engine.

In order to execute really complex action I think it might be a good idea, In the name of simplicity, to consider a separate entity which executes the action.

Separating rule evaluation from action execution might provide a cleaner design and more powerful components if you just let them solve one particular problem and not both. But I think nobody has built such an action execution engine yet.