SYS-CON MEDIA Authors: Stackify Blog, Zakia Bouachraoui, Elizabeth White, Pat Romanski, Liz McMillan

Blog Feed Post

Let’s Get Esper Up & Running

For our CEP in the Cloud example, I’ve briefly outlined a stream based load balancing idea.  In this example, RuleBots (pieces of code that do something – think of the procedural extensions built into some vendor’s products) send utilization statistics to the CEP Load Balancer via RabbitMQ.  The CEP Load Balancer in this case is written in Java and uses Esper to create a stream containing all available processes (destinations) in a particular service pool.

What’s a Service Pool?

For this example, we define a service pool as available compute resources in the cloud available to perform a particular service.  In our example, where we’re implementing map/reduce and the classic word count tutorial, our service pool name is “WordCountMap.”

How Does It Work?

Our Twitter OnRamp, which listens to the Twitter stream, will publish the tweet via RabbitMQ.  The RabbitMQ queue that the OnRamp will use as the destination will be retrieved from the CEP Load Balancer.

Why Is This Cool?

Because the CEP Load Balancer doesn’t need to know all that much about any particular process – only that it’s available to do work.  Also, the Twitter OnRamp doesn’t have to embody some load balancing algorithm.  So if we decide to change how we do load balancing in our word count example, we make the change in one place.  And by avoiding partitioning at the RuleBot level, we maintain some flexibility – more processing power required?  Add processes and there’s no need to stop services, re-partition, and restart the services.  Easy peasy!

Esper Code

Here’s some example Esper code – load this up to get started.  We’ll modify it over time.  In the next installment, we’ll modify this code to receive utilization statistics from RuleBots. (if anyone knows a better way to make code look better in my posts, let me know – my html ‘code’ formatting doesn’t seem to be working – wordpress 2.9.2).

import com.espertech.esper.client.*;
import java.util.Random;
import java.util.Date;

import org.apache.log4j.ConsoleAppender;
import org.apache.log4j.SimpleLayout;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;

public class CEPLoadBalancer {

public static class Utilization {
String destination;
String servicePool;
Double load;
Date timeStamp;

public Utilization(String Des, String Ser, Double Ld, long t) {
destination = Des;
servicePool = Ser;
load = Ld;
timeStamp = new Date(t);
}

public String getDestination() {return destination;}
public String getServicePool() {return servicePool;}
public Double getLoad() {return load;}
public Date getTimeStamp() {return timeStamp;}

@Override
public String toString() {
return "Destination: " + destination + " ServicePool: " + servicePool + " Load " + load.toString()+ " Time " + timeStamp.toString();
}
}

private static Random generator = new Random();

public static void GenerateRandomUtilization(EPRuntime cepRT) {

// first map server
double load = (double) generator.nextInt(10);
long timeStamp = System.currentTimeMillis();
String destination = "MAP01";
String servicePool = "WordCountMap";
// create utililzation event
Utilization util = new Utilization( destination, servicePool, load, timeStamp);
cepRT.sendEvent(util);

// second map server
destination = "MAP02";
load = (double) generator.nextInt(10);
timeStamp = System.currentTimeMillis();
util = new Utilization( destination, servicePool, load, timeStamp);
cepRT.sendEvent(util);
}

public static class CEPListener implements UpdateListener {

public void update(EventBean[] newData, EventBean[] oldData) {
System.out.println("Event received: " + newData[0].getUnderlying());
}
}

public static void main(String[] args) {
SimpleLayout layout = new SimpleLayout();
ConsoleAppender appender = new ConsoleAppender(new SimpleLayout());
Logger.getRootLogger().addAppender(appender);
Logger.getRootLogger().setLevel((Level) Level.WARN);

Configuration cepConfig = new Configuration();
cepConfig.addEventType("Util", Utilization.class.getName());
EPServiceProvider cep = EPServiceProviderManager.getProvider("myCEPEngine", cepConfig);
EPRuntime cepRT = cep.getEPRuntime();

EPAdministrator cepAdm = cep.getEPAdministrator();
EPStatement cepStatement = cepAdm.createEPL("select * from Util.win:length(5)");

cepStatement.addListener(new CEPListener());

// simulate loads
for (int i = 0; i < 10; i++) {
GenerateRandomUtilization(cepRT);
}
}
}

Read the original blog entry...

More Stories By Colin Clark

Colin Clark is the CTO for Cloud Event Processing, Inc. and is widely regarded as a thought leader and pioneer in both Complex Event Processing and its application within Capital Markets.

Follow Colin on Twitter at http:\\twitter.com\EventCloudPro to learn more about cloud based event processing using map/reduce, complex event processing, and event driven pattern matching agents. You can also send topic suggestions or questions to [email protected]

Latest Stories
The benefits of automated cloud deployments for speed, reliability and security are undeniable. The cornerstone of this approach, immutable deployment, promotes the idea of continuously rolling safe, stable images instead of trying to keep up with managing a fixed pool of virtual or physical machines. In this talk, we'll explore the immutable infrastructure pattern and how to use continuous deployment and continuous integration (CI/CD) process to build and manage server images for any platform....
AI and machine learning disruption for Enterprises started happening in the areas such as IT operations management (ITOPs) and Cloud management and SaaS apps. In 2019 CIOs will see disruptive solutions for Cloud & Devops, AI/ML driven IT Ops and Cloud Ops. Customers want AI-driven multi-cloud operations for monitoring, detection, prevention of disruptions. Disruptions cause revenue loss, unhappy users, impacts brand reputation etc.
Atmosera delivers modern cloud services that maximize the advantages of cloud-based infrastructures. Offering private, hybrid, and public cloud solutions, Atmosera works closely with customers to engineer, deploy, and operate cloud architectures with advanced services that deliver strategic business outcomes. Atmosera's expertise simplifies the process of cloud transformation and our 20+ years of experience managing complex IT environments provides our customers with the confidence and trust tha...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
Enterprises are adopting Kubernetes to accelerate the development and the delivery of cloud-native applications. However, sharing a Kubernetes cluster between members of the same team can be challenging. And, sharing clusters across multiple teams is even harder. Kubernetes offers several constructs to help implement segmentation and isolation. However, these primitives can be complex to understand and apply. As a result, it’s becoming common for enterprises to end up with several clusters. This...
Is advanced scheduling in Kubernetes achievable?Yes, however, how do you properly accommodate every real-life scenario that a Kubernetes user might encounter? How do you leverage advanced scheduling techniques to shape and describe each scenario in easy-to-use rules and configurations? In his session at @DevOpsSummit at 21st Cloud Expo, Oleg Chunikhin, CTO at Kublr, answered these questions and demonstrated techniques for implementing advanced scheduling. For example, using spot instances and co...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Public clouds dominate IT conversations but the next phase of cloud evolutions are "multi" hybrid cloud environments. The winners in the cloud services industry will be those organizations that understand how to leverage these technologies as complete service solutions for specific customer verticals. In turn, both business and IT actors throughout the enterprise will need to increase their engagement with multi-cloud deployments today while planning a technology strategy that will constitute a ...
GCP Marketplace is based on a multi-cloud and hybrid-first philosophy, focused on giving Google Cloud partners and enterprise customers flexibility without lock-in. It also helps customers innovate by easily adopting new technologies from ISV partners, such as commercial Kubernetes applications, and allows companies to oversee the full lifecycle of a solution, from discovery through management.
Using serverless computing has a number of obvious benefits over traditional application infrastructure - you pay only for what you use, scale up or down immediately to match supply with demand, and avoid operating any server infrastructure at all. However, implementing maintainable and scalable applications using serverless computing services like AWS Lambda poses a number of challenges. The absence of long-lived, user-managed servers means that states cannot be maintained by the service. Lo...
Today most companies are adopting or evaluating container technology - Docker in particular - to speed up application deployment, drive down cost, ease management and make application delivery more flexible overall. As with most new architectures, this dream takes significant work to become a reality. Even when you do get your application componentized enough and packaged properly, there are still challenges for DevOps teams to making the shift to continuous delivery and achieving that reducti...
Using serverless computing has a number of obvious benefits over traditional application infrastructure - you pay only for what you use, scale up or down immediately to match supply with demand, and avoid operating any server infrastructure at all. However, implementing maintainable and scalable applications using serverless computing services like AWS Lambda poses a number of challenges. The absence of long-lived, user-managed servers means that states cannot be maintained by the service. Lo...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Signs of a shift in the usage of public clouds are everywhere. Previously, as organizations outgrew old IT methods, the natural answer was to try the public cloud approach; however, the public platform alone is not a complete solution. Complaints include unpredictable/escalating costs and mounting security concerns in the public cloud. Ultimately, public cloud adoption can ultimately mean a shift of IT pains instead of a resolution. That's why the move to hybrid, custom, and multi-cloud will ...
Docker and Kubernetes are key elements of modern cloud native deployment automations. After building your microservices, common practice is to create docker images and create YAML files to automate the deployment with Docker and Kubernetes. Writing these YAMLs, Dockerfile descriptors are really painful and error prone.Ballerina is a new cloud-native programing language which understands the architecture around it - the compiler is environment aware of microservices directly deployable into infra...