SYS-CON MEDIA Authors: Liz McMillan, Zakia Bouachraoui, Elizabeth White, Pat Romanski, Yeshim Deniz

Blog Feed Post

Twitter in the Data Center: A model for data consumption

Twitter has been ridiculously successful at embedding itself into the lives of hundreds of millions. Part of its success is that the service lends itself to a variety of use cases depending on its users' consumption models. These use cases are actually a social manifestation of common data sharing practices. And the same models that helped Twitter raise close to $2B in its IPO are relevant across the infrastructure that makes up all data centers.

Twitter is essentially a message bus. Individual users choose to publish when they want to, and can subscribe to content that is important to them for some reason. Content can be consumed in whatever way makes sense for the subscriber – in realtime, at set intervals, when it is directed to them specifically, or whenever something particularly interesting is going on. The magic in Twitter is in relaying information, not in dictating a specific consumption model or requisite set of actions as a result of any of that content.

The same consumption models exist in the data center.

  • The update: When something important happens, it sometimes makes sense to let everyone know. When a new application instance is deployed, it likely starts with the server. The act of setting up the server generates information that might be of interest to other elements within the data center. The specific application might require some allocation of storage or some network configuration (VLAN, ACL, QOS, whatever). By sending out a general update, followers can take appropriate action to ensure a more automated and orchestrated response.
  • The follow: Not all constituents are interesting to everyone. It might not matter to the load balancers what the application performance monitoring tools are doing. Rather than clutter their data timeline, they follow only those elements that are producing content that is relevant to them. This simplifies data consumption and reduces overhead on the subscriber side.
  • The list: It could be that there are lots of interesting sources to follow, but the sum of all of the updates is overwhelming. In this case, updates can be grouped into relevant streams, each of which is consumed differently. It might be sufficient to simply monitor some updates while other updates require careful consideration and subsequent action. For instance, it might be interesting for servers to monitor changes in network state but not necessarily meaningful to act on all changes. Additionally, some streams might require more constant attention with tighter windows around activity, while others can be periodically parsed for general updates.
  • Intermittent monitoring: Some entities might only parse relevant updates periodically. It is not important to stay up-to-date in realtime, and it might not even be important to pay careful attention to every update. They want to consume content asynchronously and in batches. Analytics tools, for example, might be able to poll periodically and report overall health without needing to consume a realtime feed.
  • Trendspotting: Individual updates are interesting, but when multiple sources all report the same thing, it becomes newsworthy. An error message, for example, might indicate a random issue. But a flood of error messages from multiple data center entities might indicate more serious issues that require attention (perhaps a DDOS attack)
  • Message threading: The threading function within Twitter is simply a sort to help provide context and preserve temporal order around some exchange. This is very similar to reviewing changes or state information during common troubleshooting tasks.  

The thing of central importance in all of the consumption models is the data. In Twitter's case, the 140-character update is the data. The users determine what that data is, with whom that data is shared, and ultimately how that data is consumed. Twitter neither produces the updates nor consumes them. Its sole function is to relay those updates to the appropriate subscribers and to allow data access to those doing searches. 

When this is working well, Twitter's message bus is a powerful enabler of human orchestration. Twitter's role in the Arab Spring uprisings has been well-documented. Entire movements have been coordinated across the globe using Twitter as a means to broadcast organizing thoughts. In most of these cases, the origin of the information was not even directly connected to its recipients. Merely publishing information was enough to spur action.

When our industry talks about orchestration in the data center, it need not be that different. Orchestration doesn't require a tight linkage between all elements within the data center ecosystem. Orchestration only requires that data be made available as and when it is needed. The rules for data consumption ought not be uniformly applied. Individual elements will consume information in different ways depending on what their needs are. 

This is all to say that delegating application workloads to resources across the data center does not rely on the existence of a tightly-integrated system. Integration and orchestration serve different needs. Integration is about performance – controlling both sides of an interface allows for fine-grained optimization required to eke out every last bit of performance available. Orchestration is about seamless handoff between resources. 

The SDN movement broadly can be applied to both performance and workflow automation. Different use cases demand one, the other, or both. But architects and administrators will be best served by explicitly determining whether their objective is integration or orchestration. The differences go well beyond semantics. The architectural implications are profound.

[Today's fun fact: Canadian researchers have found that Einstein's brain was 15% wider than normal. And you thought it was the hair.]

The post Twitter in the Data Center: A model for data consumption appeared first on Plexxi.

Read the original blog entry...

More Stories By Michael Bushong

The best marketing efforts leverage deep technology understanding with a highly-approachable means of communicating. Plexxi's Vice President of Marketing Michael Bushong has acquired these skills having spent 12 years at Juniper Networks where he led product management, product strategy and product marketing organizations for Juniper's flagship operating system, Junos. Michael spent the last several years at Juniper leading their SDN efforts across both service provider and enterprise markets. Prior to Juniper, Michael spent time at database supplier Sybase, and ASIC design tool companies Synopsis and Magma Design Automation. Michael's undergraduate work at the University of California Berkeley in advanced fluid mechanics and heat transfer lend new meaning to the marketing phrase "This isn't rocket science."

Latest Stories
The KCSP program is a pre-qualified tier of vetted service providers that offer Kubernetes support, consulting, professional services and training for organizations embarking on their Kubernetes journey. The KCSP program ensures that enterprises get the support they're looking for to roll out new applications more quickly and more efficiently than before, while feeling secure that there's a trusted and vetted partner that's available to support their production and operational needs.
Serverless Architecture is the new paradigm shift in cloud application development. It has potential to take the fundamental benefit of cloud platform leverage to another level. "Focus on your application code, not the infrastructure" All the leading cloud platform provide services to implement Serverless architecture : AWS Lambda, Azure Functions, Google Cloud Functions, IBM Openwhisk, Oracle Fn Project.
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. Kubernetes was originally built by Google, leveraging years of experience with managing container workloads, and is now a Cloud Native Compute Foundation (CNCF) project. Kubernetes has been widely adopted by the community, supported on all major public and private cloud providers, and is gaining rapid adoption in enterprises. However, Kubernetes may seem intimidating and complex ...
As you know, enterprise IT conversation over the past year have often centered upon the open-source Kubernetes container orchestration system. In fact, Kubernetes has emerged as the key technology -- and even primary platform -- of cloud migrations for a wide variety of organizations. Kubernetes is critical to forward-looking enterprises that continue to push their IT infrastructures toward maximum functionality, scalability, and flexibility.
DevOps has long focused on reinventing the SDLC (e.g. with CI/CD, ARA, pipeline automation etc.), while reinvention of IT Ops has lagged. However, new approaches like Site Reliability Engineering, Observability, Containerization, Operations Analytics, and ML/AI are driving a resurgence of IT Ops. In this session our expert panel will focus on how these new ideas are [putting the Ops back in DevOps orbringing modern IT Ops to DevOps].
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Intel is an American multinational corporation and technology company headquartered in Santa Clara, California, in the Silicon Valley. It is the world's second largest and second highest valued semiconductor chip maker based on revenue after being overtaken by Samsung, and is the inventor of the x86 series of microprocessors, the processors found in most personal computers (PCs). Intel supplies processors for computer system manufacturers such as Apple, Lenovo, HP, and Dell. Intel also manufactu...
The use of containers by developers -- and now increasingly IT operators -- has grown from infatuation to deep and abiding love. But as with any long-term affair, the honeymoon soon leads to needing to live well together ... and maybe even getting some relationship help along the way. And so it goes with container orchestration and automation solutions, which are rapidly emerging as the means to maintain the bliss between rapid container adoption and broad container use among multiple cloud host...
Serverless applications increase developer productivity and time to market, by freeing engineers from spending time on infrastructure provisioning, configuration and management. Serverless also simplifies Operations and reduces cost - as the Kubernetes container infrastructure required to run these applications is automatically spun up and scaled precisely with the workload, to optimally handle all runtime requests. Recent advances in open source technology now allow organizations to run Serv...
GCP Marketplace is based on a multi-cloud and hybrid-first philosophy, focused on giving Google Cloud partners and enterprise customers flexibility without lock-in. It also helps customers innovate by easily adopting new technologies from ISV partners, such as commercial Kubernetes applications, and allows companies to oversee the full lifecycle of a solution, from discovery through management.
In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at, will discuss how to use Kubernetes to setup a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace....
SUSE is a German-based, multinational, open-source software company that develops and sells Linux products to business customers. Founded in 1992, it was the first company to market Linux for the enterprise. Founded in 1992, SUSE is the world's first provider of an Enterprise Linux distribution.
Dito announced the launch of its "Kubernetes Kickoff" application modernization program. This new packaged service offering is designed to provide a multi-phased implementation and optimization plan for leveraging Kubernetes on Google Kubernetes Engine (GKE). Kubernetes, a relatively new layer of the modern cloud stack, is a production-ready platform that allows companies to deploy and manage containerized applications, update with zero downtime, and securely scale their deployments.
In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). According to the survey, a quarter of the respondents have already deployed Docker containers and nearly as many (23 percent) are employing the AWS Lambda serverless computing framework. It's clear: serverless is here to stay. The adoption does come with some needed changes, within both application development and operations. Th...