SYS-CON MEDIA Authors: Elizabeth White, Maria C. Horton, Andy Thurai, Liz McMillan, Zakia Bouachraoui

Blog Feed Post

Deconstructing the '2014 State of Devops' Report

There is a very famous maxim in statistics - correlation does not imply causation.

If you Google 'interesting correlations', you'll see that correlations have been established between ice cream consumption and murder, pirate shortage and global warming, organic food and autism, internet explorer and murder (this one is not so surprising ), obesity and the debt bubble and many other such extremely disconnected phenomenon.

Having said that, the reverse is also true - correlation does not imply no causation.

Moreover, there is this entire universe of literature available on 'cognitive biases' - we see what we want to see and we  hear what we want to hear - but that's a topic for another day.

Why all this?
Well, went through the "2014 State of Devops' Report - recently co-published by PuppetLabs, ThoughtWorks and the IT  Revolution Press. With distinguished names like Gene Kim and Jez Humble associated with this report as contributors I had  high expectations.

Instead of reading the report cover to cover and absorbing the information which the report threw at me, I decided on a 'pull' approach. I listed down the questions I would typically ask if I were to do a current state assessment of Devops in the industry and then observe to what extent the report addressed those.

I know it’s a bit unfair to analyze any piece of work with a set of parameters which it was never meant to comply to.  However, being in this journey myself along with many of my colleagues, I am assuming that I am also a target recipient  of this report and in that sense a 'customer'. And like any customer, I narrate my experience of this product.


Below are the questions and the answers I got after reading the report.

Q1. What is the degree of awareness about Devops among the IT community today?

I know this is a difficult question to answer considering that there are about 25 million IT professionals in the world (a conservative guess as the number was pegged at 15 million in 2006).

The report mentions that around 9200 people participated in this survey from some 110 countries. From a sampling  perspective this is less than 0.04%. I am assuming that this was not a pre-screened survey population who (a). were aware of Devops and (b). involved in some Devops related activities in their organizations. The survey was made publicly available  and people were free to participate. However, a person who is not aware of Devops is not likely to participate in a State of Devops survey so probably all the respondents were aware.

So, my first question is not answered. The intention to was to know the degree of awareness about Devops across the global  IT community. A difficult question to answer no doubt but an important one nevertheless for a 'state of Devops' survey.

Certain demographic details of the respondents are available though - country, industry, role, size of the organization (employees, servers) and department (IT development or operations).

Two interesting observations in the demographics. There is a new role called Devops Engineer (30% claiming this role) and a new department called Devops Department (16% claiming to be working in one).

Q2. How many IT organizations have started implementing Devops?

In absolute terms, obviously, the report does not have the data.

However, the report does mention another statistic which can be extrapolated to get this data. It says that 92% of  respondents working in DevOps departments said they were currently implementing DevOps practices, or already had. This is  sharply higher than the overall group, of which 57 percent say they are implementing, or have already implemented, DevOps  practices.

My interpretation of this is that 57% of the 9200 respondents said that they are or have implemented Devops. That is 5250  people. How many organizations do these people represent?

It’s not clear as to how many organizations were represented by the 9200 participants. The report does mention that around 1000 people volunteered to disclose the names of their organizations which were publicly traded and that number is 355. 
Does this mean there were other people who volunteered the names of their organizations which were not publicly traded? 

This part is not clear.

Assuming that ONLY 1000 people volunteered the names of their organizations (and all of them were publicly traded!) and  they represented 355 companies (35%), then 5250 people would roughly represent 1800 companies. Provided I am getting all  my assumptions right, that's an encouraging number.

Q3. What have they started implementing?

The report mentions 4 Devops practices
Continuous delivery
Use of version control for all production artifacts
Automated testing
Monitoring application and system health

The report refers to these as 'Top' practices so it's not clear whether existence of other practices were also checked.

This is, in fact, the most disturbing part. It almost amounts to trivializing Devops if we're equating it to these 4  practices, all of which are related to tooling (although not surprising considering this is coming from PuppetLabs - a  primary vendor for IT infra automation solutions).  In my view, without the more foundational practices related to demand and workload synchronizing among Dev and Ops, 
alignment of their KPIs and goals, fixing product responsibilities beyond projects and interventions at all levels to  remove the trust deficit between the two silos, no amount of tooling and automation will achieve the required synergies and performance breakthroughs.

Q4. What are the initial results?

This is where the report becomes fuzzy. My expectation from this question was to know whether IT performance improved  after following Devops practices. The report does not answer that at all, directly.

What it says is that good organizational performance is dependent on good IT performance. I already know that and I also  know that it is not a straight forward relationship but that's a different topic altogether.

I don't want to know whether IT performance improves organizational performance and the fact that the latter also depends  on good organizational culture and job satisfaction - when I am reading a 'state of Devops' survey report.

The report does say that Devops 'top' practices have a strong correlation with IT performance metrics like deployment  frequency, lead time to change and MTTR. This is very good and almost ground breaking. We can always debate whether these  are the only metrics which define IT performance but nobody can refute that these are very important metrics. And if we've been able to establish that Devops is positively influencing these metrics, it is great news for all us.

I hope in the next survey we get to know the quantum of improvement as well.

In a somewhat convoluted way, the report does say how Devops can contribute to better organizational culture and job satisfaction. While they are useful, in my view they have been used more as filler material for the report to compensate for lack of data.

There are these predictable lofty claims like ' high performing IT organizations deploy code 30 times more frequently with 50 % fewer failures' or 'high-performing IT teams had 50 percent higher market capitalization growth over three years than those with low-performing IT organizations'. However, it still begs the question 'did Devops practices helped them achieve these goals and if yes, which ones?'.

Q5. What challenges are they facing?

The report is dead silent on this which is very surprising.

Weren't questions like 'what are your biggest roadblocks for Devops' or 'have you had any failed Devops initiatives', part of the survey?

While there are some very useful 'recommendations for improvement' section at the end, more real life voice of  practitioners is the expectation from this kind of a survey.

Q6. What innovative solutions / approaches are they coming up with, if any?

Again an area not addressed at all.

The only one which can remotely qualify for this is establishing a Devops ' department' which seems to be more successful  in initiating Devops practices atleast, if not getting the benefits out of them.

07. What is the summary of the survey? 

The summary of the report paints a very rosy picture about the future of Devops but I can't say whether this was derived  from the summary of the survey.

This is the biggest disconnect.

The survey data seems to have focused on demographic details, IT and organizational performance metrics and existence of  some 'top' devops practices. Rest was all statistical jugglery - mainly selective correlation analysis.

There are 2 sections devoted to organizational culture and job satisfaction which would have been otherwise good reading  if the authors had not tried to force fit Devops into them.
The relationship being implied was circular in nature - good organization culture = x, y, z, devops also needs x, y, z and hence devops = good organization culture.

The report states "in our survey, we were able to measure culture  quantitatively, and found it is one of the top predictors of organizational performance. Culture really does matter to the business"

The "measurement" goes like this - Ron Westrum,a sociologist who found that organizational culture was predictive of safety in the health care industry. He had defined 3 types of organizational cultures - pathological, bureaucratic and  generative. All of these have some characteristic traits. The generative one has traits like high cooperation, risks are  shared, failure lead to inquiry, etc., typically the good desirable traits. Then the report simply says 'you can easily  map these attributes to Devops practices". I fail to see the measurement in this leave alone the quantitative part.

08. Did the report meet its intended objective?

The report starts with this "our goal is to provide a picture of how DevOps works today, based on real data from real  people. We hope this picture will help IT managers and practitioners understand how to build greater value in their teams  — and help their companies win in the marketplace"

Well in terms of real data from real people are concerned, yes that part is correct. But what exactly did those 9200  people say? Except talking about their demographic details and sharing some metrics about their IT departments / companies and whether or not they are falling those 4 Devops practices, did they share any real life experiences? Experiences  related to success and failure, what worked and what did not, business triggers, challenges, innovative solutions? From  that perspective the survey / report was a huge disappointment. If I had wanted to read Gene Kim and Scott Ambler's views 
on Devops I could have easily read their blogs and books.

Here my expectation was to hear voices from the trenches - from  people who are in the process of making it happen. That in my view would have helped IT managers and practitioners 
understand how to build greater value in their teams - far more than the statistical correlations, theory on types of  organizational cultures and well intentioned albeit predictable advice from the experts.

To summarize, looking at the 8 answers I got to my questions I think the report failed to fulfill my expectations. I give  3.5 out of 8.

But of course, as I stated at the beginning of post, it is unfair to apply a set of parameters to a product which it was  never mandated to comply with. So, if the authors of the report happen to read this, my request to them is to treat this  as an honest attempt to dissect their hard work with the same professional rigor which they had applied when they created  it.

Read the original blog entry...

More Stories By Sujoy Sen

Sujoy is a TOGAF Certified Enterprise Architect, a Certified Six Sigma Black Belt and Manager of Organizational Excellence from American Society for Quality, a PMP, a CISA, an Agile Coach, a Devops Evangelist and lately, a Digital enthusiast. With over 20 years of professional experience now, he has led multiple consulting engagements with Fortune 500 customers across the globe. He has a Masters Degree in Quality Management and a Bachelors in Electrical Engineering. He is based out of New Jersey.

Latest Stories
While the focus and objectives of IoT initiatives are many and diverse, they all share a few common attributes, and one of those is the network. Commonly, that network includes the Internet, over which there isn't any real control for performance and availability. Or is there? The current state of the art for Big Data analytics, as applied to network telemetry, offers new opportunities for improving and assuring operational integrity. In his session at @ThingsExpo, Jim Frey, Vice President of S...
"We were founded in 2003 and the way we were founded was about good backup and good disaster recovery for our clients, and for the last 20 years we've been pretty consistent with that," noted Marc Malafronte, Territory Manager at StorageCraft, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In their Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, and Mark Lav...
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settl...
"At the keynote this morning we spoke about the value proposition of Nutanix, of having a DevOps culture and a mindset, and the business outcomes of achieving agility and scale, which everybody here is trying to accomplish," noted Mark Lavi, DevOps Solution Architect at Nutanix, in this SYS-CON.tv interview at @DevOpsSummit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
According to the IDC InfoBrief, Sponsored by Nutanix, “Surviving and Thriving in a Multi-cloud World,” multicloud deployments are now the norm for enterprise organizations – less than 30% of customers report using single cloud environments. Most customers leverage different cloud platforms across multiple service providers. The interoperability of data and applications between these varied cloud environments is growing in importance and yet access to hybrid cloud capabilities where a single appl...
@CloudEXPO and @ExpoDX, two of the most influential technology events in the world, have hosted hundreds of sponsors and exhibitors since our launch 10 years ago. @CloudEXPO and @ExpoDX New York and Silicon Valley provide a full year of face-to-face marketing opportunities for your company. Each sponsorship and exhibit package comes with pre and post-show marketing programs. By sponsoring and exhibiting in New York and Silicon Valley, you reach a full complement of decision makers and buyers in ...
In today's always-on world, customer expectations have changed. Competitive differentiation is delivered through rapid software innovations, the ability to respond to issues quickly and by releasing high-quality code with minimal interruptions. DevOps isn't some far off goal; it's methodologies and practices are a response to this demand. The demand to go faster. The demand for more uptime. The demand to innovate. In this keynote, we will cover the Nutanix Developer Stack. Built from the foundat...
"Cloud computing is certainly changing how people consume storage, how they use it, and what they use it for. It's also making people rethink how they architect their environment," stated Brad Winett, Senior Technologist for DDN Storage, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Sold by Nutanix, Nutanix Mine with Veeam can be deployed in minutes and simplifies the full lifecycle of data backup operations, including on-going management, scaling and troubleshooting. The offering combines highly-efficient storage working in concert with Veeam Backup and Replication, helping customers achieve comprehensive data protection for all their workloads — virtual, physical and private cloud —to meet increasing business demands for uptime and productivity.
Two weeks ago (November 3-5), I attended the Cloud Expo Silicon Valley as a speaker, where I presented on the security and privacy due diligence requirements for cloud solutions. Cloud security is a topical issue for every CIO, CISO, and technology buyer. Decision-makers are always looking for insights on how to mitigate the security risks of implementing and using cloud solutions. Based on the presentation topics covered at the conference, as well as the general discussions heard between sessio...
"NetApp's vision is how we help organizations manage data - delivering the right data in the right place, in the right time, to the people who need it, and doing it agnostic to what the platform is," explained Josh Atwell, Developer Advocate for NetApp, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
The Software Defined Data Center (SDDC), which enables organizations to seamlessly run in a hybrid cloud model (public + private cloud), is here to stay. IDC estimates that the software-defined networking market will be valued at $3.7 billion by 2016. Security is a key component and benefit of the SDDC, and offers an opportunity to build security 'from the ground up' and weave it into the environment from day one. In his session at 16th Cloud Expo, Reuven Harrison, CTO and Co-Founder of Tufin, ...
Historically, some banking activities such as trading have been relying heavily on analytics and cutting edge algorithmic tools. The coming of age of powerful data analytics solutions combined with the development of intelligent algorithms have created new opportunities for financial institutions. In his session at 20th Cloud Expo, Sebastien Meunier, Head of Digital for North America at Chappuis Halder & Co., discussed how these tools can be leveraged to develop a lasting competitive advantage ...
A look across the tech landscape at the disruptive technologies that are increasing in prominence and speculate as to which will be most impactful for communications – namely, AI and Cloud Computing. In his session at 20th Cloud Expo, Curtis Peterson, VP of Operations at RingCentral, highlighted the current challenges of these transformative technologies and shared strategies for preparing your organization for these changes. This “view from the top” outlined the latest trends and developments i...