In the past, we ran small data on expensive big boxes, and it worked fine. For example, most database applications were sized in gigabytes or a few terabytes. Today, we're moving to a world of Big Data measured in petabytes, running on lots of inexpensive small boxes. Data volumes are growing at 50 percent per year worldwide and at more than 100 percent compounded annually in many companies. This growth presents a fundamentally different computer science problem, and it requires a redesign of the way we build and operate data centers.

Simply put, rapid data growth is outstripping legacy IT designs, and the new world is all about distributed systems: commodity hardware, scale-out design, open standards, Ethernet, programmability, automation and self-service. The problem: your favorite large IT vendors don't have products that look like this, and their pace of innovation is too slow to catch up to the need.

In the mid-1990s, the explosive growth of the Web drove a wave of innovation in distributed computing designs built on commodity hardware. Necessity was the mother of these inventions, because early Web companies faced geometric growth in demand without the lucrative online advertising budgets that developed post-Y2K. Companies like Inktomi, Akamai, Google and Amazon rejected big-iron computing models from traditional vendors and instead built massive infrastructures on commodity hardware with a new breed of scale-out software.

As Moore's Law, data growth and competitive pressures advance, the same disruptive IT model is showing up in the enterprise. The ability to store and process petabytes of data in a scalable and cost-effective way has now become fundamental to competitiveness in a variety of industries. Pharmaceuticals, financial services, video, manufacturing, cloud services and military/intelligence are all examples of increasingly data-centric sectors. Organizations that learn to harness 10 times more data than their competitors will increasingly gain a dominant competitive edge.

The webmail market provides a good example of this dynamic. Yahoo! held a strong leadership position with its Yahoo! Mail service and operated its service on large enterprise-style storage arrays from NetApp. Based on the economics of its storage infrastructure, Yahoo! offered users 25 megabytes of free email storage. Google came along with a very different software-defined design, running on huge clusters of commodity hardware, and trumped Yahoo! with an offer of 1 gigabyte of free data per user. Yahoo! eventually responded with unlimited space, but it was too late; millions of users had switched over to Gmail. As storage analyst Robin Harris said back in 2007, Yahoo! was "bringing a knife to a gun fight."

So how do the Web-scale players like Google and Amazon think about computing? It's at a whole different macro scale. The outside of the building is the computer chassis, and everything inside is just chips. The big guys employ thousands of PhDs to program the entire building as a single system, enabling incredible scalability, elasticity and flexibility. Because everything is automated, it becomes possible to reconfigure and program these resources to deliver end-user services with one-click simplicity.

Most enterprises and smaller cloud services, however, don't have thousands of PhD programmers to write their own operating systems and data center automation software. They will be looking for a new generation of vendors to deliver commercialized versions of this cloud architecture, enhanced with enterprise-class performance and features.

This new model of "software-defined data centers" running on clusters of commodity hardware will be the new battleground for data-centric industries. It's equivalent to the Industrial Revolution, when automated factories displaced manual assembly, but this time it's about data. The upsides include significantly lower OPEX and CAPEX versus traditional IT solutions. Even more important, software-defined systems can redefine business agility and velocity, shrinking wait times for computing resources from weeks to minutes.

IT shops that fail to make the jump will increasingly face cloud outsourcing threats and competitive pressure. The IT pros who are successful may find a huge upside for their companies and their careers.