Full-stack Philosophies

Jeffrey Needham's Blog

RSS Feed

The 7 Deadly Sins of Cloud Computing

Posted on 11:12 pm August 28, 2014 by Jeffrey Needham

Seven deadly sins

As if anyone needs to be reminded, there’s a ridiculous amount of hype surrounding clouds and big data. There’s always oodles of hype around any new technology that is not well understood—I believe the correct term for this is product marketing. There are at least seven deadly sins that can be committed when determining a cloud strategy. Your mortality may vary.

Anger: Clouds Will Fix Your Intractable IT Department

Many enterprises and agencies have become frustrated by the pace and inflexibility of their IT departments to embrace innovation. As the industry has evolved, more process—encouraged by laws like HIPAA and Sarbanes Oxley—has been wrapped around (mostly) transactional forms of computing. It has been so difficult for IT departments just to get things to work and be compliant, that once achieved, most were loathe to entertain any hint of further change. The darker side of hiding behind compliance and policy can often be tribal conflict, headcount Darwinism or just plain politics.

Clouds cannot fix an IT organization’s behavior. Some of the motivation to move to clouds can often be traced back to this frustration, but some of the motivation is an authentic (possibly naïve) desire to move to demonstrably effective technology.

Many IT departments see big data initiatives as a huge disruptive threat, not an opportunity. When the frustration with IT causes business units to hire their own scientists to go around IT altogether and get their own cloud, completely by-passing IT for new projects, this bleeds power away from entrenched constituencies, which can work in the CIO’s favor. I have witnessed IT-sponsored witch-hunt meetings disguised as use-case outreach. Not all shops are this crazy, and not all politics can be bypassed, but some enterprises and agencies won’t have the motivation or political capital to re-engineer their IT strategy and that will limit the benefits of using new technologies such as Hadoop to improve a business.

A successful business analytics strategy (or any strategy) can never ignore political reality. Turns out, organizations are full of humans with varying degrees of motivation. Who knew?

Greed: Clouds Are Cost-Effective

Investing in a cloud strategy based solely on the need to save operating costs might be an easier argument than one based on anthropology, but the ROI math can be quite tricky. Any strategy that claims to save zillions should be run by somebody who understands enterprise-licensing contracts.

Realizing a net reduction in OPEX is dependent on how easy it is to stop paying Tier 1 vendors for support, which is their margins’ lifeblood.   Many enterprise-wide contracts have been negotiated in such a way that a net reduction in license or support fees is either too complex to tackle or would result in a larger support bill for a smaller footprint. Vendors deep-discount the initial licensing and use their leverage when it comes time for an enterprise to actually reduce their support burden and then they start charging full retail prices. Short of completely deleting all their RDBMS instances, it will take organizations years to reduce this burden. Read the fine print.

Sloth: Private Clouds Are Easy to Set Up

If you decide to have a private cloud, then you must build it, not install it. The catch to DIY is that some platform skill is required to stand up the first couple of racks. And building a few racks of big data cloud computing is cheap and low risk, but you can’t do it with 1990s transactional doctrine.

As data scientists learn how to use big data cloud technologies like Spark, Hive or HBase, their platform scientists will learn how to do the next-gen infrastructure. It isn’t easy, but both types of scientists should work together in one group. DIY cloud deployment requires a platform science development group, not an IT group that consists mostly of admins. Other names for this new group might be engineering operations or DevOps (more development than administration).

If you choose to purchase off-the-shelf cloud technology, it will still need to be customized to your needs—not impossible, but necessary. And you will still need EngOps to write production grade software. Although some production environments are more disparate and complex than OS source trees, many IT shops still don’t use version control tools like github or subversion. Treating production like a product is also helped along by a software engineering bent to operations.

Envy: Clouds Will Create Many New Jobs

Maybe. Just not entry-level administrator jobs. Any technology designed to run on cheap, unreliable hardware was also designed to be very low maintenance. There are several 4000-node Hadoop clusters in production at Yahoo! and they have a couple dozen DevOps engineers supporting tens of thousands of nodes. At that scale, administration has to be automatic and self-healing.

Unlike massive EMC SANs, Cisco backbones and Oracle databases, administrators are no longer needed to babysit cloud-computing infrastructure. There is a growing need for highly specialized Data and Platform Scientists, but software that can run in any cloud must, by definition, be resilient.

Pride: Everything Shall Run in a Cloud

The most common workloads for clouds are the webmail and e-commerce “front-end” type and the “back-end” analytics database type. Web-scale properties solved both ends of the affordable-at-scale problem for these two very different classes of workloads.

There are two very simple rules to scalability: never write, never share. Technologies like Gmail and Hadoop were designed to (mostly) never write and (rarely) share. They also run on very cheap, under-engineered gear because that’s all you (or even Sergey) can afford when you have to buy millions of servers.

Lots of legacy application stacks won’t work in a cheap, highly scalable environment, either because the legacy workloads are too much trouble to migrate, or there are …pause for dramatic effect… transactions involved. Transactional platforms often have to pretend to be in 30 places at once so that coherency is upheld. They also have to stick to their guns after the power is restored and your airline ticket still exists. This is why reservation systems remain some of the nastiest ones to scale.

Gluttony: Clouds Must be Built With Tier 1 Hardware

Since we’ve already witnessed IT sticking with what they know, and have some anthropological satisfaction as to why, whenever they attempt to build their own cloud, it ends up on hardware designed for software written in the 1990s (if you’re lucky). That multi-million dollar pile of gear underwrote the software that demanded over-engineered, extremely reliable hardware.

Whenever Oracle synchronously wrote a redo record to an EMC SAN, then those bits had better stay there, or your quarter close became a Spanish inquisition. That Tier 1 hardware is for software from an era that financially and technically pre-dated web-scale computing. Web-scale software is designed to run on cheap, simply engineered and unreliable hardware.

Lust: Clouds Need Shiny New Hardware

You may be relieved to hear that lust and gluttony are interchangeable in this atlas of sin. If one is selling Tier 1 hardware for customers to build clouds, gluttony and lust quickly reverse. But other vendors have started to sell “cheap” new commodity hardware specifically for clouds. This is where lust can be evenly distributed between groups that want shiny new objects and the groups that will accommodate them.

Doing a POC for Spark or Hadoop or setting up a migration lab is best done on gear that is 36 months old, in other words, free stuff. Cloud software is quite happy with craigslist stock and pricing.

If you just “have to have” new hardware, then this is the time to innovate and exploit some of your experience. With new cluster acquisitions, it’s possible to challenge the dogma of 19” racks and how power and cooling should be set up. This is your opportunity to move things forward. Check out Open Compute Project at opencompute.org.

Leave a Reply