Gp's Blog
Permalink
Docker is about automation

I wouldn’t be surprised if you haven’t heard of Docker. It’s still early days for this up and coming technology. Unfortunately, if you read other sites about what Docker is then you will likely get a skewed view. The reason being that most people think Docker is a lightweight container technology. While this is mostly true, that’s not the magic that Docker brings to the table.

Docker is about unitizing applications for deployment at scale. It is a packaging and shipping mechanism. The container itself is less important to the overall value that Docker brings to the table. And this is what is getting people excited. The fact that Docker has figured out how to wrap an application up and then deploy it into lightweight containers within seconds and then be able to tear it all down just as fast is the magic. It has always been about the automation around the container.

With the announcement that they will be container agnostic, Docker solidifies the fact that the container doesn’t hold as much weight as the automation. There is even more to be excited about with Docker and I can’t wait to see clouds build on Docker in the enterprise environment.

Permalink
What’s OpenStack Got? Momentum!

I work with a lot of large enterprise customers everyday. I can’t name them for reasons I hope you can understand, but rest assured they are BIG. Fortune 50 big. When I speak to those customers and they talk about private cloud, they talk about OpenStack. If they are not testing it in their lab already then they are planning to. And there is good reason for them to be looking at OpenStack at the expense of the other platforms.

A huge community of support

OpenStack’s community is growing so rapidly that commits are on a geometric growth curve (http://www.cloudscaling.com/blog/cloud-computing/havana-openstack-continues-its-march-toward-dominance/). And commits are just part of the story. At each OpenStack Summit the attendees push the limits of what the venue can hold.

The community is vast and spans customers building their own, and large software vendors trying to build solutions around OpenStack. Which brings me to my next point.

Motivation

VMware has been a dominant player in the virtualization space that has paved the way, on the infrastructure side, to the clouds that are being built today. Enterprise virtualization, on which all clouds are built, are either using VMware technology directly or are using some other virtualization technology that was inspired by VMware’s server virtualization platform.

But in the ever shrinking wallet of IT, infrastructure is not a popular cost center with CIO’s and CFO’s. To paraphrase an often over-used parable, Executives want applications (aka holes in the wall) and not infrastructure (aka drills). And what I have learned is that their is a lot of application development happening at large enterprises. It doesn’t matter if you are in logistics or financial services, great software combined with great processes is what makes their companies competitive.

And here is the rub for VMware, now that everyone has virtualized the big bang for the buck is no longer in virtualizing legacy infrastructure. The savings comes from optimization on a few fronts:

  1. Continuous Delivery Processes and Tools
  2. Autonomous and Optimized infrastructure
So VMware, Microsoft, and others move up the stack and provide more tools to accomplish this. But enterprises want the cheapest drill that will make a hole in the wall. They also do not want to be at the mercy of a single vendor. And so in the quest for the cheapest and most diverisfied solution, they look to open source projects such as OpenStack for their tooling.

Maturity

OpenStack has seen a lot of time in the field. It is getting exercised by quite a few companies and the kinks are being worked on. The Compute project (nova) has been around since the beginning and is quite stable. Most of the other projects either coincided with Compute or came shortly after it. There is a lot of strong parts of OpenStack that are ready to be flexed in the large datacenter. And customers have noticed.

Vendor Support

Canonical (makers of Ubuntu), Red Hat, Mirantis, IBM, Dell, and Cisco along with a smattering of startups and small companies are either offering OpenStack-based solutions, OpenStack distributions, or additional products on top of OpenStack. And sometimes all three of those. These companies are ready and willing to deploy OpenStack for their customers and support it fully. Not to mention the contributions that they give back to the project in the form of engineering time. The acceleration that these companies are seeing by buying into OpenStack is tremendous and they have effectively double or tripled (or more!) their engineering output with a fraction of effort. No individual company could have built a project to OpenStack’s scale in the time it has taken to build it. It’s a big win for all of them.

Final Thoughts

It’s clear that OpenStack has a huge advantage in the mindshare of enterprises looking to deploy private clouds in their datacenters. Whether they are being told by the big vendors mentioned above about new OpenStack solutions they are offerring, or just reading about all the news that comes out about OpenStack. There is a lot of hype and activity around OpenStack that makes it impossible to miss. There are still cautions that I have not covered around security, scalability, and other aspects that will likely be hammered out as production rollouts increase. But with the community and vendors behind this project it is unlikely that they let any issues stop them from a successful rollout. These companies have bet big on OpenStack as an industry. And the customers are sure to follow.

Permalink
My Review of Mindsight by Daniel J. Siegel. Read by the author.
Permalink
Great By Choice by Jim Collins and Morten T. Hansen. Read by Jim Collins.
Permalink
[Off-topic] On Agile

I don’t know why this article by Mike Gualtieri infuriates me (http://blogs.forrester.com/mike_gualtieri/11-10-12-agile_software_is_a_cop_out_heres_whats_next). Is it the fact that this guy is an analyst who has never run a software development shop? Is it because all of his information is second and third hand? Is it because he is right in some respects? Of course the rhetorical answer to all these questions is, YES.

Let me start by saying Agile can work, I know this because I have made it work. My experience of 1 is very biased. But the fact that I have a counter example means that Agile works… BUT, and there is a big one, I have never done scrum or XP. I have done agile development for over 5 years but I have never done a prescriptive version of any agile process. And so I have a problem, on one hand I cannot say that agile works because frankly we cherry-picked the best aspects and threw away the others. I can’t say that agile doesn’t work either. So let me describe the process we ran and then my insights on what I think actually works.

1. Everything is the context of the user
It is way too easy to take a technology and map it to the user needs. The flow needs to go other way. The user needs something and you build the product, the architecture, the user interface around making the users ability to solve that problem as effortless as possible. When every conversation you have is in the context of the user the flow moves so much better. Engineers begin thinking of the user more naturally instead of focusing solely on the technology which they are totally immersed into on a day to day basis.

Designing the user experience from the beginning is the point of what we are building. BUT at the end of the day the code accomplishes the goal. There are engineers out there who will shy away from the “hard” stuff because it is hard. My team has constantly heard me say to them “I don’t care if it is hard for you if it makes the users lives easier. We do the hard stuff so they don’t have to.”

You get here by doing a few things:
- Hire designers or developers, testers, documentation people with an eye for user experience design.
- Have detailed and rich conversations about the users problems before you start any project. Understand every nook and cranny of the user and their context and their problems. You will need this later.
- Design the final product based on all the current assumptions… although they will be wrong. Keep it flexible and light.
- Prototype to find the flaws with the design.
- Iterate the design (this is the one anti-waterfall/pro-agile item)

I have not had all these things all the time in my projects and the results have always shown. Every time we include these aspects we see the benefits. I am sure there is more, but a lot of teams don’t even have this much.

2. Conversations and right-sized artifacts
One aspect of agile that is key is collaboration over contracts. In software there are still companies who create contracts between management and the engineering team to deliver the requirements. And you get exactly what you asked for, what the requirements stated. In a 6 month project the amount of requirements is so fast and the interplaying complexity so intense that it takes almost 6 months to nail requirements that will still be wrong in the end. This relates to something called the Cone of Uncertainty. Basically the larger the project the more unknowns and the more uncertain you are of the timeline, quality, and resource requirements. This is true in any project whether you are building a highway across three towns or building a complex piece of software.

So your requirements are wrong, and the bigger the project the more wrong they will be. Your estimated level of effort required to achieve a specific level of quality is wrong and the longer the project is the more wrong it will be. I have lived these facts in every company I have worked in, whether it was my project or someone else’s.

Agile has the most elegant answer to this, although simplified beyond what Mr. Gualtieri is comfortable acknowledging. Working software over comprehensive documentation. Paired with the collaboration over contracts value we have a powerful concept that is best summed up as conversations and right-sized artifacts. This flows directly from #1 too.

When you are with the engineering team and you are discussing the customer in excruciating detail it is important to write that stuff down. Whether you have a formal documentation called an MRD, PRD, User Stories, Scenarios etc. or some other less formal approach, every topic discussed and planned as part of the conversation needs to be written down in some form to remind everyone about what was discussed. This is as much as is necessary because you will talk about it again in even more detail when it comes time to actually implement the item.

At the time that you actually implement the item there is always going to be new information that will change assumptions in the item you originally wrote down. There is a new context. When you got to implement that thing you must discuss the new context and write down the new assumptions so that you are always working against a set of requirements that has the best knowledge baked into it.

Last, in direct opposition to Mr. Gualtieri, working software is a measure of progress. However, that doesn’t mean you are done. Even though the software works it may not be what you wanted. It may not work right even though it matches the careful designs perfectly. It may be flawed in ways you didn’t anticipate. So throw it away and do it again. This happens, you didn’t mess up, it’s part of the creative process. The willingness to throw away working code is as important as getting working code.

3. Regular check-ins
This one is pretty much a follow-on from #2. If you throw requirements over the fence you are going to get exactly what you asked for… but not what you want. It is important for any product owner, product manager, customer, business analyst, etc. to regularly check in and get demos of the software. This is where sprint-boundaries help. It is a mutually agreed upon time where the non-developers get to but their noses in and help steer the project.

This also means that outside of those boundaries you need to leave the team alone. A well put together team can go for a few weeks without direct monitoring. Giving them time to think and put together their best ideas. The middle of that process can show some pretty ugly children of the mind, but the boundaries are when they show off something they are proud of. Like an artist, you want to see the painting when it is done, not when it is being painted. Anyone who has watched Bob Ross has seen him take his paint knife and ruin a painting only to make happy little trees or mountains. But man do you get nervous on that first swipe of the knife. Step back and let the artist do their work. If they are a good team they will give you something beautiful. Just don’t let them do the entire Sistine Chapel without checking in every few weeks.

4. Finish before you start
The worst software projects I have been on are the ones where too much stuff is going on in parallel. It is by far the buggiest and hardest to integrate when so many things that are interrelated are always in flux. A great engineering team will find the right ways to split the project into parallel and serial streams of work. Within each parallel stream there will probably be a mini-serial stream of work. Before a team goes off and starts something knew they should complete the current work as fully as they can based on the requirements. There are different ways you can approach this and the situation will call for the right thing, but there are some successful tricks my teams have used in the past:

- If you are building a broad API for a group of features. Do it in parallel with 1 of those features and finish them together before working on the next feature that uses that API.
- Database changes always come first
- Multi-sprint items need to be cut up to achievable milestones within the sprint boundaries
- Done means: design, code, unit tests, and functional tests are all closed
- Bugs stay flat or go down from sprint to sprint
- Automated tests increase and the pass rate is as close to 100% as possible

This list is not exhaustive and I have other tricks very specific to the projects that I have worked on. To use a hiking analogy, “you always move from water and not to water.” If you have ever been hiking in the Grand Canyon you will understand this. Every time you move from a stable (or at least known) state to another known state then the in between is not so hard to deal with. If the code base is entirely unstable from sprint to sprint you will find that the end of the project is a very hard road with lots of late nights and emergency re-dos of code. Keeping the bug count as small as possible and automated test case pass rate on an increasing trajectory on the boundaries keeps everything in check. When you move from water, you can always go back if you need to.

5. Minimizing the Test-Dev progress delta
This is the hardest thing in all of engineering. There is always a tendency for development to run ahead and put together tons and tons of code and then leave it on the testing team’s doorstep. This approach will and has always failed. To finish something you need it to be fully tested. Without that, the state is frankly unknown and unstable. Having lots and lots of code in that state is bad. So the dev progress should only be slightly ahead of the test process. In Agile they would say it should be totally in sync, but in practice that is never the case. Test and dev have a back and forth and a natural order that falls into line the same way in every team I have been in. TDD and BDD are great goals but I have seen great software come out of teams that have not followed that practice.

When dev and test are separated by entire sprints or entire milestones, that’s when things get squirrely. The closer you can keep the dev and test work together the higher the code quality on a day-to-day basis. You will move much faster with a tested, bug-free code base then you will with a code base that has a ton of features that haven’t been tested.

6. The team doing the process decides the process
The very last item is the kicker. The team is ultimately the judge of the best process to use. In other words the process needs to be tailored to the needs of the business rather than the other way around. When a team is forming for the first time it is good to use a text book version of a process to get going. But continuous improvement determines that it will not be in that state forever. To get better you need to have a process that works for all the teams involved. You need to have a process that works for the management team, the product managers, the engineers, the testers, the documentation team, the support team. The process doesn’t begin or end with engineering.

The process cannot be handed down from on-high. This gets into the same problems as #2. The assumptions will be wrong. You need to analyze the process and the users of the process as much as you would for the users of the product you are building. Mr Gualtieri came up with his PISS paradigm (which he could have made IPSS if he wanted to), but he is wrong, and scrum is wrong, and XP is wrong, and waterfall is wrong, and feature crews are wrong, and my 5 points above are wrong. They are all wrong for anyone but the people who created them. The process that works is the one that works, not the one in the text book.

Here is my advice. Find a process you think would work for your business. Apply it by the letter, then improve it rapidly at first, maybe even throwing it out entirely and starting again with something else. Once you find the one the works, improving it on a regular interval. Be empirical about it. Make improving the process as much a part of your business as building great products.

Permalink
An interesting sci-fi book by John Scalzi and read by Wil Wheaton.
Permalink
Review of The Game of Thrones, the first book in The Song of Fire and Ice series by George R.R. Martin. Read by Roy Dotrice.
Permalink
My Review of Spark by John J. Ratey and read by Walter Dixon.
Permalink
Review of Echoes of Honor by David Weber. Read by Allyson Johnson. The 8th book in the Honor Harrington series.
Permalink
[Off topic] 24 Hours Later: Switching to android

Phone: Samsung Nexus S
Carrier: Rogers

I just had to write a post about this. I have heard the theoretical arguments for so long that I wanted to give the practical. Instead of discussing what feature gaps there are between the two, which is exactly how everyone else approaches, I am going to discuss the usefulness of the differentiating android features as well as the things that annoy me since I am coming from iPhone.

Why “Open” is useful:
- It lets you customize the crap out of stuff.
- Download whatever you want, at your own risk. No parents watching over you.

Why “Curated” is useful:
- Way easier path to find things. iTunes, App Store.
- Better protection from rogue apps. I noticed that I wasn’t paying attention to the services that apps were using when I installed them on android. I am sure I am not the only one.

Differentiating features for Android and how they play in my day-to-day:
Flash - Doesn’t play at all. Haven’t even installed it yet.
Widgets - Meh, interesting and fun. Not useful. Why is there no gmail widget from Google?
“Open” - Since I a tech savvy I enjoyed the fact that I could download custom apps like swype off-market. But I can feel the danger of the system immediately. I would never recommend non-technical people to do this kind of thing.
Device Choice - This was nice. For $99 I got a relatively new Nexus S whereas 279 would have bought me the 1+ year old iPhone 4. This ultimately was my buying decision criteria.
Google integration - This is spotty and I might say dreadful. Android and Chrome don’t sync bookmarks. Tell me that doesn’t look like an internal battle between the organizations. Sure Calendar, Mail, and Contacts sync but that isn’t different than the iPhone.
“Real” multitasking - I don’t know how people stand behind this “feature.” It is just another thing that the user has to track and maintain. I would actually say that this is deficiency. People don’t want to manage things, believe me I work in the management software business. Application state should be automatically managed. Rogers recommended to download an app like Juicedefender which does the management for you, seems like Google missed a feature that people need.
Switchable keyboard - I use swype now. Swype itself could be better, but the potential is great.
Built in Navigation - Very useful. I want to buy a dock to take better advantage of this.
Extra hardware buttons (Back, Menu, Search) - I am still having a hard time switching my brain from the iPhone way of doing things. Menu button I think is just bad design.

Things that bug the crap out of me about android:
Mac integration - Doesn’t work out of the box with iTunes or iPhoto. Very painful… very very painful. iPhoto especially.
Hands free calling - Anyone who claims that what android “offers” in the way of voice dialing is any good is just smoking something wacky. Every phone I have used ever (feature phone, BB, or iPhone) has done this better.
Browser - Doesn’t respond to taps appropriately. Doesn’t provide appropriate feedback to the user that something is happening (or not happening). I am constantly tapping, re-tapping, hitting the back button, then tapping again.

Thanks for reading.