Continuous Delivery – Vision vs. Reality

Source –


Those were the words I saw on the presentation screen during my first introduction to the Army’s concept for the Stryker brigade. Some two-star general program manager was explaining to us why the Stryker brigade was the Army’s newest whiz-bang gizmo.

It sounded like malarkey to me. I was more interested in actually seeing a Stryker vehicle for the first time – we hadn’t received any of them yet. Then, when we finally received our first shipment of Strykers to the unit, I was disappointed – it was basically a shorter, meaner-looking Winnebago, painted some hideous shade of green, with a paltry-looking .50 caliber machine gun mounted on top. WHAT?!? Where are the phaser cannons?!? What exactly do you want us to “FINISH DECISIVELY” riding around in that sardine can?!?

Fast forward two and a half years, and I finally got it. I wasn’t able to understand all that “SEE FIRST” malarkey until I saw Stryker units operating in context amongst other, “regular” Army units. Stryker brigades are FAST – we were able to identify opportunities, process intel, plan on the fly, and reach the objective before more regular units even knew something was going on.

What was the secret to all this speed and agility? It certainly wasn’t the sardine can, and I only wish I could attribute it to phaser cannons…

It was the IT infrastructure – the system of systems that allowed each “department” of the brigade to communicate with each other, to share relevant information with each other at lightning speed. You see, the Army, back then at the turn of the century, understood what today’s enterprises are starting to grasp – IT is a strategic asset, and if employed correctly, IT is a key enabler of corporate strategy. The ability to identify business opportunities and make sense of what’s going on faster than the competition allows enterprises to FINISH DECISIVELY in the market.

I know, I know. What does the Army know about trying to get a release of changes into production during the assigned maintenance window? It sounds bizarre, but imagine yourself in the middle of a maintenance window where the satellite system goes down during an Iraqi sandstorm right before the unit is to depart on mission. Pre-combat checks have occurred and units are staged, ready to depart. All we’re waiting on is for all vehicles to digitally receive the final version of the plans, which is now held up because the satellite dish went down in this stupid sandstorm. No pressure, IT guys and gals – you are now the lynchpin to the entire operation.

As enterprises grasp the strategic importance of IT they’ve begun to explore and launch initiatives to accelerate the delivery of services. One of these practices is Continuous Delivery, synonymous with the underlying principles of DevOps. Continuous Delivery is all about ensuring production-ready code at all times, and shortening the feedback cycles from the market back to the business. In most cases, ensuring production-ready code at all times requires automation of the delivery pipeline, reducing the risk of human error in deployment processes and cutting down the time it takes to complete manual processes.

The enterprise has a vision of Continuous Delivery that will enable it to ACT FIRST in the marketplace, but the reality of the situation is that there are still manual processes in place which prevent the enterprise from achieving that vision. Companies have invested in release automation to help them automate their delivery pipelines, orchestrating a deployment as a system of systems. This is a terrific step in the right direction toward Continuous Delivery, but there are still some application components which are relegated to manual processes that hamper the overall pipeline – database changes being one of those quirky components.

Datical DB was architected to enable initiatives like Continuous Delivery for the database component of application releases, and to “snap” into your existing automation frameworks so you can leverage your investment in automation. If you’re investigating Continuous Delivery or have already invested in release automation, I invite you to join us for an upcoming webcast hosted by our partners at Serena Deployment Automation on the topic of Automating Database Deployments in Your Continuous Delivery Pipeline.


IT is the key.

Tagged : / / /

QA Roles and Levels in Continuous Delivery Model | Software Testing


0 – Manual compilation, no unit tests, manual testing by QA
Before we all recoil in horror it’s worth recalling the pure horror of how things used to be with an enormous gulf between developers and QA. Thankfully this approach is lost in the mist of time for almost everyone but if has not, you have my deepest sympathies.

1 – Automated compilation, some unit tests by dev, manual testing by QA
We start to see the process become more agile. Build scripts have made their appearance. Life is a bit easier for developers with the introduction of Continuous Integration but not much different to QA who are left out in the cold.

2 – Automated compilation, high standard unit tests by dev, automation of manual testing by QA
Build frameworks that allow failing a build process that successfully compiled but did not pass the unit tests are now used. QA are using tools which theoretically allow them to sign off individual user stories while still being able to quickly regression test the entire application to ensure that no new errors have been introduced. However, the separation of dev and QA ensures that almost every change by developers results in failing QA tests. The only way to get all the tests to pass is to either stop development or to fork the codebase so that the QA’s can work on stable code. Bugs are fed back into the development code branch and promoted to the QA branch. Congratulations for reinventing waterfall and for ensuring that the ratio between developers and QA remains 1:1.

3 – Automated compilation, high standard unit tests by dev, integration tests by dev, automated testing by QA using better tools
Some of the QA workload is taken over by developers who automate obvious integration points with the rest of the system. Some more load is taken off the QA who can re-use tests with tools like Fitnesse but the phrase “dev complete” is still heard. Developers may provide QA with test utilities and abstractions like Stubs and Fakes to make testing easier and more deterministic. It’s almost inevitable that every team’s story board will have 3 or 4 times as many stories waiting for QA than are either “In Development” or “Undergoing QA”. You can do Continuous Delivery at this stage but everyone wonders why it’s so painful and costs so much in time and effort.

What do we really need?

For us to ship reliable quality code, we have to change not just the tooling but how the delivery organisation is structured. When manual testing was the norm, a separate QA organisation existed because it was more efficient and more effective to have non-developers test the code. With the levels of automation now available a separate QA organisation is an anachronism which should no longer exist. So what do we really need?

4 – Automated compilation, high standard unit tests by dev, integration tests by dev, agile testing by dev, sign-off by QA
The role of the QA has mutated to that of QABA (aka ‘a bloody good BA’) – domain experts that represent the business in the delivery team but who are also responsible for creating the acceptance criteria for user stories where the acceptance criteria are expressed as scenarios that can be easily converted into actual test code. Developers write the application code and the code that tests it, including creating any tooling. The QABA can then sign off the story on completion without having to go back to the business. Business still see new features and capabilities at weekly demos and show’n’tells but are rarely involved with the delivery team on a regular basis. If you are B2C, even better is where new code is released into production without being signed off but hidden behind feature throttles that stop the new functionality from being seen outside of the company network. That way the entire company gets to do UAT on new functionality before it hits the customers.
A fundamental pillar of continuous delivery is that all* your tests must be automated. To achieve this the QA organisation should be in the business of writing the test scenarios that the code needs to be evaluated against and for signing off that the code does this. Test code should be a first class citizen of the application and should be written by people who’s primary job is writing code – the developers. I will say it again – QA should not be in the business of writing test code.
Developers are responsible for quality and should act like it. Sometimes that means taking responsibility from the QA organisation that should never should have been given to them. Quality is too important to leave to QA. Developers need to take full responsibility for the quality of their code and they should be in the firing line if something is broken. The role of the QA is to keep the developers on the straight and narrow and the most effective way of doing this is to get them to apply their confrontational mindset to the code via the acceptance criteria used to sign off the new functionality. Those of you familiar with BDD are probably nodding your heads right now but I’m not sure that BDD is the answer. It’s the right approach from the perspective of test case creation but the tooling is currently a zero sum game – the effort you save by having natural language test descriptions run is equivalent to the amount of effort you have to expend in extending your tooling to support your test cases in all bar the most trivial cases.
If you take a look at a company that prioritizes the ability to ship code, e.g. Facebook, you will see the developer taking far more responsibility for QA than is seen in the rest of the industry. It’s time for the rest of us to catch up.

Tagged : / / / / / / / / / / /