Category: Uncategorized

Progressive delivery is the practice of rolling out an application in a managed fashion to designated cohorts, making canarying, blue/green deployments, and A/B testing foundational to testing and deploying applications. Progressive delivery thus reduces risk and increases control by decoupling deployment and release, allowing more groups to be included in the release process, from product management to marketing, development to IT operations. Progressive Delivery allows release progression and radical delegation. 

The progressive delivery trend is on the verge of entering the mainstream. However, as we see with any technology-driven wave of innovation, we need platforms to drive the culture. Change of tools drives culture change, just as culture change drives a tool change. Many companies are now coalescing around concepts that enable progressive delivery led by feature management firms, like LaunchDarkly and Split.io. 

But just as important, other innovations from companies in the CI/CD space, whether that be CloudBees, CircleCI, GitLab, or GitHub. At the same time, Observability and integration with software delivery is becoming ever more important. If we look at Honeycomb, Datadog and Dynatrace, they’re all coalescing around this ability to look at application traffic patterns, before rolling them out more broadly.

These patterns generally begin in terms of control, minimizing risk, and ensuring compliance. They may also be in terms of a performance, they may be questions of application functionality or even and this is where things get interesting as a possibility, A/B testing and experimentation. The market is still young, but the platforms are becoming more and more mature, which is going to make it easier for organizations to adopt progressive delivery approaches. 

Now all of this stuff is hard as we’ve seen with DevOps, agile, and observability. The enterprise, or at least one department of it, may get some of the way there, but we’re certainly not seeing wall to wall adoption by enterprises. Modern engineering excellence is found in pockets. The question is how to encourage and enable broader adoption. There are a set of disciplines that are interconnected that need to be adopted, leading to Production Excellence as Honeycomb calls it – in order for us to move forward and take advantage of some of the new possibilities that FAANG companies and startups have demonstrated. 

CD is often the top of the spear for progressive delivery changes in an organization. Enterprises seeking to accelerate the rate at which they can develop new products and services, with improved quality, are looking to remake their testing and build workflows into modern automated pipelines. Flux, now a CNCF incubator project, is a good example – while it’s a CD platform it also brings an associated project Flagger into customer adoption, and thus we see organizations such as the US DoD adopting these new practices. The DoD is an early adopter. While the practice of progressive delivery may not be fully mainstream yet, the DoD is a bellwether for broader adoption.

For many organizations trying to adopt continuous delivery they realize that CD is not complete for their business. Progressive delivery provides the control and inclusion of business teams that many organizations need to be successful.

As an industry, we’re increasingly getting there, with more standardised tooling, with workflows becoming more automated, and the ability of operations and developer teams to work together more effectively. Platform teams are seeking to pull all of this together, in providing managed services to developer teams. 

So progressive delivery, the delegated rollout to designated user populations in order to reduce risk and increase business functionality, is increasingly becoming a reality.

The word ‘control’ often has an immediate connotation for folks. For some there is the idea of ownership, enablement, or success. While for others the word conjures images of restriction, repression, or failure. In the context of Progressive Delivery we talk about control in the context of release progression and radical delegation. In both cases we view control as a benefit to both the provider AND the consumer of software.

Release progression — 

progressively increasing the number of users that are able to see (and are impacted by) new features

Controlling the cadence of software release is a requirement for most businesses. The reasons may vary, but the options for control are typically similar. Let’s consider a couple examples:

Productivity SaaS platform – When a new feature is being developed there is an immediate need to begin testing this on the production environment. The reason for this is that for platforms operating at scale, the time and resources required to simulate the production environment are far more expensive and inadequate compared to using feature flags, canary’s, blue green deployments, or some combination of these techniques. As the feature gains stability and completeness it can be released to a larger audience. This control on the provider side limits the blast radius of any potential outage or bug. Microsoft refers to this practice as ring deployments. This control on the consumer side allows the selection of one’s risk profile. The provider can allow folks to opt-in to new features or even explicitly opt-out. 

Financial service platform – For certain verticals the cadence of change can have a significant impact. Often consumers in the finance space need maximum efficiency during seasonal peaks (e.g. tax season, quarter-end, payroll, etc.). These peaks vary by customer, but it is critical that the platform doesn’t release changes that would cause changes if workflows or worse break existing functionality. Control for the provider in this use case enables them to continue innovating and forward development. The consumer is provided the ability to control the timing of when updates are applied to their account and in most cases allowed to delay changes until a peak has passed. 

Radical delegation — 

progressively delegating the control of access to a feature to the owner that is closest to the outcome

In the context of delegation there are multiple vectors of control. 

• Control of what can be changed
• Control of what changes are possible
• Control of who can make a change
• Control of when a change can be made
• Control of all the above options in combination or negation

Here is an example to help show how this benefits both the provider and the consumer:

Trying out the premium version – In this scenario a consumer is looking to try out the premium offering form a provider. The consumer is a current satisfied customer and wants to try the advanced feature before committing to pay the higher rate. Using feature flags and a radical delegation approach, the provider can enable a Customer Support Representative (CSR)  to turn on the premium functionality for the customer for a trial period. At the end of the trial the CSR could then upgrade the account to the premium offering or turn off the trial and have the customer account revert to the standard offering. 

In this example the radical delegation of control was a benefit to multiple folks. The developer that built the control point benefited by setting the parameters of what would change on the account and what would remain the same. This aspect of control allowed for a trial experience that had predictable outcomes. The variations were defined as part of the control options. For the operations team, this control point was of benefit because the service level agreements for the premium offering were able to be incorporated into the control point as well. This meant that during the trial the customer was automatically provided a premium experience. If the customer decided to revert to the standard offering the SLAs were automatically adjusted. The CSR was able to benefit from the direct control of the customer experience. And the customer benefited from the ability to try a premium experience before having to commit.

Let it go

When control is given to the individual closest to the outcome it is often a benefit. A key consideration is that the control point needs to be well defined. There needs to be clear articulation of what changes are possible and the outcomes that will result. A well designed platform will allow for changes to be reverted. Changes that are delegated and not recoverable but the person that has the authority to make the change is an antipattern. In the case where a change is not recoverable, this is an indication that the change is not ready for delegation.

When a system is architected with delegation in mind, all sorts of changes are possible. One of my favorite examples is using feature flags for live database migration. This is an extreme example of what is possible when you recognize the value of delegating control.