Develop on Cadence. Release on Demand.
—A SAFe mantra
Release on Demand
Release on demand is the fourth and last element in the four-part Continuous Delivery Pipeline, as can be seen in Figure 1.
The three processes that precede Release on Demand in Figure 1 help ensure that the value is ready to be deployed in the production environment. But since tangible development value only occurs when end users are operating the Solution in their environment, delivery value at the right time is critical for the enterprise to gain the real benefits of agility. The decision to release is a key economic driver that requires careful consideration. While there may be situations where deployed features are immediately available to all end users, more often the release is a decoupled on-demand activity, occurring for specific users, timed for when they need it.
Note: This article is the companion article to Develop on Cadence.
The ability to release on demand is a critical aspect of the continuous delivery pipeline, which raises three questions that allow ARTs to embrace Principle #3, Assume variability and preserve options:
- When should we release?
- What elements of the system should be released?
- Which end-users should receive the release?
Having all three options accelerates, and provides flexibility in delivering value. Further, release on demand ‘closes the loop’ on evaluating the benefit hypothesis proposed during the Continuous Exploration stage of the pipeline. Releasing provides the data needed to decide whether further exploration of an idea is warranted, or if new functionality should be added, or perhaps a different approach is warranted.
Release on Demand requires the organization to develop the following capabilities:
- Ability to decouple the release from the development cadence
- Decoupling the release elements from the solution
- Architecting the solution to support incremental releases
The following sections describe each of these.
Decouple the Release From the Development Cadence
Develop on cadence is the other half of a strategy that allows ARTs and the Solution Trains to operate in a predictable pattern and synchronize the efforts of multiple development teams. But when it comes to releasing value, a different set of rules may apply.
Once we have a reliable stream of value through the Continuous Deployment process, the next consideration is when and how to release. The release strategy is decoupled from the development cadence, as shown in Figure 2.
Several release strategies may apply depending on the context and situation:
- Releasing on the Program Increment (PI) cadence – The simplest case is when an enterprise can release on the PI boundaries. It’s simplest because PI planning, releasing, and Inspect and Adapt will have predictable calendar dates. Also, the Innovation and Planning Iterations can be timed, designed, and organized to support the more extensive release activities. The IP iterations may include final, Verification and Validation (V&V), User Accepted Testing (UAT), and training and release documentation.
- Releasing less frequently – In many cases, however, releasing on a fast PI cadence may not be possible or even desirable. For example, in some enterprise settings, deployed systems constitute critical infrastructure for an operating environment. Even if the customer would like to have the new software, the service level, and license agreements may be prohibitive. And then there is the overhead and disruption of deployment. In other cases, the timeline for building complex systems with both software and hardware (e.g., mobile phones or geophysical mapping satellites) are driven by their hardware components (displays, chipsets, and the like), which typically have longer lead times. The new hardware must be available first, which means releasing early and incrementally may not be an option. In these cases, releasing on a PI cadence may not be practical, and the planning and releasing activities are entirely decoupled.
- Releasing more frequently – For many, the goal is to release as often as possible—hourly, daily, weekly, and so on. Achieving frequent releases requires DevOps capabilities, an efficient continuous delivery pipeline, and architecture that supports incremental delivery. Even then, the periodic planning function still provides the cadence, synchronization, and alignment the enterprise needs to manage variability and limit deviations from expectations.
- Release on demand – For organizations building complex solutions (e.g., systems of systems), the cases above are probably overly simplistic. Big systems often contain different types of components and subsystems, each of which may leverage its release model. In that case, the most general model is: Release whatever you want and whenever it makes sense, within an appropriate governance and business model.
Decouple Release Elements from the Solution
Applying different release frequencies raises another question: Is a release a monolithic, all-or-nothing process? If so, the release strategy would be limited. Fortunately, that’s not the case. In fact, even simple solutions will have multiple release elements, each operating with different release strategies, as Figure 3 illustrates.
For example, the SAFe website that’s hosting this article has multiple release cycles:
- We can make a fix to a deployed version or address a security at any time (ad hoc, but expedited)
- We can update any article at any time and simply notify readers via blog post (high frequency)
- We can add new articles to the Advanced Topics section whenever significant new content is available (medium frequency)
- But making significant revisions to the framework is only done periodically, based on new content, new ways to render the Big Picture, and most importantly, when the market is ready for a new release (low frequency)
We call these separate flows value streamlets, as they represent a full, end-to-end flow of value within a Value Stream. Each streamlet (or value stream segment) can and should be managed to deliver value according to its own needs and pace. Identifying streamlets is critical to enable release on demand, as they allow the different elements of the solution to be released independently in a separate cadence.
Architect the Solution for Incremental Release
Achieving the different release strategies requires solutions to be architected for component (or service-based) deployability, releasability, and fast recovery. Intentional design of these attributes involves the collaboration of System Architects, Agile Teams, and deployment operations. The goal is a quick and flawless delivery process, ideally requiring little manual effort.
The following capabilities allow for a more flexible release process:
- Feature toggles – Feature ‘toggles or switches’ enable deployment of features into production without making them visible to the end-user. They help avoid the need for multiple code branches, allowing developers to deploy new features to production but only activate them when the enterprise is ready to consume them. After the feature has been released and deemed stable, the toggles are removed to avoid technical debt.
- Canary releases – Is a technique to reduce the risk of introducing a release in production, by slowly rolling out the change to a small subset of users, before rolling it out to the entire user base. Canary releases allow selective production testing and feature validation without impacting everyone.
- Dark launches – In some situations, it’s important to test new functionality in a production setting before making it available to customers, especially when there is high uncertainty on how well the system will meet its Nonfunctional Requirements (NFRs) under a production load.
Implementing these capabilities may require the organization to upgrade its infrastructure and architecture, including moving to standard technology stacks, decoupling monolithic applications with micro-services, data preparation and normalization, third-party APIs and data exchange, and logging and reporting tools.
Close the Loop on the Feature Hypothesis
Finally, after the feature is released, it’s time to evaluate the benefit hypothesis. Were the intended outcomes achieved? For example, should a canary release be extended to more customers? Turn off the feature toggles? The feature is considered done once we have an understanding of the production results. However, it may be necessary to add new backlog items to extend the functionality (persevere), or pivot (remove the feature) to find a different solution.
Building the Release
Building large-scale systems that are ready for release and fit for use requires an incremental development approach, as Figure 4 illustrates. There are four types of increments described next.
The first step in this process is that each Agile team ensures that it produces a working increment for the stories, features, and components they are responsible for by doing the following:
- Completing user stories from their Team Backlog, assuring that each meets its local (story) Definition of Done (DoD)
- Applying Continuous Integration practices with test automation to monitor and confirm progress
Every two weeks, teams build a system increment, an integrated stack of new functionality, representing all the backlog items completed by the ART during the current and all previous iterations. During the System Demo, stakeholders review the system increment and provide feedback. In this way, new features are added incrementally to the system, usually a few per iteration.
When developing large solutions, ARTs typically contribute to only a part of the solution, whether it’s a set of subsystems, some solution capabilities, or a mix. Integrated, verified, and validated, the solution increment must satisfy both the functional and Nonfunctional Requirements. This solution increment is the subject of the all-important Solution Demo, which brings all the system increments together into a working system.
Building solutions incrementally affect the release increment as a whole. As discussed earlier, decoupling solution elements allows for smaller, independent releases of the solution. These items might be at the team, system, or solution levels, so the same logic should be applied across the entire integration stack.
Some additional activities might be required when releasing. These include final, V&V, release notes, UAT, documentation, and training materials, as well as marketing activities. As much as possible, these activities are part of the DoD of previous increments. Some, however, will remain as release activities.
A Scaled Definition of Done
The continuous development of incremental system functionality requires a scaled DoD, an example is shown in Table 1.
Release management is the process of planning, managing, and governing solution releases, which helps guide the value stream toward the business goals. In some enterprises, especially those with significant regulatory and compliance criteria, this is a centralized, portfolio-level team or function that assures releases meet all the relevant business criteria.
In other circumstances, ART and Solution Train leadership and stakeholders from development operations, quality, sales, and other stakeholders take part in the release management and governance responsibilities.
In either case, the release management function facilitates the activities needed to help internal and external stakeholders receive and deploy the new solution. It also ensures that the most critical governance quality elements are appropriately addressed before deployment—particularly internal and external security, regulatory, and other compliance guidelines.
Release planning is part of the PI planning process. But that’s the easy part; the hard part is coordinating the implementation of all the capabilities and features over multiple iterations within a release. This is especially true when new issues, roadblocks, dependencies, and gaps in Vision and backlogs are uncovered. Due to these challenges, the scope of each release must be continually managed, revalidated, and communicated. Primary considerations include:
- Ensuring that the organization’s release governance is understood and adhered
- Communicating release status to external stakeholders
- Ensuring that an appropriate deployment plan is in place
- Coordinating with marketing and with Product and Solution Management on internal and external communications
- Validating that the solution meets relevant solution quality and Compliance criteria
- Participating in Inspect and Adapt (I&A) to improve the release process, value stream productivity, and solution quality
- Providing final authorization for the release
- Acting as a liaison with Lean Portfolio Management (LPM), as appropriate
- Participating in, and overseeing the final release activities
Many enterprises hold release management meetings weekly to address the following questions:
- Is the vision still understood, and are the trains and teams aligned for that purpose?
- Does everyone understand what they are building, and is it aligned with the understanding of the purpose of the Value Stream and current Strategic Themes?
- Are the trains tracking to the scheduled release dates?
- Is the appropriate quality built-in to the solution?
- What impediments must be addressed to facilitate progress?
This weekly meeting provides senior management with regular visibility into the release progress. It’s also the place to approve any scope, timing, people or resource adjustments necessary to assure the release. In a more continuous delivery environment, the participants closely monitor the release section of the Program Kanban, making sure items are released when needed, to the right audiences, managing dark and canary releases, verifying that hypotheses are evaluated, and that feature toggles are removed after production verification.
Learn More Kim, Gene and Jez Humble, Patrick Debois, John Willis. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press.
Last update: 12 November 2017