Walk by a construction site and you’ll quickly recognize several types of activities. There’s the work constructing the actual building including the foundation, the frame, and the interior. There is also the work that enables the construction such as positioning the cranes, installing the workers’ elevators, and putting up the scaffolding and other safety structures.
Developing applications is similar. There’s the work required to develop the databases, applications, and user interfaces. But like construction, there’s also the investment in tools and practices that enable productive, higher quality, and reliable software development and delivery.
These practices are part of devops, and continuous integration and continuous delivery (CI/CD) are foundational practices for delivering quality software on more frequent schedules.
CI/CD pipelines and practices drive productivity, quality, and speed
CI/CD automates the application integration and delivery steps and standardizes application configurations. When developers check in new code, CI/CD pipelines run a sequence of builds, tests, data migrations, application deployments, service calls, and other scripted procedures to make the code changes available in targeted environments. With this automation in place, teams adapt their practices and look to check in code, integrate, test, and deliver application more frequently.
When fully matured, some organizations take this one step further and implement continuous deployments to production environments; however, such continuous deployment may not be optimal for all businesses and product types.
A recent survey of 549 software professionals provides insight on the benefits mature organizations practicing devops with CI/CD pipelines experience: 60 percent of respondents claimed that they can get new code to production in under seven days and almost 40 percent can deploy in under 24 hours. Nearly 50 percent of respondents claim their mean time to restore service when there is a production issue is under two hours.
Practicing CI/CD has other tangible benefits by aligning teams on development, test, and operational best practices:
- It addresses the misalignment between developers who want to push changes frequently with operations staff who want stable applications.
- It forces development teams to implement continuous testing so code changes can be validated before they are pushed to users.
- It encourages developers to make more small, incremental code changes that are more reliably integrated and tested versus larger changes.
- It provides more flexibility to teams that are asked to make quick fixes in parallel to the development of larger development efforts on new capabilities and functionality.
- It enables more functionality, performance, and data-driven testing so that higher quality applications can be delivered and fewer production defects introduced.
Continuous integration (CI) enables development productivity and quality
Let’s say you’re working on an upgrade to an existing application. Your team has been asked to develop new capabilities, fix some issues reported by users, address some technical debt by upgrading some of the underlying code libraries, and improve the performance of some commonly used reports. Your team consists of application and database developers, report writers, testers, and a system engineer. You are following a scrum development process and already have the application code in a Git repository.
But some things didn’t work well in the last release. First, it took several days for the application to be tested because most of the testing was done manually. When testing was completed, it produced a significant list of defects that needed to be addressed at the end of the development cycle. The cycle of bug fixing followed by long manual testing efforts forced the team to delay the release. Then, once deployed there were production issues because the application and database changes weren’t properly synchronized.
This time, you and the team are determined not to repeat the same mistakes, and you have the agreement from business and IT leaders to implement a continuous-integration practice. This changes your priorities and development practices tied to the release.
You can’t have a continuous-integration practice without regression tests and other continuous testing practices in place. So instead of jumping into the code to work on the required changes, the whole team partners with the testers and automates the most important tests that validate the build. When completed, you have multiple tools and scripts that can perform some of the more critical unit and functional tests.
The next thing you look at are the tools used to build the application. For the application, you recognize that these scripts work, but they don’t handle error conditions on a failed build gracefully. You also don’t have a process to push and rollback database changes, so you establish a basic one. You select a continuous integration tool that is used to sequence all the build and testing scripts into one automated process and then connect this to Git so that builds are triggered when code is merged.
With a continuous-integration tool and automation developed, you now consult with the team on development practices. The team can no longer develop in silos and wait for the end of a release cycle to validate that everything is working.
The first thing you do is encourage developers to commit their code changes daily. This sounds scary at first because it means that the code needs to be in a functional state and that the continuous-integration processes don’t fail because of a broken build or failing test. The team quickly realizes that it must work on bite-size chunks of code changes instead of developing a full feature or attempting a complex refactoring.
The team also agrees that a development is considered “done” only when tests validate the changed functionality and the tests can be incorporated into the continuous-testing process. This forces the team to reconsider how it commits and manages its time during the sprint so that testers have sufficient time to develop tests. The team discusses how to best apply test driven development methodologies where unit tests are developed before coding new functionality.
You then introduce several methodologies to better control what changes and functionality are production-ready. For very small changes, the team can check in to the main development branch with the requirement that all tests pass and the change can be deployed to production at any time. For slightly larger development exercises, you introduce feature flags, a way to deploy functional code that is not visible or running operationally to users. These feature flags can be enabled when the product is deployed to development and testing environments and disabled for production. For even larger developments or production hot fixes, you follow Gitflow and educate the team on how to leverage feature and hotfix branches in Git.
You’ll use a combination of these practices to establish a release-management strategy. For the priorities at hand, the team agrees to address the technical debt up front and release them to production over the first few weekly releases. The team then uses agile estimation and other techniques to decide which features to build in the development branch, others that will use feature flags, and one more complicated improvement that will be developed in a feature branch.
Continuous delivery (CD) standardizes and automates pushing changes
With a continuous-integration practice in place, some of the team members get antsy and want to get started with developing. But the system engineer is not so eager because although the builds and tests are automated, the steps to push the application and database changes into production and other environments are not.
The engineer has a multistep, part manual/part automated process to push new applications and an even more complicated process to roll them back. Furthermore, every deployment seems to be slightly different from previous ones and requires a review and update to these procedures depending on the types of changes being introduced. The engineer is tired of being accused of messing up a deployment because the process steps weren’t perfectly documented.
The engineer won’t be able to support weekly pushes and testing of a combination of changes happening in development and feature branches unless the team invests in continuous delivery.
The engineer does have some good news. He or she is already invested in infrastructure as code, so he or she can easily set up new environments in a consistent way. The matter at hand is to be able to push the application changes into these environments in a consistent way.
After selecting a continuous-delivery tool, the team first must make some code changes so that environments specific information is abstracted from the application. This means environment variables, endpoints, user names and passwords, and other runtime variables need to use variables that are configured by environment and defined in the continuous-delivery tool. To make the configurations easier to manage, they agree on naming conventions for the environment variables.
It also means some of the manual pre- and post-deployment steps need to be scripted so that they can be executed by the continuous-delivery tool. Steps to push data changes, synchronize service configuration, or run database indexes are now automated with their own validations. The team also automates performance and security tests so that these tests must pass before a delivery is confirmed to an environment. Last, every step in the delivery process has an equivalent step that is run if the delivery fails and needs to be rolled back.
The team does the application configurations using container technology and they debate the merits of using Docker, Kubernetes, Windows Server containers, and other container technologies. This ensures all the application services are configured and packaged in a uniform way and can be constructed consistently.
The team also thinks further about the data issues it experienced from the last release. Production data is always changing, so how can this data get mirrored into development and testing environment? The answer: This is also scripted so that production data is masked and can be shipped into development and testing environments on a schedule or on demand.
The team then rethinks its environment needs. Instead of one development environment, the team lets the creation of a new feature branch trigger the creation of a dedicated environment to develop this branch and the configurations necessary to deploy to this environment. The team also decides to set up two testing environments, one where the data is standardized and enables functional testing against a stable data set, and a separate environment with a full production data set to enable performance testing.
Establishing CI/CD in bite-size increments
The reality is that most organizations can’t easily mature a CI/CD practice in one shot or put business requirements on hold until one is established. Therefore, development organizations must change the wheels on the bus while the bus is driving.
In addition, many legacy applications can’t easily be converted holistically to CI/CD pipelines when some of the best practices can’t be implemented in reasonable timeframes or with the skills of the team that’s available.
So, in most organizations, CI/CD is matured over time, starting with the practices and automation that address the most important issues and avoid the biggest technical impediments.
A good place to start is to define and prioritize the problems and opportunities. Teams can then establish a devops epic in their agile backlogs and look to develop elements of their CI/CD pipeline over time and in parallel to addressing business needs.