The road to CI/CD

Remember those days when CI/CD was something developers did because they hated manual, repetitive work? Fast-forward to now and CI/CD is an industry-wide best practice that’s on top of every IT manager’s agenda.

Not without reason, as CI/CD is a powerful concept that helps teams deliver value faster and increase the quality of their output. But what exactly is CI/CD and what do you need to start using it?

First of all, let’s look at the definition: CI stands for continuous integration, and CD refers to continuous delivery or continuous deployment.

• Continuous integration is the process of validating software code and its interdependencies in an automated way. This process involves building, testing, and merging code to a central repository without any manual interference.
• Continuous delivery is the process of delivering the output of the continuous integration process to the next stage (e.g. a package repository or a release platform). This process is triggered automatically, but manual approval steps are still in place to perform verification checks or accumulate a batch of changes for a release window.
• Continuous deployment takes this one step further by removing manual approvals and automating the release process to a production environment.

Although the goal might seem to achieve the state of continuous deployment, this is not suited for all organizations, products, and environments. The application’s complexity, dependency on external integrations, or the maturity of an organization’s processes might dictate otherwise.

Let’s take a closer look at the required steps for implementing a CI/CD workflow and the prerequisites to make this a success, starting with the configuration of a pipeline.

Creating a Build Pipeline

Having a build pipeline is the first step to take when implementing CI/CD. A best practice is to write this build pipeline in code and store this in your repository. Stay away from configuration via a user interface, as this is error-prone and prevents rolling out changes in your build pipeline in an isolated and controlled way.

The build pipeline should automatically build every pull request of a feature branch to the main branch. The trunk-based development workflow is one of my favorite workflows that suits this process very well without adding a lot of complexity.

Depending on your software application, the pipeline often starts with a checkout of your code, followed by installing dependencies, building, testing, and publishing the build artifact. Depending on your situation, you can add different steps, such as code formatting checks, static code analysis, or code coverage analysis. The key is that all steps in this process must be fully automated, as these steps will be executed many times per day.

The duration of each step also needs to be optimized, as no one likes waiting on a pipeline to finish before a pull request can be merged. To keep pipeline execution time-limited, some teams decide to perform long-running integration tests on a nightly basis and only perform the most relevant integration tests in the pipeline.

The output of a build pipeline is a build artifact. This artifact should be used in subsequent steps in your CI/CD setup, as this artifact is tested and verified. Build only once and promote the result through the pipeline.

Merge to Main

A pull request is one of the most common methods in software development workflows to assure code quality and validate the implemented changes. When you work on a feature and you’re ready to have it reviewed, you create a pull request. This triggers the build pipeline and runs all necessary automated checks.

Next, one or more fellow developers should manually review your code, validate that the change is as intended, and provide feedback when needed. This creates a common understanding of the changes and also facilitates knowledge sharing within the team. After all automated and manual checks are performed, the pull request can be merged with the main branch.

To prevent your work from deviating too much from the main branch, it is advised to limit the time you’re working on a separate branch and to keep up to date with the main branch as much as possible.

When using trunk-based development, the main branch should always be ready to be deployed to production. This can, however, be a problem when you’re working on a feature that is technically ready, but not from a business perspective. Or it might be that parts of the feature are ready, but they should be released simultaneously. To solve this, you can use feature toggles (also known as feature flags), a technique to hide, disable, or enable functionality by encapsulating its code within a conditional statement.

Build and Deliver

After merging your pull request, the build pipeline will trigger once again but now also include the steps to package the software and store the artifact (e.g. in a package feed, Docker registry, or within the pipeline, ready to be used for the next stage).

The version number of your code should automatically be increased and registered within your code so that when your application runs in production, the monitoring software knows exactly what build version is used.

Deploy

The deployment step distinguishes between continuous delivery and continuous deployment. With continuous delivery, the release of the software is gated, as a manual approval step is necessary. This might be because manual verification is necessary due to limitations with using automated tests. Having sufficient code coverage using automated tests sometimes involves large investments, while a person to perform manual tests might be readily available. This can seem like an easy choice, but my advice is to always strive for automation when possible. In the long run, this pays off, as risks are minimized (scripts are much better at performing repeatable tasks than humans), it creates opportunities to increase release frequency, and it decreases lead time in the release process.

When you have sufficient code coverage using automated tests, you can think of implementing continuous deployment. Your software will then be automatically released to production without manual interference. Canary releases (referring to the canaries that miners used to take into mines to warn them of toxic gasses) can help you detect issues with the release. You can direct a limited amount of traffic (e.g. 5%) to the new version, monitor the logs, and when exception rates don’t increase, gradually direct all other traffic to the new version until all users have been reached. At this point, the old version can be taken offline.

As mentioned before, feature toggles are essential when using continuous deployment. Another good addition to your CI/CD toolset is dark launching, a process of releasing functionality to a subset of your users. This gives you real user feedback, lets you test for bugs, and monitor performance before the functionality is released to all of your users. Tools such as LaunchDarkly, ConfigCat, and Flagsmith can help you manage and integrate this pattern.

Prerequisites to Make CI/CD a Success

Having a CI/CD workflow doesn’t guarantee success, but it does help to increase quality and efficiency and to decrease lead time to generate business value. As a summary to help you get started on your journey to CI/CD, you can use the following checklist:

• Use build pipelines created in code.
• Use a git model that fits CI/CD (e.g. trunk-based development). Use short-lived branches and often update with the main branch.
• Enforce quality with a strict pull request and release workflow. Have at least one external reviewer. Automate code quality checks, such as test coverage, complexity, reliability, technical debt, etc. Automate code formatting checks and automate security checks on dependencies, OWASP vulnerabilities, etc.
• Optimize build pipeline time. Pick the right test for the job, such as unit tests, integration tests, visual regression tests, or performance tests. When integration tests take too long, investigate options to run them nightly. Also, use parallel execution whenever possible.
• Use feature toggles and optionally dark launching.
• Focus on monitoring and logging.

Some examples of CI/CD tools to get you started are Azure DevOps, Jenkins, Travis CI, CircleCI, AppVeyor, and GitHub Actions.

And remember:

• Automate everything (but only as far as reasonably possible).
• Continuous deployment is not for all teams. Continuous delivery can be the next best thing, depending on your team, product, organization, and environment.