Engineering teams can only be as efficient as the processes they employ during development. The need for increased efficiency is why software development has shifted from the “waterfall” approach to a more responsive, agile methodology.
In an agile development environment, quality software can be delivered consistently to suit the ever-changing needs of stakeholders and end users. However, the benefits of an agile approach aren’t exclusive to the planning and prioritization of feature work. The same philosophy should also be applied to Continuous Integration (CI) pipelines.
CI pipelines are relied upon to validate the quality and functionality of software before it’s released to customers. Unfortunately, these pipelines are often built to be unchanging monoliths using Graphical User Interface (GUI) tools provided by DevOps services such as Microsoft Azure DevOps. Changes to GUI pipelines can be error-prone and disruptive, as the concurrent modification of a pipeline remains unsupported. This doesn’t suit the agile philosophy and could lead to serious efficiency bottlenecks when requirements change or new functionality needs to be added.
What Is YAML, and Why Should I Use It?
Note: While the examples used in this blog post are within the context of Azure DevOps, these principles also apply to other DevOps services.
YAML is a data-serialization language used by DevOps services such as Azure DevOps to store integration and deployment configuration as code in a .yml file. In Azure DevOps, once you create an azure-pipelines.yml file in the root of your code repository, your specified configuration will automatically be used to create a CI pipeline.
The first benefit of a YAML pipeline is, unlike a GUI pipeline, the YAML file can be checked into source control. This allows an engineer to revert a pipeline back to a functioning state if a breaking change is introduced. With this safety net in place, modifications to the CI pipeline are less intimidating and can be performed by those without extensive pipeline experience.
The amount of concurrent work a pipeline can support is also an important consideration when architecting CI processes. GUI pipelines only support being modified by one person at a time, which can lead to scenarios in which one engineer’s pipeline changes can be interrupted and overwritten by another if there isn’t careful communication between the parties involved.
In contrast, YAML pipelines scale well with many concurrent editors. Since the YAML files are stored in source control, they can be edited on feature branches. This allows an engineer to test and verify their modifications work as they expect without fearing their work will interfere with other teams that might be working on the same pipeline.
Finally, like any other piece of code, changes made to the YAML pipeline are great candidates for code reviews. Proposed changes to the YAML pipeline can be openly discussed, critiqued, and improved during the code review process. All stakeholders have a chance to review pipeline changes as they’re happening, which can decrease knowledge silos and lead to better CI process outcomes in the future. Now, let’s look at how to convert your current GUI pipeline to a YAML pipeline.
Starting the Conversion Process
Each CI pipeline has a specific configuration to accommodate for the needs of the software using the pipeline. This means the YAML templates and tutorials available on the internet might fall short of the needs of your current GUI pipeline.
However, some DevOps tools allow you to export your existing pipeline as a YAML file. Depending on the complexity of your existing GUI pipeline, this feature might be able to generate a YAML pipeline that works right away.
In Azure DevOps, you can access this functionality by navigating to the editing view of your GUI pipeline and clicking the View YAML button for a specific job.
A window will open with the YAML code that represents the configuration of the pipeline job.
Once you insert the generated YAML for each job into a file named azure-pipelines.yml and store it at the root directory of your project, your new YAML pipeline will be ready to test.
The configuration of your pipeline might contain sensitive or non-static variables that can’t be programmatically converted from GUI to YAML. If this is the case, you’ll see an “undefined variable” message (as shown in the screenshot above), which means more work is required before the YAML code will execute. Typically, adding those variables manually into your new YAML pipeline will fix the issue. In Azure DevOps, you can do so by navigating to the editing tab for your YAML pipeline and selecting the Variables button.
However, if the complexity of your GUI pipeline makes bulk YAML generation impossible, there’s an alternate approach I have found useful.
Begin by breaking the existing pipeline jobs into individual tasks. The easiest tasks to begin with might be the build, testing, and publishing tasks because they’re often relied upon by other steps and stages. Once you have the basic tasks working, break up the rest of the tasks into “feature” work and prioritize them according to their immediate value. With this approach, each task can be tested and organized according to its goal within the pipeline without other potential issues obscuring the true error.
Tips and Tricks
The process of converting a GUI pipeline into a YAML pipeline is a natural opportunity to improve the architecture of your existing pipeline. Start by dismantling your complicated single stage build pipeline into logically separated stages. This makes the build pipeline easier to understand and creates an opportunity to parallelize tasks that aren’t dependent on one another. Parallelization within your YAML pipeline is a great way to increase the efficiency of your CI process. A more efficient CI process means faster build times and more agile engineering teams.
Another benefit of a multi-stage YAML pipeline is the ability to run a subset of build stages when a full build isn’t required. This is useful for testing isolated features or running a subset of the full test suite.
To accomplish this in a GUI pipeline, you would have to manually create a build and disable individual tasks, which defeats the purpose of an automated CI process. A YAML pipeline gives you the option to accomplish this with a stage level condition expression. The condition expression you provide determines whether the stage will run.
The following is an example of a condition expression that allows you to toggle MyStage based on a commit message flag from MyDependency:
- stage: MyStage
displayName: My Stage
pool:
vmImage: 'windows-2019'
dependsOn:
- MyDependency
condition: |
contains(stageDependencies.MyDependency.outputs['commitStep.commitMessage'], '{CommitFlag}')
Conclusion
Transitioning from a GUI build pipeline to a YAML build pipeline isn’t trivial, but the benefits will pay dividends in the long term. YAML pipelines are easier to maintain, optimize, and scale. A source-controlled YAML file introduces support for concurrent pipeline changes and reduces the risk of regression. The culmination of these benefits is a faster, more reliable build pipeline that allows engineering teams to focus on innovation, not their continuous integration.
