Ship Azure Pipelines templates in-box with ImageBuilder

[lbussell]

Background

If you make changes to ImageBuilder that change any part of how ImageBuilder is used in pipelines, you have to make the change changes in two phases:

1. Make source code changes to ImageBuilder, and then wait for a new build to be shipped and for an automated update to flow to this repo in the form of a PR.
2. Then, in that PR, you can make the necessary pipeline changes. Latest example: [main] Update Image Builder tag reference #2129.

There are two problems with this:

1. It splits changes across multiple commits. This makes it difficult to track down when/why certain pipeline changes were made.
2. It's not clear what changes are incoming when ImageBuilder is updated.

Proposal

ImageBuilder source code and pipeline changes should be coupled together.

Obviously, this does not work for ImageBuilder changes that change behavior in pipelines. However, we are in full control of all ImageBuilder updates. We can use ImageBuilder or any other process to emit whatever new content we want in eng/docker-tools/ at update time.

Practically, this could look like:

• ImageBuilder carries its own pipeline templates, and ships them in its build (alongside its executable during publish output). These would live under src/ImageBuilder.Pipelines/ or something.
• ImageBuilder has a new command which writes its templates to the disk.
• During ImageBuilder updates, we get the new ImageBuilder version to output its internal templates before submitting the automated PR.
• The docker-tools repo is still both the producer of and a consumer of ImageBuilder.

This repo would then carry two copies of the pipeline templates. When you make changes to ImageBuilder, you change the source code and the pipeline templates at the same time.

This approach also opens up interesting future opportunities like automatically generating some or all pipeline templates with ImageBuilder.

I see this proposed MVP (carrying two copies of pipeline templates) as being very low cost to implement.