Feature Request: Allow Triggering Bundle Builds via API

[mjungsbluth]

What is the problem

While I appreciate the simplicity of the single binary design, the fixed period loop that polls for changes does not address use cases where you have an outside event that will need a fresh bundle build and also it prevents the control plane to be scaled horizontally.

When integrating in a CI/CD pipeline, esp. if mixed with push data sources, we would like to have external control on the trigger

Proposed solution

• Expose triggering a bundle build via the API
• Add a configuration option that plainly disables the built-in periodic trigger

This would both allow to provide external knowledge without increasing the complexity of the control plane itself and enable horizontal scaling of the service if all triggers are externalized.

[koponen]

Thanks for pointing this out. While the design choice of favoring simplicity was reasonable to start with, it would indeed be good to allow external drivers to trigger individual builds. Allowing the disabling of the default polling loop makes sense too.

[srenatus]

I think there might be different cases here?

1. trigger rebuild all bundles
2. trigger rebuild of a specific bundle

Also, I could imagine a query parameter to make the request synchronous vs a default of being an asynchronous trigger. If used in CI, you could trigger a synchronous build using curl and expect a 200 response status. You'd then know -- if the build goes green -- that your bundle build and deployment was successful.

@mjungsbluth what do you think? I might be overthinking it -- which of the variants and sync/async options here would be most fitting for your use case?

[srenatus]

📝 Another thing, I believe this warrants another permission. You can then have a token for CI usage that is only allowed to trigger builds, not CRUD anything.

Also, what would the service do if an external trigger comes in while it's already running a rebuild? Options would be

1. return some status/message indicating it's "already in progress"
2. queue a rebuild

I think (2.) is preferable. After all, a trigger would be sent to ensure that a build with the state of all data sources as of now will happen. If a previous build, that might be taking long, is still underway, we cannot guarantee that it's going to contain the fresh data.

We cannot do a rebuild right away, as we're using the file system for git repos, and it's not save to do so concurrently. (Gitsync might do a fetch while we're building a bundle off the files in the repo.)

[mjungsbluth]

I think there might be different cases here?

1. trigger rebuild all bundles
2. trigger rebuild of a specific bundle

I would start simple with the second case and the allow triggering a specific bundle.

This already allows triggering a „build all“ by looping externally while also potentially scaling the service horizontally.

As for sync/async. Depends on what is simpler but sync is always easier for clients to handle and can also return revision information or similar plus return errors immediately. I would also expect typical bundle builds to be way < 60s to within a normal timeout range.

[mjungsbluth]

📝 Another thing, I believe this warrants another permission. You can then have a token for CI usage that is only allowed to trigger builds, not CRUD anything.

Yes, that would probably be a good thing to separate concerns.

Also, what would the service do if an external trigger comes in while it's already running a rebuild? Options would be

1. return some status/message indicating it's "already in progress"
2. queue a rebuild

I think (2.) is preferable. After all, a trigger would be sent to ensure that a build with the state of all data sources as of now will happen. If a previous build, that might be taking long, is still underway, we cannot guarantee that it's going to contain the fresh data.

We cannot do a rebuild right away, as we're using the file system for git repos, and it's not save to do so concurrently. (Gitsync might do a fetch while we're building a bundle off the files in the repo.)

Queueing also makes it more complex, the queues can overrun, everything becomes async.

I would be fine to immediately return a 425, 429 or simply a 503 to indicate that the request cannot be executed at the given time, making it a client decision to retry. Most of the data and even policies are eventually consistent. So things happening a bit later is typically fine depending on the case

[mjungsbluth]

Also, to outline the use case: we are currently exploring building policy bundles with stacks and libraries with local ocd binaries (not running as a service) inside CI/CD. We don‘t follow a GitOps model for changes to Rego code where this would be more like you had in mind @srenatus

We‘d then run ocd as service to build context data bundles with the trigger being periodic and for push/event based datasources immediate. At least for us, the context data is updated independently from when CI/CD runs (triggered by git events)

Discovery documents would then stitch both worlds together.

[srenatus]

Queueing also makes it more complex, the queues can overrun, everything becomes async.

I would be fine to immediately return a 425, 429 or simply a 503 to indicate that the request cannot be executed at the given time, making it a client decision to retry. Most of the data and even policies are eventually consistent. So things happening a bit later is typically fine depending on the case

Yeah I think this is in line with the prototype of #124. A trigger event for a bundle will either pull it in front of the queue (if it's queued), or, if it's already getting executed, will cause it to immediately re-run once it's done.