[feat]: Implement dynamic global target topics that resembles the functionality of reclass exports but simpler

[ramaro]

Description of the feature

Reclass exports allow exposing and querying parameters between targets. While great, it is tricky to wrap (at least for me) how it works due to its somewhat cryptic querying expressions and the fact that is is specific to reclass. See reclass exports and inventory queries here
There is also the current support for calling inventory_global() in a component to allow full scanning to all targets, however this is usually extremely expensive and inefficient.

For this reason, I am thinking that we can create new mechanics to offer similar functionality for this use case.
Let's call these "topics".

Here's my proposal:
Let's imagine that target-1 and target-2 need to expose the colour parameter for target-3 to consume. We do this by declaring the parameters under colours topic in parameters.kapitan.topics.colours.parameters:

# target-1
parameters:
   colour: red

   kapitan:
     topics:
       colours:
         parameters:
           colour: ${colour}

# target-2
parameters:
   colour: blue

   kapitan:
     topics:
      colours:
       parameters:
         colour: ${colour}

Kapitan's inventory logic (regardless of the inventory backend implementation) would then merge these topic parameters as the parameters of the new dynamic _topic_colours target. Note that the purpose of dynamic targets is for the consumption of topics, so these will not be included in the compile loop and therefore will not have any compiled outputs.

So _topic_colours's parameters would look like this:

# _topic_colours
parameters:
  targets:
    target-1:
      colour: red
    target-2:
      colour: blue

Now a component in target-3 can consume colours declared in the other targets via the inventory() (or maybe a new function):

# components.painter
from kapitan.inputs.kadet import inventory

def main():
  target_colours = {}

  for target, parameters in inventory(topic="colours").parameters.targets.items():
    target_colours[target] = {"colour", parameters.colour}
    
  return target_colours

this should compile the output:

# compiled/target-3/painter.yml
---
target-1:
  colour: red
target-2:
  colour: blue

I am thinking this would simplify what we can do today with reclass (and can't with omegaconf) without the cryptic query expressions in the inventory.

Additional context

No response

[simu]

I like this approach because it's much more digestible than the current reclass exports feature.

I've been trying to wrap my head around the reclass exports feature for projectsyn/reclass-rs#168 and have found that it's quite powerful, cryptic and fragile at the same time. Additionally, the exports feature is tangled all through Python reclass's evaluation logic, and hard to replicate in the (currently) much cleaner Rust implementation. Notably, implementing exports in reclass-rs creates a lot of implementation complexity due to the interdependence between Reclass refererences and export queries for the same parameter, which has the potential to cause a lot of duplicated evaluations, since export queries need rendered data from all Reclass nodes (Kapitan "targets").

I am thinking this would simplify what we can do today with reclass (and can't with omegaconf) without the cryptic query expressions in the inventory.

In addition to this, this would remove the big remaining discrepancy between the two reclass implementations (Python & Rust), potentially allowing more people to benefit from the performance of the Rust implementation.

Kapitan's inventory logic (regardless of the inventory backend implementation) would then merge these topic parameters as the parameters of the new dynamic _topic_colours target. Note that the purpose of dynamic targets is for the consumption of topics, so these will not be included in the compile loop and therefore will not have any compiled outputs.

Arguably, there's no need to inject topics as dynamic targets, since the topics aren't really targets in the way that Kapitan currently uses the term. However, by tracking topics in a separate data structure in Kapitan, we'd probably need to distribute an additional object to all compile subprocesses. However, I believe that keeping the current targets and topics cleanly separated in the Kapitan implementation would make future changes easier, although I've not fully thought this through yet.

Additionally, some components may iterate over all targets, and such components may get confused if they come across dynamic topic targets mixed in with "real" targets.

Now a component in target-3 can consume colours declared in the other targets via the inventory() (or maybe a new function):

I'd probably introduce a new function which enables components to query topics, in particular if we go with a separate internal data structure for topics.