Relocatable compiler in dune

[Alizter]

Thanks to David's great work upstream, OCaml 5.5 will be relocatable. Older versions of the compiler also have backports living in a custom opam repository https://github.com/dra27/opam-repository/blob/relocatable/packages/relocatable-compiler/relocatable-compiler.5.4.1.20251109.1/opam.

Supporting the relocatable compiler in package management would mean we can move away from toolchains and build the compiler like a regular package allowing it to be cached.

The constant rebuilds of the toolchains have been a common pain-point for users and the main focus of alternative proposals such as #12179.

In order to use this work in Dune, we need to be able to detect when a given compiler is relocatable. Checking the compiler version for 5.5 is a good start, but for older versions living in David's opam repo we can also check if we are using the relocatable meta package.

For the relocatable compiler to be fully integrated in Dune we will need to do the following work:

🧱 Done

• Be able to interpret the build plans appearing in David's opam repo. They include special opam variables such as build-id which we have so-far avoided interpreting in dune.
  • feat(pkg): interpret %{pkg-self:build-id} variable #13312
  • followup cleanup test(pkg): cleanup opam-var-pkg.t #13318
• Come up with a way to resolve symlinks in the install layout. The compiler will symlink ocamlc -> ocamlc.opt, this is fine but will mean that Dune will refuse to cache the directory target.
  • fix(pkg): resolve symlinks from (install) action  #13239
• Interpret conditional opam variables like %{pkg:ocaml-system:installed?system}
  • support %{pkg:foo?bar} syntax from opam #13322
  • feat(pkg): translate additional conditional syntax #13320
  • Typos in dra27/opam-repository#relocatable fix: missing colon in conditional dra27/opam-repository#27
  • Typos in ocaml/opam-repository Correct compiler variable generation in ocaml.5.5.0 opam-repository#29224
• Convert more undefined variables to default value at lock time #13660
  • fix(pkg): catch undefined package variables #13319
  • test: variables of undefined packages #13465
  • fix(pkg): unknown packages should have variables evaluate to empty #13466
  • test(pkg): add test for version variable of absent packages #13752
  • test(pkg): add test for installed variable of absent packages #13751
  • fix(pkg): substitute installed variable at solve time #13776
  • fix(pkg): substitute version variable in string interpolation #13785
• Set OPAMSWITCH in dune when building compiler #13887
  • fix(pkg): set OPAMSWITCH in package build environment #13888
• Detect if a compiler is relocatable (using the heuristic detailed above)
  • feat(pkg): relocatable compiler support #13321

🚧 Work-in-progress (in priority order)

• Work out why the compiler misses being stored in the cache occasionally
• Always use the relocatable compiler over the older OCaml ones. #14012
• relocatable ocamlbuild #13678
  • ocamlbuild failing to build dependents if sandbox is not properly cleared up #13467
  • overlay Add relocatable ocamlbuild.0.16.1, patched to remove a symlink ocaml-dune/opam-overlays#10
  • [pkg] Resolve directory symlinks in fetched targets #13792
• relocatable ocamlfind
  • need to match overlay from relocatable with our overlay