Migrate threat detection to external threat-detect binary behind feature flag (#40166)

* Initial plan

* Initial plan for external threat detector migration

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Migrate gh-aw threat detection to external threat-detect binary (feature flag gh-aw-detection)

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Fix @q Bash 4.4+ compat in install script; add version assertion to test

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Update threat-detect to v0.2.2 and fix binary name to threat-detect-linux-amd64

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Enable gh-aw-detection feature flag in smoke-copilot and changeset workflows

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Fix changeset detection: replace engine: false with empty config to enable detection job

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Use bash/jq instead of Python for AWF chroot config patch in detection runs

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Merge main and recompile lock workflows

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Fix external detection job missing AWF install step

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Recompile smoke-copilot and changeset lock workflows

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Fix threat-detect invocation to use artifacts dir

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Strengthen external detector invocation assertion

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Install detection engine binary in external detector path

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Log skipped engine installs and clarify AWF step predicate

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Harden external detector engine install filtering and logging

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Clarify external detector engine-install fallback behavior

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Configure detection engine env like agent job

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Align detection job engine env with agent job

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Plan: handle recompile request

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Fix detection false-fail by writing result JSON output

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Mark gh-aw-detection feature as experimental

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Warn that gh-aw-detection is experimental

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Merge main and recompile smoke-copilot workflow

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* chore: outline plan for review feedback

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Fix flaky pin expectation and refresh wasm golden outputs

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Fix actions-lock container ordering in failing CI run

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com>

Author: Copilot

.github/aw/actions-lock.json

@@ -182,6 +182,11 @@
       "digest": "sha256:ca96b8acb27d8cf601a8faef86a084602cffa41d8cb18caa1e29ba4d16989d22",
       "pinned_image": "docker.io/mcp/brave-search@sha256:ca96b8acb27d8cf601a8faef86a084602cffa41d8cb18caa1e29ba4d16989d22"
     },
+    "ghcr.io/chopratejas/headroom:latest": {
+      "image": "ghcr.io/chopratejas/headroom:latest",
+      "digest": "sha256:af709363c4f9515a88a50939baec513be13c7cd778fb6635527b104d5173cb1e",
+      "pinned_image": "ghcr.io/chopratejas/headroom:latest@sha256:af709363c4f9515a88a50939baec513be13c7cd778fb6635527b104d5173cb1e"
+    },
     "ghcr.io/github/gh-aw-firewall/agent-act:0.25.29": {
       "image": "ghcr.io/github/gh-aw-firewall/agent-act:0.25.29",
       "digest": "sha256:97b4cc14dc2123a45b9d5b9927489f66882dec5857de6afc0e5bab257be92ef1",
@@ -621,11 +626,6 @@
       "image": "semgrep/semgrep:latest",
       "digest": "sha256:17d89ddd91a7729bbd5de09402f7f79a70204289e2a94635086e9db532a495f2",
       "pinned_image": "semgrep/semgrep:latest@sha256:17d89ddd91a7729bbd5de09402f7f79a70204289e2a94635086e9db532a495f2"
-    },
-    "ghcr.io/chopratejas/headroom:latest": {
-      "image": "ghcr.io/chopratejas/headroom:latest",
-      "digest": "sha256:af709363c4f9515a88a50939baec513be13c7cd778fb6635527b104d5173cb1e",
-      "pinned_image": "ghcr.io/chopratejas/headroom:latest@sha256:af709363c4f9515a88a50939baec513be13c7cd778fb6635527b104d5173cb1e"
     }
   }
 }