Skip to content

hulumi-github-m1.md

Latest commit

 

History

History
208 lines (173 loc) · 75 KB

hulumi-github-m1.md

File metadata and controls

208 lines (173 loc) · 75 KB
Raw

Milestone 1 — /hulumi-threat-model GitHub scenarios + @hulumi/baseline.github.SecureRepository + sandbox-org E2E

Parent runbook: docs/slo/completed/RUNBOOK-hulumi-github.md. Read the runbook's Global Execution Rules (especially Rule 0 — the infra-only scope contract) + Global Entry Rules before starting.

Goal: After M1, the /hulumi-threat-model skill can produce framework-ID-cited threat models for the four highest-gap GitHub scenarios identified in research (github-oidc-trust-cloud-account, github-actions-supply-chain, github-app-token-exposure, github-self-hosted-runner), AND @hulumi/baseline.github.SecureRepository ships as a hardened-by-default ComponentResource declaring a single GitHub repo with private visibility, repository ruleset, security-and-analysis enablement, and Sandbox / Startup-Hardened tier defaults — verified end-to-end by an integration test that creates and destroys a real repo in a sandbox GitHub org.

Context: Research synthesis §"Threat-model skill scenarios" ranks (d) OIDC trust to cloud, (c) Actions supply-chain, (f) GitHub App token exposure, (e) self-hosted runners as the highest demand-minus-supply scenarios — those are the four shipped here. SecureRepository mirrors the existing packages/baseline/src/aws/secure-bucket.ts pattern (pulumi.ComponentResource, child registration via { parent: this }, assertValidTier(args.tier), hulumi:component/hulumi:tier tag triple). The four prebuilt skill scenarios extend the existing 5 AWS scenarios additively — no rename. pulumi-github v6.13.0 is the mandatory baseline (dossier §Library & Tool Evaluations) and is allow-listed as the only new runtime dependency in this milestone.

Important design rule: The Tier type is shared between AWS and GitHub. packages/baseline/src/github/index.ts re-exports Tier/TIERS/isTier/assertValidTier from packages/baseline/src/aws/tier.ts without duplication. packages/baseline/src/github/secure-repository.ts imports Tier from ../aws/tier. The same Sandbox / Startup-Hardened semantics apply: Startup-Hardened MUST emit strictly more controls than Sandbox (Policy H4 invariant — to be enforced in M3 by HulumiGithubHardeningPack). M1 ships the hulumi:component and hulumi:tier tags but omits hulumi:controls because the cis-github.ts and nist-ssdf-v1.1.ts mappings land in M3. M3 adds hulumi:controls as an additive change — this is recorded in M3's Compatibility Checklist as a known additive surface.

Refactor budget: Surgical addition only. New packages/baseline/src/github/ directory, four new skill scenarios, four new threat-model exemplars. The only existing files modified are packages/baseline/src/index.ts (re-export ./github), packages/baseline/package.json (add @pulumi/github to peer + dev deps), skills/hulumi-threat-model/SKILL.md (extend allowed-scenarios list), skills/hulumi-threat-model/scripts/list-scenarios.mjs (add four entries), skills/hulumi-threat-model/scripts/generate-threat-model.mjs (allow-list amendment captured during /slo-execute M1, 2026-04-26: extend BUNDLED_STUBS with GitHub framework prefixes so the new scenarios' STRIDE controls resolve to citations rather than Open-Questions entries — the full mapping ID tables still land in M3 per the staged-migration discipline; this is the minimum mechanical change to satisfy M1's BDD contract that the four GitHub scenarios cite ≥5 distinct framework IDs spanning CIS-GitHub-v1.2.0 / NIST-SSDF-v1.1 / OpenSSF-Scorecard / MITRE-ATT&CK / GitHub-Well-Architected), tests/skill-bdd/hulumi-threat-model.test.ts (extend coverage), scripts/exact-pin-guard.mjs and scripts/cooling-off-diff.mjs (add @pulumi/github to the pin-checked + cooling-off-checked lists), docs/slo/completed/RUNBOOK-hulumi-github.md Milestone Tracker (mark M1 in_progress → done).

Contract Block

Field Value
Inputs (Skill) /hulumi-threat-model <scenario-id> where scenario-id ∈ {<existing 5 AWS scenarios>, github-oidc-trust-cloud-account, github-actions-supply-chain, github-app-token-exposure, github-self-hosted-runner}. (Component) new SecureRepository(name, args) where args: SecureRepositoryArgs is a discriminated union on visibility: the default branch SecureRepositoryArgsPrivate requires visibility: "private" | "internal"; the public branch SecureRepositoryArgsPublic requires visibility: "public" AND acknowledgePublic: true AND publicJustification: string (a non-empty written reason captured in the hulumi:public-justification tag and the audit-event log). Shared fields: tier: Tier, defaultBranch?: string, description?: string, topics?: string[], vulnerabilityAlerts?: boolean (default true), secretScanning?: boolean (default true at startup-hardened, opt-in at sandbox), pushProtection?: boolean (default true at startup-hardened), provider?: github.Provider opt-in.
Outputs (Skill) Markdown file at docs/threat-model-<scenario-id>-<YYYYMMDD>.md in cwd, structurally identical to existing AWS scenarios (frontmatter name/scenario/generated_at/citations[]; STRIDE rows; framework-ID-only citations from CIS GitHub v1.2.0, NIST SSDF v1.1, MITRE ATT&CK T1195, OpenSSF Scorecard, GitHub Well-Architected SSDF; recommended Hulumi components forward-referencing OrgFoundation (M2) + policy pack (M3) + drift adapter (M4)). (Component) SecureRepositoryOutputs exposing repoFullName, repoNodeId, defaultBranch, rulesetId outputs.
Interfaces touched New stable surface: @hulumi/baseline/github#SecureRepository, SecureRepositoryArgs, SecureRepositoryOutputs, SECURE_REPOSITORY_COMPONENT_TYPE = "hulumi:baseline:github:SecureRepository". Re-exports from @hulumi/baseline/github#{Tier,TIERS,isTier,assertValidTier}. New skill scenario IDs (locked from M1).
Data classification Internal — the milestone creates ephemeral GitHub repos in a sandbox org during integration tests (no PII, no customer data, no production secrets). Skill output is markdown citing public framework IDs; no embedded user data. SecureRepository itself is declarative IaC — it does not handle user data at runtime. The integration test's GitHub App credentials are Restricted (handled per Forbidden shortcuts (g) below — never logged, never committed, OIDC-federated when in CI).
Proactive controls in play (a) C1 Define Security Requirements — this milestone's threat-model skill scenarios ARE the security requirements record for the GitHub variant; the four scenarios cite CIS GitHub v1.2.0 + NIST SSDF v1.1 + Scorecard + ATT&CK T1195 IDs. (b) C5 Validate All Inputs — assertValidTier(args.tier) rejects unknown tier values; args.visibility is constrained to a string-literal union that excludes "public"; scenario IDs are checked against an allow-list before the skill reads any file. (c) @hulumi/baseline.aws.SecureBucket (existing precedent) — SecureRepository mirrors the same ComponentResource discipline (child registration via { parent: this }, tag triple, tier-gated config). (d) C9 Implement Security Logging and Monitoring — skill writes an audit footer recording every refusal (license-boundary refusal, scenario-not-found, unknown framework ID requested) into the output markdown. (e) C10 Handle All Errors and Exceptions — component throws plain Error for tier and visibility violations (matches existing AWS pattern); no detail-leakage paths cross the pulumi up → CrossGuard boundary.
Abuse acceptance scenarios Four BDD rows in the table below cite tm-hulumi-github-abuse-N. Slug-keyed: tm-hulumi-github-abuse-license-boundary (skill refuses verbatim CIS GitHub Benchmark control text), tm-hulumi-github-abuse-public-visibility (component refuses visibility: "public" without explicit acknowledgePublic: true opt-in flag, deferred to M3 enforcement), tm-hulumi-github-abuse-scenario-id-traversal (skill rejects path-traversal scenario IDs via allow-list), tm-hulumi-github-abuse-sandbox-leak (integration teardown removes every test-tagged repo even on partial test failure). Threat-model row IDs trace to the four shipped scenarios (github-oidc-trust-cloud-account etc.) which are themselves the threat-model artifact in the Hulumi pattern.
Files allowed to change New: packages/baseline/src/github/{index.ts,secure-repository.ts,secure-repository.args.ts,secure-repository.outputs.ts}; packages/baseline/tests/github/secure-repository.test.ts; packages/baseline/tests/integration/github/secure-repository.integration.test.ts; skills/hulumi-threat-model/scenarios/github-{oidc-trust-cloud-account,actions-supply-chain,app-token-exposure,self-hosted-runner}.json; docs/threat-model-examples/github-{oidc-trust-cloud-account,actions-supply-chain,app-token-exposure,self-hosted-runner}.md; docs/slo/runbook-milestones/hulumi-github-m1.md (this file — Evidence Log only, in execution); docs/slo/lessons/hulumi-github-m1.md; docs/slo/completion/hulumi-github-m1.md. Modified: packages/baseline/src/index.ts (re-export ./github); packages/baseline/package.json (add @pulumi/github peer + dev dep at exact version with integrity hash); skills/hulumi-threat-model/SKILL.md (allowed-scenarios list); skills/hulumi-threat-model/scripts/list-scenarios.ts; tests/skill-bdd/hulumi-threat-model.test.ts; scripts/exact-pin-guard.mjs; scripts/cooling-off-diff.mjs; docs/slo/completed/RUNBOOK-hulumi-github.md Milestone Tracker. Files outside this milestone's allow-list — REFUSE TO TOUCH including any packages/baseline/src/aws/*, any packages/policies/src/, any packages/drift/src/, any existing skills/hulumi-threat-model/scenarios/aws-*.json, any examples/, any docs/slo/completed/RUNBOOK-hulumi.md.
Files to read before changing anything docs/slo/completed/RUNBOOK-hulumi-github.md (Global Execution Rules + this milestone in full); docs/slo/idea/hulumi-github.md (recommendation + tier decision); docs/slo/research/hulumi-github/synthesis.md; docs/slo/research/hulumi-github/dossier.md (Direct competitors + Open questions); packages/baseline/src/aws/secure-bucket.ts (pattern precedent); packages/baseline/src/aws/tier.ts; packages/baseline/src/aws/index.ts; skills/hulumi-threat-model/SKILL.md; one existing AWS scenario JSON (e.g. skills/hulumi-threat-model/scenarios/aws-multi-account-baseline.json) and its example output at docs/threat-model-examples/; tests/skill-bdd/hulumi-threat-model.test.ts.
New files allowed All "New" entries in Files allowed to change.
New dependencies allowed Runtime peer + dev: @pulumi/github@^6.13.0 (exact-pin via integrity hash, mirroring @pulumi/aws@7.27.0 discipline). Test-only dev: @octokit/webhooks-types@^7.x IF needed for type-checking integration test fixtures (decide during implementation; default no). No other runtime deps; the four skill scenarios remain runtime-dep-free .mjs plus JSON.
Migration allowed no — additive only. Existing AWS scenarios + components untouched.
Compatibility commitments SecureRepository, SecureRepositoryArgs, SecureRepositoryOutputs, SECURE_REPOSITORY_COMPONENT_TYPE are stable from M1 (no rename in v1.x). The four new skill scenario IDs are locked from M1. Existing AWS interfaces from Hulumi v1.0.0 unchanged: SecureBucket, AccountFoundation, Tier, HulumiHardeningPack, CisV5Pack, DriftClassifier, the four AWS adapters, /hulumi-threat-model skill name, SKILL.md frontmatter spec.
Forbidden shortcuts (a) NEVER embed verbatim CIS GitHub Benchmark / NIST SSDF / OpenSSF Scorecard text in scenario JSONs, exemplar markdown, or component source — IDs only. The license-boundary-lint script (existing) catches this. (b) NEVER ship a public-visibility shortcut — public visibility is reachable only through the SecureRepositoryArgsPublic branch of the discriminated union, which requires both acknowledgePublic: true and a non-empty publicJustification: string. The constructor refuses partial opt-in (one flag without the other) and refuses an empty / whitespace-only justification. (c) NEVER call child_process.exec, eval, or shell-interpolate user input in skill scripts or component code — argv-based spawning only (existing rule, extended). (d) NEVER commit GitHub App private keys, fine-grained PATs, or classic PATs to the repo — env vars only, integration tests skip when unset. (e) NEVER console.log token material in any skill or component path. (f) NEVER add @pulumi/github to the manifest without a 72h/24h cooling-off CI check — scripts/cooling-off-diff.mjs extension lands in this milestone. (g) NEVER leak sandbox-org repos: every integration test creates repos with the hulumi-github-m1-<test-id> description prefix, and afterAll runs a teardown that lists + deletes by prefix even on partial test failure. (h) NEVER import from packages/baseline/src/aws/ other than tier.ts.

Out of Scope / Must Not Do

  • No OrgFoundation — that's M2.
  • No HulumiGithubHardeningPack — that's M3.
  • No CIS GitHub Benchmark IDs in cis-github.ts mapping — that's M3 (gated behind WorkBench access; until then hulumi:controls is omitted from the M1 tag triple).
  • No drift classifier extension — that's M4. The /hulumi-threat-model skill output may forward-reference the M4 webhook adapter as "available in v1.x — see [docs/slo/completed/RUNBOOK-hulumi-github.md M4]" but ships no code.
  • No SLSA-L3 release, no atomic three-package version bump — that's M5.
  • No CodeQL queries, Semgrep rules, or custom secret-scanning patterns — Rule 0 (infra-only) is in force.
  • No GHEC-tier-only features: no EnterpriseSecurityAnalysisSettings, no audit-log REST polling, no secret_scanning_validity_checks_enabled. The skill scenarios may discuss these in their threat models but mark them as "v1.1 deferred — Team/Pro tier wedge."
  • No GitHub App / fine-grained PAT auth design beyond the cookbook stub linking to existing GitHub docs — the IaC role auth deep-dive is M2.
  • No examples/secure-repository-smoke/ example package in M1; smoke examples are M5 launch-readiness work.

Pre-Flight

  1. Complete the Global Entry Rules in ../RUNBOOK-hulumi-github.md.
  2. No docs/slo/lessons/hulumi-github-m0.md exists. Skip "read prior lessons" with a note in the Evidence Log.
  3. Read files listed in Files to read before changing anything.
  4. Copy the Evidence Log template into the milestone's Evidence Log section (already present below — clone the row shape).
  5. Re-state in working notes the four load-bearing constraints: (i) infra-only contract — no CodeQL/Semgrep/custom-pattern ships in this milestone or any future one; (ii) IDs-only citations, no verbatim framework text; (iii) Tier is shared with AWS, imported from ../aws/tier; (iv) hulumi:controls tag is deliberately omitted in M1 and added in M3 — record this in the M1 lessons file as an intentional staged-migration.
  6. Confirm HULUMI_INTEGRATION=1 + HULUMI_GITHUB_SANDBOX_ORG=<org-name> + HULUMI_GITHUB_APP_* env vars are available in the integration-test environment (skip integration test cleanly when unset; mock-runtime BDD always runs).

Files Allowed To Change

File Planned Change
packages/baseline/src/github/index.ts NEW: re-exports SecureRepository, SECURE_REPOSITORY_COMPONENT_TYPE, SecureRepositoryArgs, SecureRepositoryOutputs, Tier, TIERS, isTier, assertValidTier
packages/baseline/src/github/secure-repository.ts NEW: SecureRepository extends pulumi.ComponentResource; mirrors secure-bucket.ts shape; uses pulumi-github Repository + RepositoryRuleset + security-and-analysis settings
packages/baseline/src/github/secure-repository.args.ts NEW: SecureRepositoryArgs type
packages/baseline/src/github/secure-repository.outputs.ts NEW: SecureRepositoryOutputs type
packages/baseline/src/index.ts MODIFY: add export * as github from "./github"; after the existing aws and mappings re-exports
packages/baseline/package.json MODIFY: add @pulumi/github to peerDependencies + devDependencies at exact version with integrity hash
packages/baseline/tests/github/secure-repository.test.ts NEW: Vitest BDD covering happy path, invalid input, empty state, dependency failure, and abuse-case rows
packages/baseline/tests/integration/github/secure-repository.integration.test.ts NEW: real-sandbox-org test (gated on HULUMI_INTEGRATION=1 + HULUMI_GITHUB_SANDBOX_ORG); creates and tears down a repo
skills/hulumi-threat-model/scenarios/github-oidc-trust-cloud-account.json NEW: scenario JSON
skills/hulumi-threat-model/scenarios/github-actions-supply-chain.json NEW: scenario JSON
skills/hulumi-threat-model/scenarios/github-app-token-exposure.json NEW: scenario JSON
skills/hulumi-threat-model/scenarios/github-self-hosted-runner.json NEW: scenario JSON
skills/hulumi-threat-model/SKILL.md MODIFY: add four scenario IDs to the paths glob and description's prebuilt-scenarios list
skills/hulumi-threat-model/scripts/list-scenarios.ts MODIFY: add four entries with one-line descriptions
docs/threat-model-examples/github-oidc-trust-cloud-account.md NEW: exemplar output
docs/threat-model-examples/github-actions-supply-chain.md NEW: exemplar output
docs/threat-model-examples/github-app-token-exposure.md NEW: exemplar output
docs/threat-model-examples/github-self-hosted-runner.md NEW: exemplar output
tests/skill-bdd/hulumi-threat-model.test.ts MODIFY: extend the happy-path matrix to cover all 9 scenarios (5 AWS + 4 GitHub); add scenario-id-traversal abuse-case row; add license-boundary-refusal abuse-case row
scripts/exact-pin-guard.mjs MODIFY: extend the pin-checked dep list to include @pulumi/github
scripts/cooling-off-diff.mjs MODIFY: extend the cooling-off-checked dep list to include @pulumi/github
docs/slo/runbook-milestones/hulumi-github-m1.md MODIFY (during execution only): fill Evidence Log rows
docs/slo/lessons/hulumi-github-m1.md NEW (during exit): surprises, decisions, deltas-from-plan
docs/slo/completion/hulumi-github-m1.md NEW (during exit): changed files, tests added, docs updated
docs/slo/completed/RUNBOOK-hulumi-github.md Milestone Tracker MODIFY: M1 row → in_progress on start, done on exit

Step-by-Step

  1. Write BDD test stubs in packages/baseline/tests/github/secure-repository.test.ts (mock-runtime, mirroring packages/baseline/tests/secure-bucket.test.ts) covering happy path, invalid tier, invalid visibility, and the four abuse-case rows. Run — confirm all fail for "module not found" or expected-NotImplemented reasons.
  2. Write integration test stub at packages/baseline/tests/integration/github/secure-repository.integration.test.ts with HULUMI_INTEGRATION=1 + HULUMI_GITHUB_SANDBOX_ORG gating and afterAll teardown by hulumi-github-m1- prefix.
  3. Write the four skill scenario JSONs and four threat-model exemplars. The exemplars are the target shape for skill output; tests assert structural conformance (frontmatter, citation count, STRIDE-row count, recommended-Hulumi-component forward-references), NOT byte-for-byte equality.
  4. Implement SecureRepositoryArgs (discriminated union: SecureRepositoryArgsPrivate for "private" | "internal" visibility; SecureRepositoryArgsPublic requiring both acknowledgePublic: true and non-empty publicJustification: string) + SecureRepositoryOutputs types. The constructor enforces the opt-in invariant at runtime even when callers cast through as any; the audit-event log is written to stderr structured as { event: "public_visibility_acknowledged", justification, repoName, tier }.
  5. Implement SecureRepository component using pulumi-github Repository + RepositoryRuleset + security-and-analysis settings. Mirror secure-bucket.ts discipline: assertValidTier, { parent: this } child registration, hulumi:component + hulumi:tier tags (NOT hulumi:controls — see Important design rule). Tier defaults: Sandbox → ruleset with deletion + force-push protection; Startup-Hardened → adds required-status-checks placeholder, signed-commits-required, push-protection on, secret-scanning on, vulnerability-reporting enabled.
  6. Wire packages/baseline/src/index.ts re-export. Add @pulumi/github@^6.13.0 to packages/baseline/package.json peer + dev deps with integrity hash. Update scripts/exact-pin-guard.mjs and scripts/cooling-off-diff.mjs to include @pulumi/github.
  7. Run pnpm -r build && pnpm -r test && pnpm -r typecheck && pnpm -r lint && pnpm run lint:license-boundary && pnpm run lint:exact-pin-guard — all green except the integration test (which skips without env vars).
  8. Extend skills/hulumi-threat-model/SKILL.md allowed-scenarios list and skills/hulumi-threat-model/scripts/list-scenarios.ts. Make the existing tests/skill-bdd/hulumi-threat-model.test.ts happy-path matrix cover all 9 scenarios. Add the two new abuse-case rows (license-boundary refusal extended to CIS GitHub v1.2.0; scenario-id traversal).
  9. Run integration test against a real sandbox org (manual, on a developer laptop with creds in env vars): confirm a repo is created with the expected ruleset, security-and-analysis settings, and tag triple, then teardown removes it cleanly.
  10. Smoke tests + Self-Review Gate. Update Milestone Tracker to done, write lessons + completion files.

BDD Acceptance Scenarios

Feature: /hulumi-threat-model produces framework-ID-cited threat-model markdown for GitHub scenarios; @hulumi/baseline.github.SecureRepository rejects insecure inputs and creates correctly-tagged repos

Scenario Category Given When Then Threat-model row Control
Happy path — OIDC trust scenario happy path skill installed; scenario github-oidc-trust-cloud-account JSON exists user invokes /hulumi-threat-model github-oidc-trust-cloud-account output markdown at docs/threat-model-github-oidc-trust-cloud-account-<YYYYMMDD>.md contains ≥5 distinct framework-ID citations across CIS GitHub v1.2.0 / NIST SSDF v1.1 / Scorecard / ATT&CK T1195 / GitHub Well-Architected, ≥3 STRIDE rows, ≥1 forward-reference to OrgFoundation (M2), and zero verbatim CIS / SSDF prose n/a (happy path) n/a
Happy path — all 4 GitHub scenarios produce valid output happy path skill installed; all 4 GitHub scenario JSONs exist user invokes the skill on each of the 4 GitHub scenario IDs each invocation produces a markdown file matching the structural assertions above; no two outputs are identical n/a (happy path) n/a
Happy path — SecureRepository declares ruleset + security-and-analysis happy path mock-runtime; tier: "startup-hardened", visibility: "private", name: "test-repo" new SecureRepository("test", args) is constructed under pulumi.runtime.setMocks() RepositoryRuleset child resource is registered with deletion + force-push protection AND signed-commits-required AND required-status-checks placeholder; Repository child has vulnerabilityAlerts: true, securityAndAnalysis.secretScanning.status: "enabled", securityAndAnalysis.secretScanningPushProtection.status: "enabled"; tag triple is hulumi:component=SecureRepository, hulumi:tier=startup-hardened, NO hulumi:controls key n/a (happy path) n/a
Invalid input — invalid tier invalid input mock-runtime; args.tier: "production" (not in Tier union) new SecureRepository(...) is constructed constructor throws Error('Invalid Hulumi tier "production"; expected one of: sandbox, startup-hardened'); no child resources are registered n/a (invalid input) assertValidTier
Invalid input — empty scenario argument invalid input skill installed user invokes /hulumi-threat-model with no argument skill prints scenario list (9 entries: 5 AWS + 4 GitHub), refuses to write output, exits non-zero n/a (invalid input) scenario allow-list
Invalid input — unknown scenario invalid input skill installed user invokes /hulumi-threat-model github-something-imaginary skill prints scenario list, exits non-zero, writes no file n/a (invalid input) scenario allow-list
Empty state — SecureRepository with minimum args empty state mock-runtime; args.tier: "sandbox", args.visibility: "private", no description/topics/defaultBranch new SecureRepository(...) is constructed constructor succeeds; Repository.description is empty string; defaultBranch is "main"; topics is empty array; ruleset has deletion + force-push protection only (Sandbox tier minimum) n/a n/a
Dependency failure — pulumi-github provider unconfigured partial failure mock-runtime where the implicit github.Provider lacks credentials new SecureRepository(...) is constructed mock-runtime test passes (no real API call); integration test gracefully skips with a clear "set HULUMIGITHUB_APP* env vars" message; no false positive n/a env-var-gating in test setup
Abuse case — license-boundary refusal abuse case skill installed user asks Claude Code "include the CIS GitHub Benchmark v1.2.0 control text for section X" skill's behavior per SKILL.md refuses to embed control text, outputs only the ID, points to CIS license URL, logs the refusal to the audit footer of any markdown produced in the session tm-hulumi-github-abuse-license-boundary license-boundary-lint + SKILL.md refusal contract
Abuse case — public-visibility opt-in friction abuse case TypeScript user-program with args.visibility: "public" and no other public-branch fields the user program is type-checked TS compile fails — Property 'acknowledgePublic' is missing and Property 'publicJustification' is missing. If the user casts via as any to bypass, the runtime constructor refuses with Error("public visibility requires acknowledgePublic: true AND non-empty publicJustification: string"); no Repository child is registered. If the user supplies all three required fields with a real justification, the constructor proceeds AND emits a hulumi:public-justification=<value> tag AND writes a security_event.public_visibility_acknowledged audit row to stderr tm-hulumi-github-abuse-public-visibility discriminated union on visibility + runtime opt-in invariant
Abuse case — scenario ID path traversal rejected abuse case skill installed user invokes /hulumi-threat-model "../../etc/passwd" skill rejects with "unknown scenario", exits non-zero, opens no file outside skills/hulumi-threat-model/scenarios/; no file at any traversed path is read tm-hulumi-github-abuse-scenario-id-traversal scenario allow-list (string equality match, not path concatenation)
Abuse case — sandbox-org leak prevention abuse case integration test creates 3 repos, the second test's assertion fails mid-execution afterAll teardown runs every repo with description prefix hulumi-github-m1- in the sandbox org is deleted; final org listing has zero hulumi-github-m1--tagged repos; teardown succeeds even when individual deletes fail (logs but continues) tm-hulumi-github-abuse-sandbox-leak afterAll cleanup-by-prefix invariant
Schema / compatibility — SecureRepositoryArgs shape lock schema / compatibility packages/baseline/src/github/secure-repository.args.ts exists tests/skill-bdd/baseline-github-args.test.ts (added in this milestone) runs SecureRepositoryArgs exports the documented fields with the documented types; no field is renamed or removed in v1.x n/a type-layer schema lock
Schema / compatibility — output frontmatter for GitHub scenarios schema / compatibility skill produces output for any GitHub scenario output is parsed frontmatter has name, scenario, generated_at (ISO8601), citations[]; every citation has a non-empty url; framework names match the documented set (CIS GitHub v1.2.0 / NIST SSDF v1.1 / Scorecard / ATT&CK T1195 / GitHub Well-Architected SSDF) n/a output frontmatter lock

Regression Tests

  • All AWS BDD scenarios in tests/skill-bdd/hulumi-threat-model.test.ts continue to pass for the 5 AWS scenarios (no rename, no behavioral change).
  • All AWS BDD scenarios for SecureBucket and AccountFoundation continue to pass.
  • pnpm run lint:license-boundary continues to pass on the AWS surface.
  • pnpm run lint:exact-pin-guard passes on the extended dep list (now incl. @pulumi/github).
  • Skill SKILL.md continues to validate against the agentskills.io schema (extended paths glob + description does not break parsing).
  • packages/drift/tests/tla-alignment.test.ts continues to pass (this milestone does not touch verdict.ts or HulumiDrift.tla).

Compatibility Checklist

  • SecureRepository, SecureRepositoryArgs, SecureRepositoryOutputs documented in docs/components/ (one-line stub adequate; full reference doc in M5).
  • @pulumi/github@^6.13.0 exact-pinned with integrity hash in packages/baseline/package.json.
  • pnpm install && pnpm -r build && pnpm -r test && pnpm -r typecheck && pnpm -r lint && pnpm run lint:license-boundary && pnpm run lint:exact-pin-guard green on Node 20 LTS.
  • Tier from @hulumi/baseline/github is identity-equal to Tier from @hulumi/baseline/aws (re-export, not duplicate definition).
  • hulumi:controls tag is deliberately omitted from SecureRepository output — recorded in lessons, intentional staged-migration to M3.
  • License header present on every new .ts source file.
  • DCO sign-off required on every commit (CI enforcement carries over).
  • All 5 existing AWS skill scenarios continue to produce valid output unchanged.
  • No child_process.exec, eval, or shell-interpolation in any new file — tests/no-shell-exec.test.ts (existing) covers packages/*/src/; extend coverage to packages/baseline/src/github/ if not already glob-matched.

E2E Runtime Validation

Files: packages/baseline/tests/integration/github/secure-repository.integration.test.ts (Vitest) + extended tests/skill-bdd/hulumi-threat-model.test.ts.

E2E Test What It Proves Pass Criteria
runs_happy_path_on_all_four_prebuilt_github_scenarios Skill's GitHub happy path is not brittle For each of 4 GitHub scenarios, running the skill produces a markdown matching the structural assertions in BDD happy-path row
creates_and_destroys_real_sandbox_repo_at_startup_hardened_tier SecureRepository works end-to-end against real GitHub API Test creates a repo named hulumi-github-m1-<test-id> in HULUMI_GITHUB_SANDBOX_ORG; verifies via REST that ruleset + security-and-analysis settings are applied; teardown deletes the repo; final org listing has no hulumi-github-m1- repos
public_visibility_requires_full_opt_in Accidental public visibility is unreachable; deliberate public visibility is friction-y but possible with full opt-in TS compile fails on args.visibility: "public" without acknowledgePublic AND publicJustification; runtime cast-and-pass fails with Error("public visibility requires acknowledgePublic: true AND non-empty publicJustification: string"); full opt-in succeeds AND emits hulumi:public-justification tag AND writes security_event.public_visibility_acknowledged to stderr
scenario_id_path_traversal_blocked Skill cannot be tricked into reading arbitrary files Invocation with "../../etc/passwd" exits non-zero; process.cwd() filesystem state unchanged
sandbox_teardown_cleans_repos_on_partial_failure Sandbox-org leak prevention After a deliberately-failing test in the integration suite, afterAll removes all repos with the test prefix; no leaked state
pin_guard_catches_pulumi_github_drift scripts/exact-pin-guard.mjs covers the new dep Seeded fixture mutating @pulumi/github integrity hash → pnpm run lint:exact-pin-guard exits non-zero with file:line

Smoke Tests

  • pnpm install --frozen-lockfile && pnpm -r build && pnpm -r test && pnpm -r typecheck && pnpm -r lint && pnpm run lint:license-boundary && pnpm run lint:exact-pin-guard → all green.
  • (Optional, requires creds) HULUMI_INTEGRATION=1 HULUMI_GITHUB_SANDBOX_ORG=<org> HULUMI_GITHUB_APP_ID=... pnpm test:integration → integration test green; sandbox org has zero hulumi-github-m1- repos after the run.
  • In a fresh Claude Code session with the skill installed, invoke /hulumi-threat-model github-oidc-trust-cloud-account → output markdown appears in cwd with citations + STRIDE rows + forward-references to OrgFoundation (M2) + HulumiGithubHardeningPack (M3) + GithubWebhookFallbackAdapter (M4).
  • /hulumi-threat-model (no arg) → prints all 9 scenario IDs; exits non-zero.
  • git status shows no untracked test artifacts.
  • .gitignore covers any new generated files (e.g. integration-test transient state).
  • No hulumi:controls tag on SecureRepository outputs — confirmed by reading a generated example.

Evidence Log

Step Command / Check Expected Result Actual Result Pass/Fail Notes
Baseline tests pnpm -r build && pnpm -r test green pre-M1 Build done across 3 packages; AWS BDD + skill-bdd + 3 example smokes all green pass Confirmed pre-M1 baseline at 2026-04-26 11:09
BDD tests created pnpm --filter @hulumi/tests-skill-bdd test (post-skill-bdd extension, pre-impl) fail for module-not-found / CIS-GitHub-v1.2.0 not in citations 1 failure: OIDC test asserted CIS-GitHub-v1.2.0 framework prefix in citations but bare CIS-GitHub-v1.2.0 (no colon) was unresolved by parser pass Pre-implementation failure shape captured: expected [...] to include 'CIS-GitHub-v1.2.0'. Resolved by :PENDING-WORKBENCH placeholder ID (M3 fills with real WorkBench-resolved IDs)
Integration test stub pnpm --filter @hulumi/baseline test -- tests/integration/github skip with reason suite skipped; describe.skipIf(!ENABLED) printed "HULUMI_INTEGRATION!=1 — set to 1 to opt into integration suites" pass Sandbox-org App provisioning is the user-committed out-of-band step; integration suite remains green-on-skip on a developer laptop without creds
Skill scenarios + exemplars filesystem 4 scenario JSONs + 4 markdown exemplars present 4 JSONs at skills/hulumi-threat-model/scenarios/github-*.json, 4 exemplars at docs/threat-model-examples/github-*.md pass Each scenario cites ≥5 framework prefixes spanning the GitHub-specific frameworks
Component implementation filesystem + @hulumi/baseline/github SecureRepository + Args + Outputs + index re-export 4 new files in packages/baseline/src/github/ plus index re-export added to root index.ts pass Tier re-exported from ../aws/tier per design rule (no duplication)
Build / typecheck / lint pnpm -r build && pnpm -r typecheck && pnpm -r lint green All 3 packages build clean; typecheck clean; lint clean (5 initial warnings on as any casts → fixed by switching to as unknown as ...) pass One lint pass needed after initial impl to clean up unused-disable directives
Mock-runtime tests pnpm --filter @hulumi/baseline test -- tests/github 9 BDD tests pass 9 passed (2 happy paths × 2 tiers, 1 invalid tier, 1 empty state, 4 abuse cases for public-visibility opt-in, 1 schema lock) pass All M1 BDD scenarios from runbook table covered
Skill BDD extended pnpm --filter @hulumi/tests-skill-bdd test 9-scenario lister + 4 GitHub feature tests pass 24/24 pass (9 schema + 8 AWS feature + 7 GitHub feature including abuse cases) pass Lister contract: 5 AWS + 4 GitHub = 9 scenarios in declared order
Sandbox-org integration manual w/ env vars DEFERRED to first CI run Suite skipped on developer laptop pending HULUMIGITHUB_APP* setup DEFERRED Per /slo-execute prompt: integration test is gated by user's out-of-band sandbox App provisioning. The skill ships green-on-skip; sandbox run captured as M5 launch-readiness smoke test
License-boundary lint pnpm run lint:license-boundary OK license-boundary-lint: OK (IDs-only policy upheld across scanned trees) pass Existing AWS-side glob covers skills/ and docs/threat-model-examples/; license-boundary extension to packages/baseline/src/mappings/cis-github.ts is M3 territory
Exact-pin-guard pnpm run lint:exact-pin-guard OK with @pulumi/github added exact-pin-guard: OK (4 @pulumi/* deps match pinned hashes) pass Was 3, now 4 (+ @pulumi/github@6.13.0 integrity hash captured from pnpm-lock.yaml)
Cooling-off-diff scripts/cooling-off-diff.mjs extension @pulumi/github in PULUMI_PACKAGES PULUMI_PACKAGES = ["@pulumi/pulumi", "@pulumi/aws", "@pulumi/policy", "@pulumi/github"] pass Future bumps subject to 72h/24h cooling-off gate same as @pulumi/aws
Smoke tests pnpm install --frozen-lockfile=false && pnpm -r build && pnpm -r test && pnpm -r typecheck && pnpm -r lint && pnpm run lint:license-boundary && pnpm run lint:exact-pin-guard all green All green; 29 baseline tests pass + 5 skipped (4 are existing AWS-side test skips for integration paths, 1 is the new GitHub integration test skip-with-reason) pass Smoke confirms regression-free + new surface working
Test artifact cleanup git status --short only intentional new files + 12 modified 12 modified + 12 untracked, all intentional (4 scenario JSONs + 4 exemplars + 4 dirs of new sources/tests) pass No transient test artifacts
.gitignore review existing .gitignore covers Node/pnpm/Vitest/TLA+ no change needed existing patterns sufficient; integration-test transient state would land under .pulumi-state/ if exercised, already gitignored pass No new patterns required for M1

Definition of Done

  • All BDD scenarios pass (mock-runtime always; integration when env-vars present).
  • All E2E runtime validation tests pass.
  • pnpm -r test green; pnpm -r typecheck green; pnpm -r lint green; pnpm run lint:license-boundary green; pnpm run lint:exact-pin-guard green.
  • Smoke tests checked off.
  • Compatibility checklist complete.
  • No forbidden shortcuts present.
  • git status clean.
  • .gitignore covers all new generated files.
  • All 5 existing AWS skill scenarios produce valid output unchanged (regression-tested).
  • Sandbox-org integration test cleanly creates + tears down a repo (manual one-time verification on a developer laptop, recorded in completion summary).
  • docs/slo/lessons/hulumi-github-m1.md written (incl. the hulumi:controls-deferral decision).
  • docs/slo/completion/hulumi-github-m1.md written.
  • Milestone Tracker in docs/slo/completed/RUNBOOK-hulumi-github.md updated to done.

Post-Flight

  • docs/slo/completed/RUNBOOK-hulumi-github.md Milestone Tracker → M1 done.
  • docs/slo/completed/RUNBOOK-hulumi-github.md Component Summary Table — verify M1 row matches what was actually shipped.
  • docs/components/secure-repository.md — one-line stub if not present (full reference doc in M5).
  • docs/threat-model-examples/README.md — one-line index updated to mention 4 new GitHub exemplars.

Notes

  • This milestone ships no OrgFoundation, no policy pack, no drift adapter — those are M2 / M3 / M4. Skill output forward-references them with explicit "available in v1.x — see runbook M" markers.
  • Prior-lessons coverage category does not apply (greenfield for the GitHub variant).
  • The intentional hulumi:controls omission is the single most important design decision in this milestone — it must be recorded in the lessons file as a deliberate staged-migration, not an oversight.
  • The integration test is gated on real GitHub creds. CI in this milestone runs only mock-runtime BDD; the weekly integration workflow (.github/workflows/weekly-integration.yml) is extended in M5 to include the GitHub sandbox suite — flagged as an M5 follow-up, not an M1 deliverable.