Add containerd image verification to k8s and expand ecs-3/ecs-4 verifiers

[ginglis13]

Issue number:

Closes #4684

Description of changes:

• Add image verification settings for k8s-1.33, k8s-1.34, and k8s-1.35 variants
• Add image verification packages aws-signer-notation-plugin, digestion-image-verifier, notation-image-verifier, and bottlerocket-thar-be-image-verifiers, in k8s-1.33, k8s-1.34, and k8s-1.35 variants
• Add digestion-image-verifier and thar-be-image-verifiers in ecs-3 variants
• Add custom migration for image-verifier-plugins settings so that migration only applies to k8s variants

Pending #4827 to add these packages and settings for the new k8s-1.36 variants

Testing done:

k8s testing performed on:

{
  "os": {
    "arch": "x86_64",
    "build_id": "3adeb5d4",
    "pretty_name": "Bottlerocket OS 1.60.0 (aws-k8s-1.34)",
    "variant_id": "aws-k8s-1.34",
    "version_id": "1.60.0"
  }
}

ecs-3 testing performed on:

bash-5.2# apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "3adeb5d4",
    "pretty_name": "Bottlerocket OS 1.60.0 (aws-ecs-3)",
    "variant_id": "aws-ecs-3",
    "version_id": "1.60.0"
  }
}

• notation verifier on k8s

Details

Trust policy that allows signed image public.ecr.aws/o7o0w6s5/public-alpine-clone:latest:

bash-5.2# apiclient get settings.image
{
  "settings": {
    "image-verifier-plugins": {
      "enabled": true,
      "notation": {
        "trustpolicy": "ewogICAidmVyc2lvbiI6IjEuMCIsCiAgICJ0cnVzdFBvbGljaWVzIjpbCiAgICAgIHsKICAgICAgICAgIm5hbWUiOiJhd3Mtc2lnbmVyLXRwIiwKICAgICAgICAgInJlZ2lzdHJ5U2NvcGVzIjpbCiAgICAgICAgICAgICIqIgogICAgICAgICBdLAogICAgICAgICAic2lnbmF0dXJlVmVyaWZpY2F0aW9uIjp7CiAgICAgICAgICAgICJsZXZlbCI6InN0cmljdCIsCgkJCSJvdmVycmlkZSI6IHsKICAgICAgICAgICAgICAicmV2b2NhdGlvbiI6ICJza2lwIgoJCQl9CiAgICAgICAgIH0sCiAgICAgICAgICJ0cnVzdFN0b3JlcyI6WwogICAgICAgICAgICAic2lnbmluZ0F1dGhvcml0eTphd3Mtc2lnbmVyLXRzIiwKICAgICAgICAgICAgInNpZ25pbmdBdXRob3JpdHk6YXdzLXVzLWdvdi1zaWduZXItdHMiCiAgICAgICAgIF0sCiAgICAgICAgICJ0cnVzdGVkSWRlbnRpdGllcyI6WwogICAgICAgICAgICAiYXJuOmF3czpzaWduZXI6dXMtd2VzdC0yOjQ1ODM1ODk2MjIyNDovc2lnbmluZy1wcm9maWxlcy9ub3RhdGlvbl90ZXN0IiwKICAgICAgICAgICAgImFybjphd3MtdXMtZ292OnNpZ25lcjp1cy1nb3Ytd2VzdC0xOjI3ODMwODIxMTQzMDovc2lnbmluZy1wcm9maWxlcy9ub3RhdGlvbl90ZXN0IgogICAgICAgICBdCiAgICAgIH0KICAgXQp9Cg=="
      }
    }
  }
}

Test 3 cases: 1/ pull the signed image (allow), 2/ pull an unsigned image (deny) 3/ pull an image with a different signature not explicitly allowed in the trustpolicy (deny):

bash-5.2# ctr i pull public.ecr.aws/o7o0w6s5/public-alpine-clone:latest
...
Pulling from OCI Registry (public.ecr.aws/o7o0w6s5/public-alpine-clone:latest)  elapsed: 2.3 s  total:  3.6 Mi  (1.6 MiB/

bash-5.2# ctr i pull docker.io/library/hello-world:latest
ctr: image verifier bindir blocked pull of docker.io/library/hello-world:latest with digest sha256:f9078146db2e05e794366b1bfe584a14ea6317f44027d10ef7dad65279026885 for reason: verifier notation-image-verifier rejected image (exit code 1): image verification failed: Error: signature verification failed: no signature is associated with "docker.io/library/hello-world@sha256:f9078146db2e05e794366b1bfe584a14ea6317f44027d10ef7dad65279026885", make sure the artifact was signed successfully

bash-5.2# ctr i pull public.ecr.aws/o7o0w6s5/public-nginx-clone:latest
ctr: image verifier bindir blocked pull of public.ecr.aws/o7o0w6s5/public-nginx-clone:latest with digest sha256:bd1578eec775d0b28fd7f664b182b7e1fb75f1dd09f92d865dababe8525dfe8b for reason: verifier notation-image-verifier rejected image (exit code 1): image verification failed: Error: signature verification failedfor all the signatures associated with public.ecr.aws/o7o0w6s5/public-nginx-clone@sha256:bd1578eec775d0b28fd7f664b182b7e1fb75f1dd09f92d865dababe8525dfe8b

I also tested with private images with the following policy, which allows all system pods but requires signatures for the defined registryScopes:

 {
  "version": "1.0",
  "trustPolicies": [
    {
      "name": "aws-signer-ecr-private",
      "registryScopes": [
        "redacted.dkr.ecr.us-west-2.amazonaws.com/alpine-clone",
        "redacted.dkr.ecr.us-west-2.amazonaws.com/nginx-notation-signed"
      ],
      "signatureVerification": {
        "level": "strict",
        "override": {
          "revocation": "skip"
        }
      },
      "trustStores": [
        "signingAuthority:aws-signer-ts"
      ],
      "trustedIdentities": [
        "arn:aws:signer:us-west-2:redacted:/signing-profiles/notation_test"
      ]
    },
    {
      "name": "allow-all-others",
      "registryScopes": [
        "*"
      ],
      "signatureVerification": {
        "level": "skip"
      }
    }
  ]
}

When I apply a pod like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: notation-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notation-test
  template:
    metadata:
      labels:
        app: notation-test
    spec:
      containers:
      - name: notation-signed-alpine
        image: redacted.dkr.ecr.us-west-2.amazonaws.com/alpine-clone@sha256:24ba417e25e780ff13c888ccb1badec5b027944666ff695681909bafe09a3944
        command: ["sleep", "infinity"]
      - name: notation-signed-nginx
        image: redacted.dkr.ecr.us-west-2.amazonaws.com/nginx-notation-signed@sha256:17ae566734b63632e543c907ba74757e0c1a25d812ab9f10a07a6bed98dd199c
        command: ["sleep", "infinity"]
      - name: notation-unsigned-should-fail
        image: redacted.dkr.ecr.us-west-2.amazonaws.com/alpine-clone@sha256:216266c86fc4dcef5619930bd394245824c2af52fd21ba7c6fa0e618657d4c3b
        command: ["sleep", "infinity"]

I see only the unsigned fail to pull, rejected for image signature verification.

• digestion verifier on k8s

Details

Trust policy that allows digest of public.ecr.aws/docker/library/redis:7.4.8-alpine3.21 only:

bash-5.2# apiclient get settings.image
{
  "settings": {
    "image-verifier-plugins": {
      "digestion": {
        "trustpolicy": "ewogICJ2ZXJzaW9uIjogIjEuMCIsCiAgInRydXN0ZWREaWdlc3RzIjogWwogICAgInNoYTI1Njo3YWVjNzM0YjJiYjI5OGExZDc2OWZkODcyOWYxM2I4NTE0YTQxYmY5MGZjZGQxZjM4ZWM1MjI2N2ZiYWE4ZWU2IgogIF0KfQo="
      },
      "enabled": true
    }
  }
}

Test for allow/deny:

bash-5.2# ctr i pull public.ecr.aws/docker/library/redis:7.4.8-alpine3.21
...
Pulling from OCI Registry (public.ecr.aws/docker/library/redis:7.4.8-alpine3.21)        elapsed: 2.3 s  total:  16.5 M  (7.2 MiB/s)

bash-5.2# ctr i pull public.ecr.aws/docker/library/redis:6.2.20-alpine3.21
ctr: image verifier bindir blocked pull of public.ecr.aws/docker/library/redis:6.2.20-alpine3.21 with digest sha256:77697a75da9f94e9357b61fcaf8345f69e3d9d32e9d15032c8415c21263977dc for reason: verifier digestion-image-verifier rejected image (exit code 1): image verification failed: digest not in allowlist

• custom verifier on k8s

Details

Used the Dockerfile and Go program below. The Go program checks for existence of a mock config file; if present, pass, if not, fail. The Go binary is written to /.bottlerocket/rootfs/opt/civ/bin/custom-verifier, which is the bin_dir configured for containerd image verifiers. Configured as a bootstrap container on k8s instance with user-data script below b64 encoded for the "allow" case. For deny case, adjusted the script to not include configuration file.

FROM golang:1.21-alpine AS builder

WORKDIR /app
COPY custom-civ/ .

RUN CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o custom-verifier .

FROM public.ecr.aws/bottlerocket/bottlerocket-bootstrap:v0.2.15

RUN mkdir -p /.bottlerocket/rootfs/opt/civ/bin /.bottlerocket/rootfs/var/lib
RUN echo "civ-test-config" > /.bottlerocket/rootfs/var/lib/civ-config

COPY --from=builder /app/custom-verifier /.bottlerocket/rootfs/opt/civ/bin/custom-verifier
RUN chmod +x /.bottlerocket/rootfs/opt/civ/bin/custom-verifier

#!/bin/bash
# Create directories and copy binary to host
mkdir -p /.bottlerocket/rootfs/opt/civ/bin /.bottlerocket/rootfs/var/lib
cp /usr/local/bin/custom-verifier /.bottlerocket/rootfs/opt/civ/bin/custom-verifier
chmod +x /.bottlerocket/rootfs/opt/civ/bin/custom-verifier

# Create mock configuration file
echo "civ-test-config" > /.bottlerocket/rootfs/var/lib/civ-config

echo "Bootstrap setup complete"

// The mock verifier

package main

import (
        "os"
)

func main() {
        if _, err := os.Stat("/var/lib/civ-config"); err != nil {
                os.Exit(1)
        }
        os.Exit(0)
}

On the instance:

bash-5.2# ls /opt/civ/bin
custom-verifier  digestion-image-verifier  notation-image-verifier

bash-5.2# ls /var/lib/civ-config
/var/lib/civ-config

bash-5.2# ctr i pull docker.io/library/hello-world:latest
...
Completed pull from OCI Registry (docker.io/library/hello-world:latest) elapsed: 3.6 s  total:  42.3 K  (11.7 KiB/s)

With script that does not write config file to /var/lib/civ-config, pull rejected by custom-verifier.

• digestion verifier on ecs-3

Details

Same process as digestion verifier test on k8s with only public.ecr.aws/docker/library/redis:7.4.8-alpine3.21 in digest allowlist:

bash-5.2# ctr i pull public.ecr.aws/docker/library/redis:7.4.8-alpine3.21
...
Pulling from OCI Registry (public.ecr.aws/docker/library/redis:7.4.8-alpine3.21)        elapsed: 1.4 s  total:  16.5 M  (11.7 MiB/s)

bash-5.2# ctr i pull public.ecr.aws/docker/library/redis:6.2.20-alpine3.21
ctr: image verifier bindir blocked pull of public.ecr.aws/docker/library/redis:6.2.20-alpine3.21 with digest sha256:77697a75da9f94e9357b61fcaf8345f69e3d9d32e9d15032c8415c21263977dc for reason: verifier digestion-image-verifier rejected image (exit code 1): image verification failed: digest not in allowlist

• custom verifier on ecs-3

Details

Same userdata for image-verifier-plugins and boostrap-containers as the k8s testing above.

• Settings migration up/down testing

Details

On ecs-3 variant. After 1.61.0 upgrade, my image verifier settings remain:

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "3adeb5d4-dirty",
    "pretty_name": "Bottlerocket OS 1.61.0 (aws-ecs-3)",
    "variant_id": "aws-ecs-3",
    "version_id": "1.61.0"
  }
}
[ssm-user@control]$ apiclient get settings.image
{
  "settings": {
    "image-verifier-plugins": {
      "enabled": true,
      "my-fake-verifier": {
        "trustpolicy": "ewogICJ2ZXJzaW9uIjogIjEuMCIsCiAgInRydXN0ZWREaWdlc3RzIjogWwogICAgInNoYTI1Njo3YWVjNzM0YjJiYjI5OGExZDc2OWZkODcyOWYxM2I4NTE0YTQxYmY5MGZjZGQxZjM4ZWM1MjI2N2ZiYWE4ZWU2IgogIF0KfQo="
      }
    }
  }
}

Downgrade:

bash-5.2# signpost rollback-to-inactive
bash-5.2# reboot

...

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "c1f9ba0c",
    "pretty_name": "Bottlerocket OS 1.60.0 (aws-ecs-3)",
    "variant_id": "aws-ecs-3",
    "version_id": "1.60.0"
  }
}
[ssm-user@control]$ apiclient get settings.image
{
  "settings": {
    "image-verifier-plugins": {
      "enabled": true,
      "my-custom": {
        "trustpolicy": "ewogICJ2ZXJzaW9uIjogIjEuMCIsCiAgInRydXN0ZWREaWdlc3RzIjogWwogICAgInNoYTI1Njo3YWVjNzM0YjJiYjI5OGExZDc2OWZkODcyOWYxM2I4NTE0YTQxYmY5MGZjZGQxZjM4ZWM1MjI2N2ZiYWE4ZWU2IgogIF0KfQo="
      }
    }
  }
}

Existing settings remain for ecs ✅

Now for k8s:

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "c1f9ba0c",
    "pretty_name": "Bottlerocket OS 1.60.0 (aws-k8s-1.34)",
    "variant_id": "aws-k8s-1.34",
    "version_id": "1.60.0"
  }
}

bash-5.2# apiclient get settings.image
{}

updog update -i 1.61.0 -r -n

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "3adeb5d4-dirty",
    "pretty_name": "Bottlerocket OS 1.61.0 (aws-k8s-1.34)",
    "variant_id": "aws-k8s-1.34",
    "version_id": "1.61.0"
  }
}

Downgrade:

bash-5.2# signpost rollback-to-inactive
bash-5.2# reboot

...

[ssm-user@control]$ apiclient get os
{
  "os": {
    "arch": "x86_64",
    "build_id": "c1f9ba0c",
    "pretty_name": "Bottlerocket OS 1.60.0 (aws-k8s-1.34)",
    "variant_id": "aws-k8s-1.34",
    "version_id": "1.60.0"
  }
}
[ssm-user@control]$ apiclient get settings.image
{}
[ssm-user@control]$

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[piyush-jena]

We should add the settings for k8s-1.36 and ecs-4 (if they don't have them)

[ginglis13]

^ force push extends changes to ecs-4 and k8s-1.36

[ginglis13]

^ force pushes bump Release.toml and Twoliter.toml to 1.63.0, add migration in Release.toml, and simplify backwards migration logic.