Skip to content

instructions.md

Latest commit

 

History

History
253 lines (173 loc) · 7.75 KB

instructions.md

File metadata and controls

253 lines (173 loc) · 7.75 KB
Raw

Livy TLSN Terraform Deployment

Deploy the Livy TLSN Notary service on Google Cloud using Terraform. A normal deploy should be:

cd infra
terraform apply -var-file="environments/test.tfvars" -auto-approve

Terraform creates or updates the infrastructure, writes the cloud-init payload to the VM metadata, and the VM runs the bootstrap scripts automatically. You should not need to SSH into the VM or run the scripts by hand during a normal deployment.

The docs app version of this runbook lives at docs/content/docs/deployment/terraform-gcp.mdx.

Prerequisites

  • Google Cloud project with billing enabled.
  • gcloud authenticated for the target project.
  • Application Default Credentials configured for Terraform.
  • Terraform installed locally.
  • Intel Trust Authority API key.
  • Optional DNS A record for HTTPS.

For the test environment:

gcloud config set project livy-infra
gcloud auth application-default set-quota-project livy-infra

If authentication is missing, run:

gcloud auth login
gcloud auth application-default login

Do not commit local credential files or Terraform variable files containing secrets.

Configuration

Create a local tfvars file from the example:

cp infra/environments/example.test.tfvars infra/environments/test.tfvars

Set these values in infra/environments/test.tfvars:

project_id = "livy-infra"

trustauthority_api_key = "your Intel Trust Authority API key"
tlsn_branch            = "benchmark"
domain_name            = "tlsn.livylabs.xyz"
certificate_email      = "contact@livylabs.xyz"

infra/environments/test.tfvars is ignored by git because it contains secrets.

Deploy

Run Terraform from infra:

cd infra
terraform init
terraform apply -var-file="environments/test.tfvars" -auto-approve

After apply, Terraform prints the instance IP, HTTP endpoint, HTTPS endpoint, SSH command, and Intel Trust Authority test commands.

What Terraform Creates

  • A TDX-enabled Compute Engine VM named ${environment}-notary-instance.
  • A custom VPC named ${environment}-vpc.
  • A subnet named ${environment}-subnet.
  • A Cloud Router and Cloud NAT for outbound internet access.
  • Firewall rules for SSH, internal traffic, the TLS Notary port, HTTP, and HTTPS.
  • A service account named ${environment}-notary-sa.
  • Logging and Monitoring IAM roles for the service account.

The VM runs Ubuntu 24.04 with Intel TDX confidential computing enabled.

Bootstrap Flow

Cloud-init runs these scripts on first boot:

/opt/scripts/core.sh
/opt/scripts/install.sh
/opt/scripts/run.sh
/opt/scripts/setup-https.sh

core.sh prepares the host:

  • Installs OS build dependencies.
  • Fixes /home/livy ownership.
  • Verifies /dev/tdx_guest and kernel TDX state.
  • Installs Rust.
  • Installs Intel Trust Authority CLI.
  • Validates /home/livy/config.json.
  • Configures restricted passwordless sudo for trustauthority-cli.

install.sh builds TLSN:

  • Creates /home/livy/tls-notary-config.
  • Creates the Notary config and signing key if missing.
  • Clones or updates https://github.com/livylabs/tlsn.git on the configured tlsn_branch.
  • Builds notary-server and the notary-tee proxy example in release mode.

run.sh starts services:

  • Writes systemd units for tls-notary-server and tls-notary-proxy.
  • Starts and enables both services.
  • Waits for local health checks on ports 7047 and 7048.

setup-https.sh configures ingress:

  • Installs Nginx and Certbot.
  • Proxies /healthcheck, /info, /session, and /notarize to 127.0.0.1:7047.
  • Proxies /api/v1/prove and job artifact endpoints to 127.0.0.1:7048.
  • Requests a Let's Encrypt certificate when domain_name is set and resolves to the VM external IP.
  • Leaves HTTP proxying configured and skips certificate issuance when DNS is not ready.

DNS And HTTPS

For automatic HTTPS on the first apply, domain_name must resolve to the VM external IP before setup-https.sh runs.

Check the current Terraform IP:

cd infra
terraform output -raw instance_external_ip

Check DNS:

dig +short tlsn.livylabs.xyz

Expected public health check after HTTPS is configured:

curl https://tlsn.livylabs.xyz/healthcheck

Expected response:

Ok

If DNS is not ready during first boot, the VM still deploys and Nginx serves HTTP. To avoid manual VM commands, point DNS at the instance IP before the VM's first bootstrap reaches setup-https.sh.

Destroy And Apply

The current Terraform VM uses an ephemeral external IP:

access_config {}

That means terraform destroy followed by terraform apply can assign a different IP. The services should still deploy, but HTTPS will only configure automatically when domain_name already resolves to the new VM IP.

For a repeatable destroy/apply flow with HTTPS, use a reserved static IP and point DNS at that static IP once.

Destroy the test deployment from infra:

cd infra
terraform destroy -var-file="environments/test.tfvars"

Bootstrap Status

After terraform apply, the VM may still be running cloud-init. Check the status with:

gcloud compute ssh test-notary-instance --zone=us-central1-a --project=livy-infra \
  --command="sudo cloud-init status --long"

Status meanings:

  • status: running: installation is still in progress. This is normal while packages install and TLSN builds.
  • status: done: installation finished.
  • status: error: installation failed. Check /var/log/cloud-init-output.log.

The last_update field can show an early Unix epoch timestamp while cloud-init is still starting. Treat status and extended_status as the useful fields.

Wait until bootstrap exits:

gcloud compute ssh test-notary-instance --zone=us-central1-a --project=livy-infra \
  --command="sudo cloud-init status --wait"

Follow the install log:

gcloud compute ssh test-notary-instance --zone=us-central1-a --project=livy-infra \
  --command="sudo tail -f /var/log/cloud-init-output.log"

Check service state:

gcloud compute ssh test-notary-instance --zone=us-central1-a --project=livy-infra \
  --command="systemctl is-active tls-notary-server tls-notary-proxy nginx"

Check recent service logs:

gcloud compute ssh test-notary-instance --zone=us-central1-a --project=livy-infra \
  --command="sudo journalctl -u tls-notary-server -u tls-notary-proxy --no-pager -n 100"

Verify Deployment

Run the repository deployment test from the project root:

./test-deployment.sh

The test checks the Terraform-managed instance, service status, health endpoint, TDX state, built binaries, service logs, and Intel Trust Authority CLI configuration.

Run the TEE impact benchmark from the project root:

./benchmark-tee-impact.sh

The script clones or updates livylabs/tlsn on the benchmark branch, builds the benchmark binaries in release mode, starts the local TLS fixture target, then runs 25 samples for each mode against the deployed Notary at https://tlsn.livylabs.xyz: no TEE, TEE without local verification, and TEE with local TDX quote verification. It writes CSV results, a summary, and per-run logs under benchmark-results/. The summary reports both group-level overheads and paired per-iteration overheads so outliers are visible instead of hidden.

The benchmark does not resolve test-server.io in DNS. It connects to the local fixture at 127.0.0.1:4000; test-server.io is the TLS server name and HTTP Host header because the fixture certificate is issued for that name and USE_FIXTURE_CA=true trusts the fixture CA.

Retry or isolate one mode with BENCHMARK_MODES, for example:

BENCHMARK_MODES=tee_verify ITERATIONS=1 ./benchmark-tee-impact.sh

Manual checks:

cd infra
terraform output -raw instance_external_ip
curl https://tlsn.livylabs.xyz/healthcheck