Running Vault as Consul Connect CA with AWS IAM Auth does not respect "header_value" in Connect config.

[billliu1992]

Overview of the Issue

We are using Vault as the Consul Connect CA. We are running Consul and Vault on AWS, and would like to authenticate to Vault using AWS IAM Auth. We are setting the iam_server_id_header_value.

However, despite setting header_value in the Consul Connect config, we are unable to connect to Vault to use as a CA, and we get the following error despite setting "header_value" as per the docs.

[ERROR] connect.ca: Failed to initialize Connect CA: routine="CA initialization"
   error=
   | error configuring provider: Error making API request.
   |
   | URL: PUT https://vault.8080.co/v1/auth/aws/login
   | Code: 400. Errors:
   |
   | * error validating X-Vault-AWS-IAM-Server-ID header: missing header "X-Vault-AWS-IAM-Server-ID"

---

Reproduction Steps

1. Stand up Vault on AWS and bootstrap it

2. Set up Vault AWS Auth. We used the following TF script:

   resource "vault_auth_backend" "aws" {
      type        = "aws"
      description = "AWS IAM Authentication into Vault"
    }
    
    resource "vault_aws_auth_backend_client" "aws_iam" {
      backend                    = vault_auth_backend.aws.path
      iam_server_id_header_value = "vault.<our domain>"
    }
   

3. Create the Vault AWS auth backend roles and policy. We used the following TF script:

   variable "aws_region" {
      default = "us-east-1"
    }
    
    variable "env" {
      default = "production"
    }
    
    locals {
      vault_consul_service_mesh_ca_root  = "service_mesh/pki_root/${var.env}"
      vault_consul_service_mesh_ca_inter = "service_mesh/pki_int/${var.env}/aws/${var.aws_region}"
    }
    
    resource "vault_aws_auth_backend_role" "consul_server" {
      backend = local.vault_aws_auth_path
      role    = aws_iam_role.consul_server.name
    
      auth_type = "iam"
    
      bound_iam_principal_arns = [aws_iam_role.consul_server.arn]
    
      token_ttl     = 300
      token_max_ttl = 300
    
      token_policies = [vault_policy.consul_service_mesh.name]
    }
    
    
    resource "vault_policy" "consul_service_mesh" {
      name   = "consul-service-mesh-${var.env}-${var.aws_region}"
      policy = <<EOT
    path "/sys/mounts/${local.vault_consul_service_mesh_ca_root}" {
      capabilities = [ "create", "read", "update", "delete", "list" ]
    }
    
    path "/sys/mounts/${local.vault_consul_service_mesh_ca_inter}" {
      capabilities = [ "create", "read", "update", "delete", "list" ]
    }
    
    path "/sys/mounts/${local.vault_consul_service_mesh_ca_inter}/tune" {
      capabilities = [ "update" ]
    }
    
    path "/${local.vault_consul_service_mesh_ca_root}/*" {
      capabilities = [ "create", "read", "update", "delete", "list" ]
    }
    
    path "/${local.vault_consul_service_mesh_ca_inter}/*" {
      capabilities = [ "create", "read", "update", "delete", "list" ]
    }
    
    path "auth/token/renew-self" {
      capabilities = [ "update" ]
    }
    
    path "auth/token/lookup-self" {
      capabilities = [ "read" ]
    }
    EOT
    }
   

4. Set up Consul on AWS EC2 with the following connect config:

   connect {
     enabled = true
     ca_provider = "vault"
     ca_config {
       address = "${vault_address}"
       auth_method {
         type = "aws"
         mount_path = "${vault_aws_auth_path}"
         params {
           type = "iam"
           role = "${vault_role}"
           header_value = "${vault_aws_header_value}"
         }
       }
       root_pki_path = "/${vault_consul_service_mesh_ca_root}"
       intermediate_pki_path = "/${vault_consul_service_mesh_ca_inter}"
       leaf_cert_ttl = "72h"
       rotation_period = "2160h"
       intermediate_cert_ttl = "8760h"
       private_key_type = "rsa"
       private_key_bits = 2048
     }
   }
   

Note that we are setting connect.ca_config.params.header_value.

Expected behavior: No error message, Consul uses Vault as CA.

Actual behavior: We get the error message as shown in the overview.

Investigation

I believe this fixes the issue: main...billliu1992:consul:bill/connect-vault-ca-header-value

I pulled main and put some print statements and move around the code. I've verified that the same behavior above is exhibited in main.

I saw here: https://github.com/hashicorp/consul-awsauth/blob/6b9b26380433c3a5b442c5ee7dd47481513939ca/util.go#L242

We are setting the header value to LoginInput.ServerIDHeaderName, but we are not setting ServerIDHeaderName here:

consul/agent/connect/ca/provider_vault_auth_aws.go

Line 99 in 554b4ba

ServerIDHeaderValue: headerValue,

I manually added ServerIDHeaderName to "X-Vault-AWS-IAM-Server-ID" as per my branch above, and verified it manages to auth into Vault and use it as the CA. Not sure if this is the ideal fix.

Consul info for both Client and Server

I managed to reproduce this on main commit 554b4ba.

Running no clients, and one server.

Please let me know if you need any more information.