erichsen.dev

Importing secrets from AWS with External Secrets Operator

2023-07-28 7:44

The Goal

  • Be able to derive all kubernetes secret from AWS (except for the secret needed to talk to AWS)

Implementation

The setup of the operator, configuration of the secret store, secret definitions, and use of the secrets by the individual apps can be found here

Defining Secrets in AWS Parameter Store

I'll begin by entering the secrets into parameter store. The secrets I currently have to create manually in the cluster are:

Linode API Token - This token is used by external secrets to update the DNS records on the Linode nameservers.

Cluster Cert - This certificate key pair is used by the Kong data planes and Ingress controller to authenticate to the control planes hosted in Konnect. I upload the same public key to each control plane I use to keep this demo environment simple, but for a real deployment it would be better to have one per control plane.

Redis Token - The password for redis clients to authenticate to the redis cluster. Again I'm re-using this secret for all redis server deployments for simplicity.

Additionally, I will have a need to store ZeroSSL credentials to be used by the Acme plugin for KIC, so I'll go ahead and create them while I'm at it.

To set the secrets in my AWS account, I access AWS Systems Manager and then navigate to the Parameter Store. I'm using parameter store over the secret manager because I'm not planning to take advantage of the features that the secret manager provides at this point.

I create the following parameters, setting their value type as SecureString and setting the values to match the manually defined secrets in the kubernetes cluster.

/konnect/redis-password	
/konnect/tls.crt	
/konnect/tls.key	
/linode/linode_api_token	
/zerossl/eab_hmac_key	
/zerossl/eab_kid

Creating IAM User Access Credentials

Since I'm not using Amazon EKS, I need to be able to access parameter store from the outside. This means creating an IAM user and applying a policy that allows reading from parameter store:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameters",
                "ssm:GetParameter",
                "ssm:DescribeParameters"
            ],
            "Resource": "*"
        }
    ]
}

With this user in place, I create an access key credential and manually set the secret in the external secrets namespace in the kubernetes cluster. This secret is not shown in the commit, so I'll include the definition here:

apiVersion: v1
kind: Secret
metadata:
  name: aws-ssm
  namespace: external-secrets
data:
  AWS_ACCESS_KEY_ID: <access-key-is>
  AWS_SECRET_ACCESS_KEY: <secret-access-key>

For the time being, I will need to maintain this secret manually. However, I plan to followup later and create a github action that uses the Github-AWS OIDC integration to populate this last manual secret into the cluster by pulling it from AWS.

Now the prerequisite work has been completed and I can move on to configuring External Secrets.

Installing External Secrets Operator

I create the familiar chart folder and wrapper chart:

external-secrets/Chart.yaml

apiVersion: v2
name: external-secrets
version: 0.8.1
appVersion: 0.8.1
description: external-secrets
sources:
  - https://charts.external-secrets.io
type: application
dependencies:
  - name: external-secrets
    version: 0.8.1
    repository: https://charts.external-secrets.io

And have helm download the external-secrets chart:

❯ cd external-secrets && helm dep update

And I create the Argo Application visible in the commit

Adding the Cluster Secret Store

Now that the operator is installed on the cluster along with the AWS credentials in the manually-defined secret, I can define a ClusterSecretStore which will make the parameters in the AWS Parameter Store available to ExternalSecret CRDs the various service namespaces. If I didn't use a ClusterSecretStore, I'd need to put a SecretStore in every namespace.

I add a directory to hold helm templates and put the ClusterSecretStore definition inside. It simply lets the operator know that the store provider (aws) along with the provider-specific config of service (ParameterStore) and region. It also uses the IAM credentials I stored as a secret to supply the auth definition. The resulting definition is:

external-secrets/templates/aws-ssm-store.yaml


apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-parameter-store
spec:
  provider:
    aws:
      service: ParameterStore
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-ssm
            key: AWS_ACCESS_KEY_ID
            namespace: external-secrets
          secretAccessKeySecretRef:
            name: aws-ssm
            key: AWS_SECRET_ACCESS_KEY
            namespace: external-secrets

Now the operator is ready to supply values from Parameter Store to the ExternalSecret resources I'll be defining.

Defining and Consuming the ExternalSecret Resources

The next part is a lot of rinse-and-repeat, so I'll just note how to set up the secret for ExternalDNS. The rest should be obvious from viewing the commit.

For ExternalDNS, I need the Linode API token. I'll start by adding that ExternalSecret definition to a new templates directory for the external-dns chart.

external-dns/templates/linode-api-token-es.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: linode-api-token
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-parameter-store
    kind: ClusterSecretStore
  target:
    name: linode-api-token
    creationPolicy: Owner
  data:
  - secretKey: linode_api_token
    remoteRef:
      key: /linode/linode_api_token

When this is applied to the cluster, it will add a new secret to the namespace called linode-api-token with a key named linode_api_token (using this value for the secret key is required by external-dns). I decode the secret value for the linode_api_token and see it indeed matches the value from parameter store.

Finally, I can update the values file for ExternalDNS to refer to the new secret, and delete the old secret I had manually added to the external-dns namespace.

external-dns/values.yaml


external-dns:
  provider: linode
  linode:
-    secretName: linode-api-key   // manual secret name
+    secretName: linode-api-token // ES-managed secret name

After applying these changes, ExternalDNS is able to still manage the Linode nameserver entries for my account, and there is no more manually-entered secret making it happen!

Wrapping Up

I applied the same pattern to define secrets for Redis and the control plane authentication certificates in the Kong Hybrid and KIC namespaces, and then updated the values for each deployment (kic-managed dataplanes, hybrid dataplanes, and the kong ingress controller itself) to use these External Secret managed secret resources in place of the manual secrets they were previously using.

That's it! Now I only have to built a credential-free way to inject the IAM access secret into the cluster, and I'll fully managed and re-deployable secrets without introducing them in code.

Next Steps

  • Use a Github Action to inject the IAM credentials into the Linode Kubernetes Cluster
  • Use the ZeroSSL credentials defined in Parameter Store to configure the Acme plugin for KIC
Go back to Journal