Making an OIDC Issuer for my k3s Cluster

My homelab k3s cluster runs pods that need AWS access, and I did not want long-lived access keys sitting in Kubernetes secrets. The usual alternative IAM Roles Anywhere, needs an AWS Private CA. that is around $450 a month. On EKS this problem is already solved by IRSA. I wanted the same thing on a k3s cluster running in my home.

How It Works

The nice surprise is that the k3s API server already does almost everything:

1. It already signs service account JWTs with an RSA key.
2. It already serves an OIDC discovery document at /.well-known/openid-configuration.
3. It already serves the matching JWKS (the public keys) at /openid/v1/jwks.

The only missing piece is that AWS IAM fetches those two documents over the public internet, and my API server is on 192.168.x.x. Exposing the API server to the internet is not an option.

But IAM only needs the public half. So instead of exposing the cluster, I copy the two JSON files out and publish them on Cloudflare:

The cluster itself is never reachable from the internet. Only bits that needs to be public is two static files served from an R2 bucket.

The Setup

Three steps.

1. Point k3s at the public issuer. The API server needs to stamp tokens with the public URL instead of the default in-cluster issuer:

# /etc/rancher/k3s/config.yaml
kube-apiserver-arg:
  - service-account-issuer=https://oidc.rjha.dev
  - service-account-issuer=https://kubernetes.default.svc.cluster.local
  - api-audiences=https://kubernetes.default.svc.cluster.local,k3s

2. Deploy the Worker. An R2 bucket holds the two files; a Worker serves them on a custom domain. The custom domain is not optional — IAM requires a publicly-trusted certificate chain.

3. Extract and publish. A script pulls the two documents from the cluster and uploads them to R2:

kubectl get --raw /.well-known/openid-configuration
kubectl get --raw /openid/v1/jwks

Consuming It From AWS

First, register the issuer as an OIDC provider. This is a one time per account:

data "tls_certificate" "issuer" {
  url = "https://oidc.rjha.dev"
}

resource "aws_iam_openid_connect_provider" "homelab" {
  url             = "https://oidc.rjha.dev"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.issuer.certificates[0].sha1_fingerprint]
}

Then each workload gets an IAM role with a trust policy pinned to its exact service account:

data "aws_iam_policy_document" "metrics_agent_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.homelab.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "oidc.rjha.dev:aud"
      values   = ["sts.amazonaws.com"]
    }

    # Exact service account, never a wildcard.
    condition {
      test     = "StringEquals"
      variable = "oidc.rjha.dev:sub"
      values   = ["system:serviceaccount:agents:metrics-agent"]
    }
  }
}

resource "aws_iam_role" "metrics_agent" {
  name                 = "homelab-metrics-agent"
  assume_role_policy   = data.aws_iam_policy_document.metrics_agent_trust.json
  max_session_duration = 3600 # the floor — no reason for longer
}

On the cluster side, boto3 picks up credentials automatically when AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are set. No SDK changes, no credential code:

spec:
  serviceAccountName: metrics-agent
  containers:
    - name: agent
      env:
        - name: AWS_ROLE_ARN
          value: arn:aws:iam::<account>:role/<role>
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: /var/run/secrets/aws/token
      volumeMounts:
        - { name: aws-token, mountPath: /var/run/secrets/aws, readOnly: true }
  volumes:
    - name: aws-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              audience: sts.amazonaws.com
              expirationSeconds: 3600

Conclusion

This replaced the last static AWS credential in my cluster. Tokens rotate themselves every hour, the trust policy pins one exact service account, and the whole "public infrastructure" is two JSON files behind a Cloudflare Worker. Total cost: $0, which is a nice improvement over $450 a month.

The one manual bit left: if k3s ever rotates its signing key that is problem for future me