Configuring DU-Specific Certificates Manually

This page applies to releases where airctl provision-certs is not available (check with airctl provision-certs --help). On releases that have it, use Using Custom Certificates instead — it automates every step described on this page.

On this release, DU-specific certificates issued by cert-manager are not provisioned or managed by airctl. To use cert-manager-issued certificates (for example, via Let's Encrypt) instead of the shared self-signed wildcard certificate, you must create the ClusterIssuer and per-namespace Certificate resources yourself, and manage the internal replication label by hand so it doesn't overwrite the certificate you just issued.

Why manual steps are required

Every DU namespace normally receives the same self-signed wildcard certificate through a kubernetes-replicator annotation on a source secret:

metadata:
  annotations:
    replicator.v1.mittwald.de/replicate-to-matching: cert-manager-tls=http-wildcard-cert

Any namespace labeled cert-manager-tls=http-wildcard-cert receives a replicated copy of that secret. Without airctl provision-certs to manage this, you must remove the label from each namespace yourself before issuing a DU-specific certificate — otherwise the replicator overwrites it with the shared wildcard certificate again.

Naming constraint — read this before creating anything

The kplane du-upgrade Helm chart, run on every airctl upgrade, contains a template gated by use_du_specific_le_http_cert that creates a Certificate object with two hardcoded values:

{% raw %}
{{- if and (ne .Values.pmk_environment "airgap") (.Values.use_du_specific_le_http_cert) }}
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-http-certificate
  namespace: {{ .Values.namespace }}
spec:
  commonName: {{ .Values.namespace }}.{{ .Values.domain }}
  dnsNames:
    - {{ .Values.namespace }}.{{ .Values.domain }}
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  secretName: {{ .Values.http_cert_secret }}
{{ end }}
{% endraw %}

• issuerRef.name — your ClusterIssuer must be named exactly letsencrypt-prod, regardless of which ACME server or solver it actually uses. Any other name means the next airctl upgrade creates a Certificate pointing at a ClusterIssuer that doesn't exist.
• metadata.name — use exactly letsencrypt-http-certificate for the Certificate object in each namespace, so this chart recognizes and preserves it across upgrades.
• secretName stays http-wildcard-cert — every ingress/deployment already points at this secret name.

Prerequisites

Requirement	How to confirm
cert-manager installed	kubectl get pods -n cert-manager
AWS IAM credentials with Route53 permissions on the hosted zone (if using the Route53 DNS-01 solver)	route53:GetChange, route53:ChangeResourceRecordSets, and route53:ListHostedZonesByName (if not pinning a zone ID)
Which namespace is the actual replication source	kubectl get secret http-wildcard-cert -A -o yaml | grep -B5 replicate-to-matching
The real namespace names for your cluster	do not assume a generic naming pattern — confirm with kubectl get ns

1

Create the Route53 credentials secret

Only the AWS secret access key is treated as sensitive — the access key ID is a plain field on the ClusterIssuer spec.

kubectl create secret generic route53-credentials-secret -n cert-manager \
  --from-literal=secret-access-key='<AWS_SECRET_ACCESS_KEY>' \
  --dry-run=client -o yaml | kubectl apply -f -

2

Create the ClusterIssuer (staging first)

Validate against Let's Encrypt's staging environment before spending a production rate-limit slot.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod            # must be this exact name
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <your-email>
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - dns01:
          route53:
            region: <aws-region>
            hostedZoneID: <hosted-zone-id>
            accessKeyID: <aws-access-key-id>
            secretAccessKeySecretRef:
              name: route53-credentials-secret
              key: secret-access-key

kubectl apply -f clusterissuer.yaml
kubectl get clusterissuer letsencrypt-prod

READY: True here only confirms the ACME account registered — it needs a syntactically valid email and a private key, nothing more. It does not prove the AWS credentials or Route53 permissions are correct; that's only exercised when a certificate actually triggers a DNS-01 challenge.

kubectl patch --type=merge on the solvers list replaces the entire list entry, not just the fields you pass — patching in only region/hostedZoneID can silently wipe accessKeyID and secretAccessKeySecretRef. Always kubectl apply -f the full ClusterIssuer spec when changing anything inside solvers.

3

Validate with a throwaway certificate

Before touching any real namespace's secret, confirm the DNS-01 → Route53 → issuance pipeline works with a disposable certificate that doesn't share a secret name with anything real:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: route53-dns01-test
  namespace: <any-namespace>
spec:
  commonName: dns01test.<your-domain>
  dnsNames:
    - dns01test.<your-domain>
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  secretName: route53-dns01-test-tls

kubectl apply -f test-cert.yaml
kubectl get certificate route53-dns01-test -n <any-namespace> -w

Issuance typically takes a couple of minutes end to end — the ACME challenge usually reports valid well before the Certificate itself flips to Ready; give it time past that point before assuming it's stuck. Clean up once confirmed:

kubectl delete certificate route53-dns01-test -n <any-namespace>
kubectl delete secret route53-dns01-test-tls -n <any-namespace>

4

Back up every real http-wildcard-cert secret before touching anything

Deleting a cert-manager Certificate object does not restore the secret's prior content — it just stops managing it. Back up every namespace you plan to switch, including the replication source namespace, before doing anything else:

mkdir -p logs
for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl get secret http-wildcard-cert -n $ns -o yaml \
    > logs/DU-CERTS-baseline-${ns}.yaml
done

5

Remove the replicator label from every target namespace (except the source namespace)

The replication source namespace never carries this label — only namespaces receiving the replicated copy need it removed. Do this immediately before the next step — if the label lingers while cert-manager writes the new certificate, the replicator can overwrite it within seconds.

kubectl label ns <namespace-1> cert-manager-tls-
kubectl label ns <namespace-2> cert-manager-tls-

# confirm gone
kubectl get ns <namespace-1> <namespace-2> --show-labels | grep cert-manager-tls

6

Apply the per-namespace Certificate

Repeat for every namespace in scope, including the replication source namespace itself (it needs its own DU-specific certificate too):

for ns in <namespace-1> <namespace-2> <source-namespace>; do
kubectl apply -f - <<YAML
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-http-certificate
  namespace: ${ns}
spec:
  commonName: ${ns}.<your-domain>
  dnsNames:
    - ${ns}.<your-domain>
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  secretName: http-wildcard-cert
YAML
done

Known issue — CannotRegenerateKey on the replication source namespace. If the source namespace's secret holds an RSA key larger than cert-manager's default (2048-bit), cert-manager refuses to replace it under the default rotationPolicy: Never and reports CannotRegenerateKey instead of regenerating silently. Namespaces holding only a replicated copy of that key are usually unaffected — only the direct owner of the original secret tends to hit this. Fix:

kubectl patch certificate letsencrypt-http-certificate -n <source-namespace> --type=merge -p \
  '{"spec":{"privateKey":{"rotationPolicy":"Always"}}}'

Then wait — it proceeds to a fresh DNS-01 challenge and issues normally.

7

Wait for issuance

for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl get certificate letsencrypt-http-certificate -n $ns
done

Full issuance (challenge valid → order valid → certificate Ready) typically takes a couple of minutes per namespace against Route53 DNS-01 — the Ready flag can lag well behind the challenge itself going valid.

8

Verify what's actually being served

Ready: True only means cert-manager wrote the secret. Confirm the ingress is actually serving it, and confirm real trust (not curl -k, which skips verification and proves nothing):

echo | openssl s_client -connect <fqdn>:443 -servername <fqdn> 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

curl -v https://<fqdn>/<path> -o /dev/null 2>&1 | grep -E "SSL certificate verify|issuer:|subject:"

Staging certificates only verify successfully on a host that explicitly trusts the Let's Encrypt staging root — that's expected and not a sign anything is broken.

DNS-01 succeeding only proves the ACME server could see the _acme-challenge TXT record — it does not prove your test client (or anyone else) can resolve the FQDN. If DNS records for the FQDN aren't public yet, resolution may only work via manually maintained /etc/hosts entries on test clients.

9

Switch staging to production

Once staging issuance is proven, re-apply the ClusterIssuer with acme.server pointed at the production endpoint (https://acme-v02.api.letsencrypt.org/directory) — always a full apply, never a partial patch (see the solvers warning above). cert-manager does not proactively reissue an already-Ready certificate just because the issuer's server field changed, so force reissuance by deleting the secret (the Certificate object still exists, so cert-manager recreates the secret automatically):

kubectl apply -f clusterissuer-prod.yaml

for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl delete secret http-wildcard-cert -n $ns
done

Confirm real (non-staging) issuance:

kubectl get secret http-wildcard-cert -n <ns> -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -issuer
# expect: issuer=C = US, O = Let's Encrypt, CN = R... or E... (no "(STAGING)" prefix)

Let's Encrypt's production environment enforces a rate limit of 5 duplicate certificates per exact domain set per 7 days.

10

Restart deployments that read the certificate at startup

Pods mounting the secret as a volume pick up the renewed certificate within about 60 seconds via kubelet sync — no restart needed. Deployments that read it once at process startup need a rollout restart. Discover the real mounts per namespace rather than assuming names:

kubectl get pods -n <ns> -o json \
  | jq -r '.items[].spec.volumes[]? | select(.secret.secretName=="http-wildcard-cert") | .name'

Then for each deployment found:

kubectl rollout restart deployment/<name> -n <ns>

11

Update airctl-config.yaml

grep clusterIssuerName /opt/pf9/airctl/conf/airctl-config.yaml || \
  echo "clusterIssuerName: letsencrypt-prod" | sudo tee -a /opt/pf9/airctl/conf/airctl-config.yaml
sudo sed -i 's/clusterIssuerName:.*/clusterIssuerName: letsencrypt-prod/' \
  /opt/pf9/airctl/conf/airctl-config.yaml

Without this, the next airctl upgrade re-triggers the kplane du-upgrade job's default behavior, which can re-add the replicator label and start the overwrite race again.

Reverting to the original wildcard certificate

Do these steps in order — each assumes the previous one finished. Substitute your own namespace list and replication source namespace name.

1

Delete the per-namespace Certificate objects

This stops cert-manager from managing http-wildcard-cert. It does not change the secret's current content by itself — that's the next step.

for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl delete certificate letsencrypt-http-certificate -n $ns
done

2

Restore the original secret content from your backups

kubectl apply fails here — the backup YAML has no kubectl.kubernetes.io/last-applied-configuration annotation and carries a stale resourceVersion, so the API server rejects it with a conflict. Delete the current secret first, then create fresh from the backup file instead of applying over it:

for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl delete secret http-wildcard-cert -n $ns
  kubectl create -f logs/DU-CERTS-baseline-${ns}.yaml
done

Verify the restore actually put the original certificate back — check subject/issuer/SANs against what you expect (self-signed, issuer == subject):

for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl get secret http-wildcard-cert -n $ns -o jsonpath='{.data.tls\.crt}' \
    | base64 -d | openssl x509 -noout -subject -issuer -ext subjectAltName
done

3

Re-add the replicator label

Only on the namespaces that are supposed to receive the replicated certificate — never on the replication source namespace, since its own secret was already restored directly in the previous step.

kubectl label ns <namespace-1> cert-manager-tls=http-wildcard-cert
kubectl label ns <namespace-2> cert-manager-tls=http-wildcard-cert
kubectl get ns <namespace-1> <namespace-2> --show-labels | grep cert-manager-tls

4

Find and restart the deployments that mount the secret

Don't assume names — discover them fresh, and filter out one-shot Jobs (they already ran and don't need restarting):

for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl get pods -n $ns -o json | jq -r \
    '.items[] | select(.spec.volumes[]?.secret.secretName=="http-wildcard-cert") | "\(.metadata.name) owner=\(.metadata.ownerReferences[0].kind)/\(.metadata.ownerReferences[0].name)"'
done

Only restart pods owned by a ReplicaSet (i.e. a Deployment), one at a time so a slow rollout doesn't get bundled into an SSH timeout on the others:

kubectl rollout restart deployment/<name> -n <ns>
kubectl rollout status deployment/<name> -n <ns> --timeout=90s

5

Remove the airctl config key

Prevents the next airctl upgrade from re-triggering DU-specific certificate behavior via the kplane chart's use_du_specific_le_http_cert hook:

sudo sed -i '/clusterIssuerName:/d' /opt/pf9/airctl/conf/airctl-config.yaml
grep clusterIssuerName /opt/pf9/airctl/conf/airctl-config.yaml || echo "clusterIssuerName absent - correct"

6

(Optional) Clean up the ClusterIssuer and Secret

Only if you don't plan to redo the DU-specific certificate switch again soon. Leaving these in place is harmless once the Certificates from the first revert step are deleted.

kubectl delete clusterissuer letsencrypt-prod
kubectl delete secret route53-credentials-secret -n cert-manager

Upgrading to a Release with provision-certs

If you followed this page to configure a DU-specific certificate manually, revert to the self-signed certificate before upgrading to a release where airctl provision-certs is available. Upgrading directly, without reverting first, risks the upgrade silently clobbering your manually-configured certificate back to the shared self-signed wildcard.

Why this matters: the upgrade-safety logic that ships with provision-certs decides whether to skip re-adding the cert-manager-tls replication label based on a certMode field in /opt/pf9/airctl/conf/airctl-config.yaml. That field only ever gets set by running provision-certs itself. A DU configured through this manual page has no certMode set — as far as the new upgrade logic can tell, it looks identical to a DU that was never customized. On upgrade, it takes the default path: it re-adds the cert-manager-tls label to every namespace. Once that label is back, kubernetes-replicator resumes watching the secret and will overwrite your manually-issued certificate with the shared self-signed wildcard, usually within seconds of the label going on.

The safe migration path is:

1

Revert to self-signed before upgrading

Follow Reverting to the original wildcard certificate above in full, and confirm every namespace is back on the self-signed certificate (issuer == subject) before proceeding. This puts the DU into the state the upgrade path actually expects, so nothing gets caught mid-migration.

2

Run the upgrade

Perform airctl upgrade as normal. Since the DU is genuinely self-signed at this point, the default upgrade behavior (re-adding the replication label) is correct and does not clobber anything.

3

Re-provision your certificate the supported way

Once the upgrade completes and airctl provision-certs --help shows the command is available, re-issue your DU-specific certificate through it instead of by hand:

airctl provision-certs --cluster-issuer <cluster-issuer-name>

See Using Custom Certificates for the full walkthrough, including the BYOC wildcard and BYOC per-namespace modes if you'd rather supply your own certificate/key directly instead of using cert-manager. This run persists certMode to the config file, so every future upgrade is protected automatically — you will not need to repeat this revert-upgrade-reprovision sequence again.