> For the complete documentation index, see [llms.txt](https://docs.platform9.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.platform9.com/private-cloud-director/getting-started/self-hosted/using-du-specific-certificates-manual.md).

# Configuring DU-Specific Certificates Manually

{% hint style="warning" %}
This page applies to releases where `airctl provision-certs` is **not** available (check with `airctl provision-certs --help`). On releases that have it, use [Using Custom Certificates](/private-cloud-director/getting-started/self-hosted/using-custom-certificates.md) instead — it automates every step described on this page.
{% endhint %}

On this release, DU-specific certificates issued by `cert-manager` are not provisioned or managed by `airctl`. To use `cert-manager`-issued certificates (for example, via Let's Encrypt) instead of the shared self-signed wildcard certificate, you must create the `ClusterIssuer` and per-namespace `Certificate` resources yourself, and manage the internal replication label by hand so it doesn't overwrite the certificate you just issued.

### Why manual steps are required

Every DU namespace normally receives the same self-signed wildcard certificate through a `kubernetes-replicator` annotation on a source secret:

```yaml
metadata:
  annotations:
    replicator.v1.mittwald.de/replicate-to-matching: cert-manager-tls=http-wildcard-cert
```

Any namespace labeled `cert-manager-tls=http-wildcard-cert` receives a replicated copy of that secret. Without `airctl provision-certs` to manage this, you must remove the label from each namespace yourself before issuing a DU-specific certificate — otherwise the replicator overwrites it with the shared wildcard certificate again.

### Naming constraint — read this before creating anything

The `kplane` du-upgrade Helm chart, run on every `airctl upgrade`, contains a template gated by `use_du_specific_le_http_cert` that creates a `Certificate` object with two **hardcoded** values:

```yaml
{% raw %}
{{- if and (ne .Values.pmk_environment "airgap") (.Values.use_du_specific_le_http_cert) }}
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-http-certificate
  namespace: {{ .Values.namespace }}
spec:
  commonName: {{ .Values.namespace }}.{{ .Values.domain }}
  dnsNames:
    - {{ .Values.namespace }}.{{ .Values.domain }}
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  secretName: {{ .Values.http_cert_secret }}
{{ end }}
{% endraw %}
```

* `issuerRef.name` — your `ClusterIssuer` **must** be named exactly `letsencrypt-prod`, regardless of which ACME server or solver it actually uses. Any other name means the next `airctl upgrade` creates a `Certificate` pointing at a `ClusterIssuer` that doesn't exist.
* `metadata.name` — use exactly `letsencrypt-http-certificate` for the `Certificate` object in each namespace, so this chart recognizes and preserves it across upgrades.
* `secretName` stays `http-wildcard-cert` — every ingress/deployment already points at this secret name.

### Prerequisites

| Requirement                                                                                          | How to confirm                                                                                                          |
| ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `cert-manager` installed                                                                             | `kubectl get pods -n cert-manager`                                                                                      |
| AWS IAM credentials with Route53 permissions on the hosted zone (if using the Route53 DNS-01 solver) | `route53:GetChange`, `route53:ChangeResourceRecordSets`, and `route53:ListHostedZonesByName` (if not pinning a zone ID) |
| Which namespace is the actual replication source                                                     | `kubectl get secret http-wildcard-cert -A -o yaml \| grep -B5 replicate-to-matching`                                    |
| The real namespace names for your cluster                                                            | do not assume a generic naming pattern — confirm with `kubectl get ns`                                                  |

{% stepper %}
{% step %}

#### Create the Route53 credentials secret

Only the AWS secret access key is treated as sensitive — the access key ID is a plain field on the `ClusterIssuer` spec.

```bash
kubectl create secret generic route53-credentials-secret -n cert-manager \
  --from-literal=secret-access-key='<AWS_SECRET_ACCESS_KEY>' \
  --dry-run=client -o yaml | kubectl apply -f -
```

{% endstep %}

{% step %}

#### Create the ClusterIssuer (staging first)

Validate against Let's Encrypt's staging environment before spending a production rate-limit slot.

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod            # must be this exact name
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <your-email>
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - dns01:
          route53:
            region: <aws-region>
            hostedZoneID: <hosted-zone-id>
            accessKeyID: <aws-access-key-id>
            secretAccessKeySecretRef:
              name: route53-credentials-secret
              key: secret-access-key
```

```bash
kubectl apply -f clusterissuer.yaml
kubectl get clusterissuer letsencrypt-prod
```

{% hint style="info" %}
`READY: True` here only confirms the ACME account registered — it needs a syntactically valid email and a private key, nothing more. It does **not** prove the AWS credentials or Route53 permissions are correct; that's only exercised when a certificate actually triggers a DNS-01 challenge.
{% endhint %}

{% hint style="warning" %}
`kubectl patch --type=merge` on the `solvers` list replaces the entire list entry, not just the fields you pass — patching in only `region`/`hostedZoneID` can silently wipe `accessKeyID` and `secretAccessKeySecretRef`. Always `kubectl apply -f` the full `ClusterIssuer` spec when changing anything inside `solvers`.
{% endhint %}
{% endstep %}

{% step %}

#### Validate with a throwaway certificate

Before touching any real namespace's secret, confirm the DNS-01 → Route53 → issuance pipeline works with a disposable certificate that doesn't share a secret name with anything real:

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: route53-dns01-test
  namespace: <any-namespace>
spec:
  commonName: dns01test.<your-domain>
  dnsNames:
    - dns01test.<your-domain>
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  secretName: route53-dns01-test-tls
```

```bash
kubectl apply -f test-cert.yaml
kubectl get certificate route53-dns01-test -n <any-namespace> -w
```

Issuance typically takes a couple of minutes end to end — the ACME challenge usually reports `valid` well before the `Certificate` itself flips to `Ready`; give it time past that point before assuming it's stuck. Clean up once confirmed:

```bash
kubectl delete certificate route53-dns01-test -n <any-namespace>
kubectl delete secret route53-dns01-test-tls -n <any-namespace>
```

{% endstep %}

{% step %}

#### Back up every real `http-wildcard-cert` secret before touching anything

Deleting a `cert-manager` `Certificate` object does **not** restore the secret's prior content — it just stops managing it. Back up every namespace you plan to switch, including the replication source namespace, before doing anything else:

```bash
mkdir -p logs
for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl get secret http-wildcard-cert -n $ns -o yaml \
    > logs/DU-CERTS-baseline-${ns}.yaml
done
```

{% endstep %}

{% step %}

#### Remove the replicator label from every target namespace (except the source namespace)

The replication source namespace never carries this label — only namespaces receiving the replicated copy need it removed. Do this **immediately before** the next step — if the label lingers while `cert-manager` writes the new certificate, the replicator can overwrite it within seconds.

```bash
kubectl label ns <namespace-1> cert-manager-tls-
kubectl label ns <namespace-2> cert-manager-tls-

# confirm gone
kubectl get ns <namespace-1> <namespace-2> --show-labels | grep cert-manager-tls
```

{% endstep %}

{% step %}

#### Apply the per-namespace Certificate

Repeat for every namespace in scope, including the replication source namespace itself (it needs its own DU-specific certificate too):

```bash
for ns in <namespace-1> <namespace-2> <source-namespace>; do
kubectl apply -f - <<YAML
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-http-certificate
  namespace: ${ns}
spec:
  commonName: ${ns}.<your-domain>
  dnsNames:
    - ${ns}.<your-domain>
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  secretName: http-wildcard-cert
YAML
done
```

{% hint style="warning" %}
**Known issue — `CannotRegenerateKey` on the replication source namespace.** If the source namespace's secret holds an RSA key larger than `cert-manager`'s default (2048-bit), `cert-manager` refuses to replace it under the default `rotationPolicy: Never` and reports `CannotRegenerateKey` instead of regenerating silently. Namespaces holding only a *replicated copy* of that key are usually unaffected — only the direct owner of the original secret tends to hit this. Fix:

```bash
kubectl patch certificate letsencrypt-http-certificate -n <source-namespace> --type=merge -p \
  '{"spec":{"privateKey":{"rotationPolicy":"Always"}}}'
```

Then wait — it proceeds to a fresh DNS-01 challenge and issues normally.
{% endhint %}
{% endstep %}

{% step %}

#### Wait for issuance

```bash
for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl get certificate letsencrypt-http-certificate -n $ns
done
```

Full issuance (challenge `valid` → order `valid` → certificate `Ready`) typically takes a couple of minutes per namespace against Route53 DNS-01 — the `Ready` flag can lag well behind the challenge itself going `valid`.
{% endstep %}

{% step %}

#### Verify what's actually being served

`Ready: True` only means `cert-manager` wrote the secret. Confirm the ingress is actually serving it, and confirm real trust (not `curl -k`, which skips verification and proves nothing):

```bash
echo | openssl s_client -connect <fqdn>:443 -servername <fqdn> 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

curl -v https://<fqdn>/<path> -o /dev/null 2>&1 | grep -E "SSL certificate verify|issuer:|subject:"
```

Staging certificates only verify successfully on a host that explicitly trusts the Let's Encrypt staging root — that's expected and not a sign anything is broken.

{% hint style="info" %}
DNS-01 succeeding only proves the **ACME server** could see the `_acme-challenge` TXT record — it does not prove your test client (or anyone else) can resolve the FQDN. If DNS records for the FQDN aren't public yet, resolution may only work via manually maintained `/etc/hosts` entries on test clients.
{% endhint %}
{% endstep %}

{% step %}

#### Switch staging to production

Once staging issuance is proven, re-apply the `ClusterIssuer` with `acme.server` pointed at the production endpoint (`https://acme-v02.api.letsencrypt.org/directory`) — always a full `apply`, never a partial patch (see the `solvers` warning above). `cert-manager` does not proactively reissue an already-`Ready` certificate just because the issuer's server field changed, so force reissuance by deleting the secret (the `Certificate` object still exists, so `cert-manager` recreates the secret automatically):

```bash
kubectl apply -f clusterissuer-prod.yaml

for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl delete secret http-wildcard-cert -n $ns
done
```

Confirm real (non-staging) issuance:

```bash
kubectl get secret http-wildcard-cert -n <ns> -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -issuer
# expect: issuer=C = US, O = Let's Encrypt, CN = R... or E... (no "(STAGING)" prefix)
```

{% hint style="info" %}
Let's Encrypt's production environment enforces a rate limit of 5 duplicate certificates per exact domain set per 7 days.
{% endhint %}
{% endstep %}

{% step %}

#### Restart deployments that read the certificate at startup

Pods mounting the secret as a volume pick up the renewed certificate within about 60 seconds via kubelet sync — no restart needed. Deployments that read it once at process startup need a rollout restart. Discover the real mounts per namespace rather than assuming names:

```bash
kubectl get pods -n <ns> -o json \
  | jq -r '.items[].spec.volumes[]? | select(.secret.secretName=="http-wildcard-cert") | .name'
```

Then for each deployment found:

```bash
kubectl rollout restart deployment/<name> -n <ns>
```

{% endstep %}

{% step %}

#### Update airctl-config.yaml

```bash
grep clusterIssuerName /opt/pf9/airctl/conf/airctl-config.yaml || \
  echo "clusterIssuerName: letsencrypt-prod" | sudo tee -a /opt/pf9/airctl/conf/airctl-config.yaml
sudo sed -i 's/clusterIssuerName:.*/clusterIssuerName: letsencrypt-prod/' \
  /opt/pf9/airctl/conf/airctl-config.yaml
```

Without this, the next `airctl upgrade` re-triggers the `kplane` du-upgrade job's default behavior, which can re-add the replicator label and start the overwrite race again.
{% endstep %}
{% endstepper %}

### Reverting to the original wildcard certificate

Do these steps **in order** — each assumes the previous one finished. Substitute your own namespace list and replication source namespace name.

{% stepper %}
{% step %}

#### Delete the per-namespace Certificate objects

This stops `cert-manager` from managing `http-wildcard-cert`. It does **not** change the secret's current content by itself — that's the next step.

```bash
for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl delete certificate letsencrypt-http-certificate -n $ns
done
```

{% endstep %}

{% step %}

#### Restore the original secret content from your backups

`kubectl apply` fails here — the backup YAML has no `kubectl.kubernetes.io/last-applied-configuration` annotation and carries a stale `resourceVersion`, so the API server rejects it with a conflict. Delete the current secret first, then create fresh from the backup file instead of applying over it:

```bash
for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl delete secret http-wildcard-cert -n $ns
  kubectl create -f logs/DU-CERTS-baseline-${ns}.yaml
done
```

Verify the restore actually put the original certificate back — check subject/issuer/SANs against what you expect (self-signed, issuer == subject):

```bash
for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl get secret http-wildcard-cert -n $ns -o jsonpath='{.data.tls\.crt}' \
    | base64 -d | openssl x509 -noout -subject -issuer -ext subjectAltName
done
```

{% endstep %}

{% step %}

#### Re-add the replicator label

Only on the namespaces that are supposed to *receive* the replicated certificate — never on the replication source namespace, since its own secret was already restored directly in the previous step.

```bash
kubectl label ns <namespace-1> cert-manager-tls=http-wildcard-cert
kubectl label ns <namespace-2> cert-manager-tls=http-wildcard-cert
kubectl get ns <namespace-1> <namespace-2> --show-labels | grep cert-manager-tls
```

{% endstep %}

{% step %}

#### Find and restart the deployments that mount the secret

Don't assume names — discover them fresh, and filter out one-shot `Job`s (they already ran and don't need restarting):

```bash
for ns in <namespace-1> <namespace-2> <source-namespace>; do
  kubectl get pods -n $ns -o json | jq -r \
    '.items[] | select(.spec.volumes[]?.secret.secretName=="http-wildcard-cert") | "\(.metadata.name) owner=\(.metadata.ownerReferences[0].kind)/\(.metadata.ownerReferences[0].name)"'
done
```

Only restart pods owned by a `ReplicaSet` (i.e. a `Deployment`), one at a time so a slow rollout doesn't get bundled into an SSH timeout on the others:

```bash
kubectl rollout restart deployment/<name> -n <ns>
kubectl rollout status deployment/<name> -n <ns> --timeout=90s
```

{% endstep %}

{% step %}

#### Remove the airctl config key

Prevents the next `airctl upgrade` from re-triggering DU-specific certificate behavior via the `kplane` chart's `use_du_specific_le_http_cert` hook:

```bash
sudo sed -i '/clusterIssuerName:/d' /opt/pf9/airctl/conf/airctl-config.yaml
grep clusterIssuerName /opt/pf9/airctl/conf/airctl-config.yaml || echo "clusterIssuerName absent - correct"
```

{% endstep %}

{% step %}

#### (Optional) Clean up the ClusterIssuer and Secret

Only if you don't plan to redo the DU-specific certificate switch again soon. Leaving these in place is harmless once the Certificates from the first revert step are deleted.

```bash
kubectl delete clusterissuer letsencrypt-prod
kubectl delete secret route53-credentials-secret -n cert-manager
```

{% endstep %}
{% endstepper %}

### Upgrading to a Release with `provision-certs`

{% hint style="warning" %}
If you followed this page to configure a DU-specific certificate manually, **revert to the self-signed certificate before upgrading** to a release where `airctl provision-certs` is available. Upgrading directly, without reverting first, risks the upgrade silently clobbering your manually-configured certificate back to the shared self-signed wildcard.
{% endhint %}

**Why this matters:** the upgrade-safety logic that ships with `provision-certs` decides whether to skip re-adding the `cert-manager-tls` replication label based on a `certMode` field in `/opt/pf9/airctl/conf/airctl-config.yaml`. That field only ever gets set by running `provision-certs` itself. A DU configured through this manual page has no `certMode` set — as far as the new upgrade logic can tell, it looks identical to a DU that was never customized. On upgrade, it takes the default path: it re-adds the `cert-manager-tls` label to every namespace. Once that label is back, `kubernetes-replicator` resumes watching the secret and will overwrite your manually-issued certificate with the shared self-signed wildcard, usually within seconds of the label going on.

**The safe migration path is:**

{% stepper %}
{% step %}

#### Revert to self-signed before upgrading

Follow [Reverting to the original wildcard certificate](#reverting-to-the-original-wildcard-certificate) above in full, and confirm every namespace is back on the self-signed certificate (issuer == subject) before proceeding. This puts the DU into the state the upgrade path actually expects, so nothing gets caught mid-migration.
{% endstep %}

{% step %}

#### Run the upgrade

Perform `airctl upgrade` as normal. Since the DU is genuinely self-signed at this point, the default upgrade behavior (re-adding the replication label) is correct and does not clobber anything.
{% endstep %}

{% step %}

#### Re-provision your certificate the supported way

Once the upgrade completes and `airctl provision-certs --help` shows the command is available, re-issue your DU-specific certificate through it instead of by hand:

```bash
airctl provision-certs --cluster-issuer <cluster-issuer-name>
```

See [Using Custom Certificates](/private-cloud-director/getting-started/self-hosted/using-custom-certificates.md) for the full walkthrough, including the BYOC wildcard and BYOC per-namespace modes if you'd rather supply your own certificate/key directly instead of using `cert-manager`. This run persists `certMode` to the config file, so every future upgrade is protected automatically — you will not need to repeat this revert-upgrade-reprovision sequence again.
{% endstep %}
{% endstepper %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.platform9.com/private-cloud-director/getting-started/self-hosted/using-du-specific-certificates-manual.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.