Integrating External DNS with Kubernetes

Kubernetes contains an internal DNS module that automatically discovers and assigns DNS names to individual containers when instructed. In practice, this works very well and there is room for customization.

However, when the time comes, we frequently need to expose some or all parts of the Kubernetes cluster to the public. For instance, if a cluster exists inside a public Cloud Provider like AWS or Google Cloud Platform, we would like to have a container service that interacts with this cloud provider and change any A Records to point to the nodes that expose those services.

This is what the ExternalDNS project does. ExternalDNS is a Kubernetes project with the main purpose of automatically creating DNS records for Ingress or Service resources.

In this tutorial, we are following a step-by-step approach to configure ExternalDNS integration within the Freedom plan of Platform9’s Managed Kubernetes backed up by a DigitalOcean Droplet.

Let’s get started.

Setting up the Platform9 Free Tier Cluster

Below are the brief instructions to get you up and running with a working Kubernetes Cluster from Platform9:

  1. Signup with Platform9.
  2. Click the Create Cluster button and inspect the instructions. We need a server to host the Cluster.
  3. Create a few Droplets with at least 8GB RAM and 2 vCPUs. Follow instructions to install the pf9cli tool and prepping the nodes.

    $ bash <(curl -sL http://pf9.io/get_cli)
    

    Once the Pf9CLI is installed run Prep-node to configure the node and connect it to Platform9.

    $ pf9ctl cluster prep-node -i
    
  4. Switch to the Platform9 UI and click the refresh button. You should see the new nodes in the list. Assign the first node as a master and the other ones as workers.
  5. Leave the default values in the next steps. Then create the cluster.
  6. Wait until the cluster becomes healthy. It will take at least 20 minutes to finish.
  7. Click on the API Access tab and select to download the kubeconfig button:

    API Access

  8. Once the KubeConfig has downloaded export the config so that your local environment has access to the cluster. Use kubectl cluster-info to test cluster connectivity:
$ export KUBECONFIG=/Users/itspare/Theo/Projects/platform9/example.yaml
$ kubectl cluster-info
Kubernetes master is running at https://134.122.106.235
CoreDNS is running at https://134.122.106.235/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://134.122.106.235/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

Now you are ready to deploy the ExternalDNS service.

Domain Name and API Keys

To test the ExternalDNS, we need to assign a Domain name. I happen to have one spare for this tutorial.

As Digital Ocean does not act as a DNS registrar, you need to assign the nameservers of the domain registrar to point to the following entries:

ns1.digitalocean.com
ns2.digitalocean.com
Ns3.digitalocean.com

After that, go to the Networking tab of your DigitalOcean Dashboard and add the domain name there:

Networking

Do not assign any droplet there, as we will let the ExternalDNS handle that.

Next, we need to create a Personal Access Token for the DigitalOcean API. Navigate to the API->Tokens and Keys and create a new API key. Note that token value, as it will be exposed only temporarily in the UI.

Setting Up ExternalDNS

Setting up the ExternalDNS is the easy part. We just need to define the manifest that consists of the following services:

  1. A Service Account for the ExternalDNS deployment.

    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: external-dns</code>
    
  2. A Cluster Role with required RBAC permissions.

    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
    name: external-dns
    rules:
    - apiGroups: [""]
    resources: ["services","endpoints","pods"]
    verbs: ["get","watch","list"]
    - apiGroups: ["extensions"]
    resources: ["ingresses"]
    verbs: ["get","watch","list"]
    - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["list"]</code>
    
  3. A Cluster Role binding assigned to the previous Cluster Role and Service Account.

    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
    name: external-dns-viewer
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: external-dns
    subjects:
    - kind: ServiceAccount
    name: external-dns
    namespace: default
    
  4. The ExternalDNS pod deployment, passing as arguments the domain name and DigitalOcean Token value.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: external-dns
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: external-dns
    strategy:
    type: Recreate
    selector:
    matchLabels:
    app: external-dns
    template:
    metadata:
    labels:
    app: external-dns
    spec:
    serviceAccountName: external-dns
    containers:
    - name: external-dns
    image: registry.opensource.zalan.do/teapot/external-dns:latest
    args:
    - --source=service
    - --domain-filter=expressiveartsfair.com
    - --provider=digitalocean
    env:
    - name: DO_TOKEN
    value: "YOUR_DIGITALOCEAN_API_KEY"
    

You can either put the manifest all in the same file or in separate files. Then apply the configuration:

$ kubectl apply -f external-dns.yml
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created
deployment.apps/external-dns created

Monitor the deployment status via the Platform9 UI or in the terminal:

Deployment Status

When everything is healthy, we can deploy our first service to test the external DNS configuration.

Create a new file named nginx.demo.service.yml with the following contents:

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx
name: nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
annotations:
external-dns.alpha.kubernetes.io/hostname: demo.expressiveartsfair.com
spec:
selector:
app: nginx
type: NodePort
ports:
- protocol: TCP
port: 80
nodePort: 30080</code>

The important parts are:

  • We used the external-dns.alpha.kubernetes.io/hostname: demo.expressiveartsfair.com annotation passing the custom domain we want to bind
  • We used a NodePort type to expose the node IP address to the public.

Now anytime we assign the external DNS annotation in a service, the daemon will monitor that event and use the Digital Ocean API to update the DNS Records. In the following image you can see that the DNS entry was added to point to the Droplet that we exposed to the NodePort service:

DNS

Now, you may notice one small thing. Currently we cannot use an external Load Balancer service, as the Platform9 Kubernetes distro is running inside a droplet. Thus, we cannot directly navigate to demo.expressiveartsfair.com as the NodePort is open in a different port (port 30080 as we specified in the configuration).

Nonetheless, if we navigate to demo.expressiveartsfair.com:30080, we can see the nginx welcome page as usual:

NGINX

In order to make this work we have a few options:

  • Create the redirect rule.
  • We can use an Ingress controller like nginx-ingress-controller to create on-the-fly nginx configurations based on some host rules. This way we can point directly to the backend service host as defined in the definition.

In either case, we have the ExternalDNS service handling and updating all the corresponding DNS records without user intervention.

Cleaning up

Follow the reverse step and destroy all created resources:

$ kubectl delete -f Nginx-service.yml
$ kubectl delete -f external-dns.yml

Then invalidate the DigitalOcean Token you created earlier.