# Ipv6 Installation This article describes how IPV6 can be enabled for a PMK cluster. This is available as an advanced option in the cluster creation wizard. PMK has support for creating Kubernetes clusters with either IPV4 or IPV6 enabled. PMK does not currently support *dual stack* installations. ## Installation Environment ### NAT64, DNS64 In order to use the managed service which is IPV4 and Kubernetes installation environment which will be IPV6 or dual stack it is recommended to: * Have a NAT64 setup ready. * Have a DNS64 setup ready. Additionally, you would also need a working DNS server that can resolve the host names. This is required for commands like `kubectl logs` and `kubectl exec` to work. ### HTTP, HTTPS\_PROXY As an alternative a HTTP/HTTPS Proxy can also be used instead of the NAT64. While most of the Platform9 products take HTTP, HTTPS proxy as configuration parameters. There are few components that may not have that capability and rely on ‘system’ settings to setup the proxy setting. * Ensure that you have http proxy server configured which has dual stack (Both Ipv4 and IPv6) configured. * If the proxy server URL is specified as IPv6 address, installer would error-out while parsing the proxy URL. To avoid this please make sure you have an appropriate DNS address for the proxy. Alternatively please update the /etc/hosts file on the host machine and add an entry like: `fc00:1:a:1:f816:3eff:fec5:2ca7 ipv6.proxy.com` * To ensure that all the PF9 services has the proxy setting configured, please add the following entries in the `/etc/profile` file: {% tabs %} {% tab title="Bash" %} ```bash http_proxy=http://ipv6.proxy.com:3128 https_proxy=http://ipv6.proxy.com:3128 ``` {% endtab %} {% endtabs %} * If the pf9-hostagent is unable to download packages and you observe following errors - \\ {% tabs %} {% tab title="Bash" %} ```bash pf9_app_cache.py ERROR - Downloading https://s3-us-west-1.amazonaws.com/package-repo.platform9.com/host-packages/pf9-kube/4.5.0-10274/pf9-kube-4.5.0-10274.x86_64.rpm failed: HTTPSConnectionPool(host='s3-us-west-1.amazonaws.com', port=443): Max retries exceeded with url: /package-repo.platform9.com/host-packages/pf9-kube/4.5.0-10274/pf9-kube-4.5.0-10274.x86_64.rpm (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out',)) ``` {% endtab %} {% endtabs %} Please set the http\_proxy and https\_proxy in the file /opt/pf9/hostagent/pf9-hostagent.env as well followed by running the command - {% tabs %} {% tab title="Bash" %} ```bash systemctl restart pf9-hostagent \ ``` {% endtab %} {% endtabs %} ## Installation ## Cluster Creation PMK cluster can be created using an API and UI. At the moment of writing this document, the UI support for creating an IPV6 cluster is NOT present and will be available in the next patch of the product. API can be used to create the cluster with IPV6 addresses for both services as well as container CIDRs. Instead of using tools like curl, Platform9 API can be invoked using a python client called ‘*qbertclient’* available on pypi. The supported version is python3 for the qbert client. Here are some important parameter restrictions: * *Ipv6:* this is the most important parameter (valid values are 0, 1 or false/true) this triggers the cluster components to use I addressing for various Kubernetes components like CoreDNS, KubeProxy, Canal, API server etc. * *networkplugin: calico:* While platform9 supports many network plugins, for the purposes of IPV6 only calico is supported. * containersCidr & servicesCidr: Please specify the ipv6 cidr when setting the ipv6=1. Please make sure the cidr specified is between \_/112 - /123. \_For example fd00:101::/64 is an invalid value but fd00:101::/112 is acceptable. * \_privileged: \_Please make sure this is set to true (or 1) * There are other parameters that you can control as well, though setting ipv6 will set default values for these parameters these includes:*calicoIPv6,calicoIPv6PoolNatOutgoing, calicoIPv6PoolBlockSize* * Coming soon: * Support for IPv6 on `metallbCidr, masterIp, masterVipIpv6, "calicoIPv4DetectionMethod": "first-found", "calicoIPv6DetectionMethod": "first-found"` The following is an example script on the creation of a PMK cluster that uses IPV6. {% tabs %} {% tab title="Python" %} ```python from qbertclient import qbert from qbertclient import keystone du_fqdn = "du-test-kplane-asriraman-3501.platform9.horse" username = "whoever@test-kplane-asriraman-3501.com" password= "some password, please substitute" project_name = "service" keystone = keystone.Keystone(du_fqdn, username, password, project_name) token = keystone.get_token() project_id = keystone.get_project_id(project_name) qb = qbert.Qbert(token, "https://{}/qbert/v3/{}".format(du_fqdn, project_id)) print(qb.list_clusters()) print(qb.list_cloud_providers()) node_pool_uuid = None for item in qb.list_nodepools(): print("Nodepool item") print(item) if item['name'] == 'defaultPool': node_pool_uuid = item['uuid'] container_cidr = "fd00:101::/112" svc_cidr ="fd00:102::/112" req_body = { "name":"asr-ipv6", "containersCidr": container_cidr, "servicesCidr": svc_cidr, "privileged":True, "appCatalogEnabled":False, "allowWorkloadsOnMaster":True, "nodePoolUuid": node_pool_uuid, "numMasters":1, "isPrivate":False, "numWorkers":1, "networkPlugin": "calico", "mtuSize": "1400", "ipv6": True } out = qb.create_cluster(req_body) print(out) print(qb.list_clusters()) ``` {% endtab %} {% endtabs %} Some other optional parameters to try if you want more control on the way Calico works {% tabs %} {% tab title="JSON" %} ```json "calicoIPv6PoolCidr": container_cidr, "calicoIPv6": "autodetect", "calicoIPv4": "none", "felixIPv6Support": True, "calicoIPv6PoolNatOutgoing": True, "calicoIPv6PoolBlockSize": 116, ``` {% endtab %} {% endtabs %} ## Host Side Installation Once the cluster definition is created using the process mentioned above. Please add nodes to the cluster definition above. This can be done in either one of the two ways. (Remember to go through the proxy configuration if proxy is needed for the outgoing traffic) **Important Prerequisite** Make sure ipv6 forwarding is turned on all the nodes. {% tabs %} {% tab title="Bash" %} ```bash sudo sysctl net.ipv6.conf.all.forwarding=1 ``` {% endtab %} {% endtabs %} ### 1. pf9ctl Download pf9ctl, the link the UI doesn’t work across DNS64 due to a redirection issue with the URL shortener, please use the following URL: {% tabs %} {% tab title="Bash" %} ```bash https://raw.githubusercontent.com/platform9/express-cli/master/cli-setup.sh ``` {% endtab %} {% endtabs %} ## Once downloaded please run it and fill in the values: * Account URL * Username, Password * Region Name * Tenant Name Followed by {% tabs %} {% tab title="Bash" %} ```bash pf9ctl cluster prep-node ``` {% endtab %} {% endtabs %} And attaching your cluster Master: {% tabs %} {% tab title="Bash" %} ```bash pf9ctl cluster attach-node -m localhost ``` {% endtab %} {% endtabs %} Worker: {% tabs %} {% tab title="Bash" %} ```bash pf9ctl cluster attach-node -w localhost ``` {% endtab %} {% endtabs %} ### 2. Using hostagent installer To use the hostagent installer, follow these step Download the installer from Platform9 {% tabs %} {% tab title="Bash" %} ```bash curl -O https:///clarity/platform9-install-[debian|redhat].sh. ``` {% endtab %} {% endtabs %} Replace the right ACCOUNT FQDN as well as the OS platform (use redhat for centos). Invoke the certless installer for *no proxy* setup If you have a proxy setup, invoke the installer as follows Run Now go to the UI and add this node to the pre-created cluster Either of the methods would work and can be used to create the cluster, it will take few minutes before the node(s) are ready. You can check the cluster pods getting ipv6 addresses. {% tabs %} {% tab title="Bash" %} ```bash $ ./kubectl get pods -o wide -n kube-system NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-79cdfb4bb5-mk8dv 1/1 Running 5 6m32s fd00:101::2001 fc00-1-a-1-f816-3eff-fe66-6373 calico-node-bmw45 1/1 Running 0 6m31s fc00:1:a:1:f816:3eff:fe66:6373 fc00-1-a-1-f816-3eff-fe66-6373 calico-typha-7f89fb5f8d-6vlws 0/1 Pending 0 6m32s calico-typha-7f89fb5f8d-b5jw8 1/1 Running 0 6m32s fc00:1:a:1:f816:3eff:fe66:6373 fc00-1-a-1-f816-3eff-fe66-6373 calico-typha-7f89fb5f8d-l8vls 0/1 Pending 0 6m32s coredns-5b985c544f-ktn49 1/1 Running 0 7m52s fd00:101::2005 fc00-1-a-1-f816-3eff-fe66-6373 k8s-master-fc00-1-a-1-f816-3eff-fe66-6373 3/3 Running 0 39m fc00:1:a:1:f816:3eff:fe66:6373 fc00-1-a-1-f816-3eff-fe66-6373 metrics-server-v0.3.6-75cdf48d5b-t58sh 2/2 Running 0 69s fd00:101::2006 fc00-1-a-1-f816-3eff-fe66-6373 ``` {% endtab %} {% endtabs %} ### Persist Hostname in /etc/hosts Once the host installation is complete, please persist the node-name as shown by the `hostname` command to `/etc/hosts`. TBD: * Describe the use of multicast instead of broadcast for API server HA ## Known Issues * Proxy configuration: Proxy configuration needs to be done through a dual process of both setting at the host level as well as passing the configuration to installers. * The DNS entries for all hosts is a must and must be distinct. This is a big departure from the IPV4 where Platform9 would use the IPV4 address as the identifier of the hosts. In IPV6 hostnames are used and the right configuration is critical to the cluster operation. * MetalLB does not work in BGP mode for IPv6. This is currently a limitation with MetalLB, and is under active development. * When using the Luigi network operator, the kernel the hosts run on need to be 4.1+. Please refer to the how-to guide for Luigi for more information.