Docker Image Management

Install Base Artifacts

To begin the installation of SMCP, run the downloaded install.sh script.

Info

The execution of the install.sh script requires root-level privileges, but users holding sudo-level permissions are also capable of carrying out these commands.

None

bash ./install.sh v-5.10.0-xxxxxxx

The files are installed under the /opt/pf9/airctl directory. See the auto$ page for more details on the directory structure.

none

echo 'export PATH=$PATH:/opt/pf9/airctl' >> ~/.bashrc
source ~/.bashrc

This command will append the directory containing airctl to your PATH environment variable, allowing you to run the command without specifying the full path each time. The source command ensures that the changes take effect immediately in the current shell session.

Info

Please note that, in SMCP, we support online and airgapped mode. The following commands: airctl configure and airctl artifacts download are specific to online mode only.

Configuration

Airctl relies on two configuration files to define the management stack. Both of these configuration files can be generated along with downloading any other artifacts using the command:

Info

During the airctl configure command, users get the prompt Download all artifacts(y/N). If artifacts are earlier downloaded as part of ./download.sh <secret-key> <release-version> prerequisite, then one can skip downloading it now(by pressing N).

Bash

$ /opt/pf9/airctl/airctl configure --config /opt/pf9/airctl/conf/airctl-config.yaml

$ /opt/pf9/airctl/airctl configure --help
generates the configuration for the management plane, including configuration options to create the management cluster and the management plane

Usage:
  airctl configure [flags]

Flags:
  -f, --du-fqdn duFQDN                             specify the management plane (DU) FQDN. Defaults to airctl-1.platform9.localnet
  -e, --external-ip4 ip                            set the external IPv4 for the DU
  -x, --external-ip6 ip                            set the external IPv6 for the DU
  -h, --help                                       help for configure
  -4, --ipv4-enabled                               set this flag to enable ipv4 network. To enable dual stack networks, set this flag and "ipv6-enabled"
  -6, --ipv6-enabled                               set this flag to enable ipv6 network. To enable dual stack networks, set this flag and "ipv4-enabled"
  -i, --master-ips ipSlice                         a set of master node IP(s), separated by commas (default [])
      --master-vip4 ip                             the VIP of type IPv4 for management cluster
      --master-vip6 ip                             the VIP of type IPv6 for management cluster
  -m, --num-masters int                            set the number of master nodes, only inputs supported are 1,3 and 5. Defaults to 1. (default 1)
      --private-reg-base privateRegistryBasePath   set the private registry base url if the registry for container images is set to "custom"
      --registry-type privateRegistryType          type of container image registry to use. Possible values: DU, custom, none. Defaults to DU
  -u, --unattended                                 set the unattended flag to disable prompts for configuration options
  -w, --worker-ips ipSlice                         a set of worker node IP(s), separated by commas (default [])

Global Flags:
      --config string   config file (default is $HOME/airctl-config.yaml)
      --json            json output for commands (configure-hosts only currently)
      --quiet           disable spinners
      --verbose         print verbose logs to the console

This is an interactive CLI that takes in various user inputs and generates the two necessary configuration files at /opt/pf9/airctl/conf location:

Bash

/opt/pf9/airctl/conf/airctl-config.yaml
/opt/pf9/airctl/conf/nodelet-bootstrap-config.yaml

You may edit these options by hand if necessary. Please refer here here for more information.

Info

The files generated by this command contains data as per the input provided by the user. It is advisable to again go through the file and make necessary changes according to your environment.

The command also lets you download other required and optional artifacts once the config files are generated. You may also download these artifacts at a later point using the command:

Bash

$ /opt/pf9/airctl/airctl artifacts download --config /opt/pf9/airctl/conf/airctl-config.yaml

$ /opt/pf9/airctl/airctl artifacts download --help
download airctl artifacts

Usage:
  airctl artifacts download <secret-key> [flags]

Examples:
airctl artifacts download <secret-key> [--kube-versions, --all-kube-versions, --download-dir, --kubevirt]

Flags:
      --all-kube-versions      download all kubernetes versions
      --download-dir string    download directory (default "/opt/pf9/airctl/artifacts")
  -h, --help                   help for download
      --kube-versions string   download only specific kube versions and corresponding addons. Use double quotes to pass in the kube versions and list comma separated versions. Example: "1.21, 1.23"
      --kubevirt               download kubevirt addon

Creating the Management Cluster

Once your configuration files are generated and the required artifacts downloaded, we are ready to create the management cluster that will host the management plane. Run the following command:

Bash

airctl advanced-ddu create-mgmt --config /opt/pf9/airctl/conf/airctl-config.yaml

This will create a head-less Platform9 based Kubernetes cluster. It is considered head-less because there is no accompanying management plane to monitor and manage it.

Warning

If running in airgapped mode, the cluster creation can take considerable time to upload and import the required container images

Once the management cluster is created, please validate that the cluster is healthy by running:

Bash

kubectl get nodes

You should see all attached nodes here and that they are marked Ready. You may find the required kubeconfig at /etc/nodelet/airctl-mgmt/certs/admin.kubeconfig on the node that you ran the airctl create-mgmt command on.

Make sure that on all nodes of the management cluster, the /etc/resolv.conf file must have a nameserver entry that is an IP reachable from pods running on the nodes.

In addition, On Ubuntu OS, the systemd-resolved service should be disabled and stopped so that it does not overwrite the contents of /etc/resolv.conf

Bash

sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved

Starting the Management Plane

Update airctl-config.yaml with any values you'd like to change. It has reasonable defaults, but we all have our preferences.

With the config file updated, we need to prep the system. This involves installing required dependencies like helm, etc.

Bash

sudo airctl init --config /opt/pf9/airctl/conf/airctl-config.yaml

Info

The SMCP management plane deployment contains nginx pod. It refers to a s3 URL that cannot be resolved in an airgapped system where the URL cannot be reached/ DNS cannot reach it. As a workaround, we add a fake DNS entry in the coredns section of the nodelet bootstrap configuration so that coredns can resolve the s3 URL and nginx will not crash.

While attaching nodes to the management plane, the airctl command gives the following error:

error preparing node Error: Unable to install hostagent. Invalid status code when identifiying hostagent type: 502

In the nginx pod, we see this issue:

host not found in upstream "s3-us-west-1.amazonaws.com" in /etc/nginx/conf.d/pf9/s3.conf

nginx crashes if it cannot resolve the S3 URL. To resolve this, make sure that there is a dummy dns entry for S3 like below in the nodelet-config yaml file.

Bash

$ grep -i dns -A3 /opt/pf9/airctl/conf/nodelet-bootstrap-config.yaml 
dns: 
  corednsHosts: 
    - 172.29.21.134 smcp510validation.platform9.localnet 
    - 34.35.69.42  s3-us-west-1.amazonaws.com

Start the management plane using:

Bash

airctl start --config /opt/pf9/airctl/conf/airctl-config.yaml

Wait for Kube Management Server services to be up

The airctl status command reports the state of the Management Server. Wait until the status command reports the task state as ready .

Bash

[centos@airctl-host ~]$ airctl --config /opt/pf9/airctl/conf/airctl-config.yaml status
------------- deployment details ---------------
fqdn:                airctl-1.pf9.localnet
cluster:             airctl-1.pf9.localnet
task state:          ready
-------- region service status ----------
task state:        ready
desired services:  42
ready services:    42

Management Server Private Registry (Airgapped)

If the workload clusters managed by the airctl Management Server are airgapped, there is a private registry service deployed. This behavior has not changed from the Management Server. To push an image bundle to the Management Server registry:

Bash

sudo airctl advanced-du push-images --img-tar <path to image .tar.gz> --config <path to airctl configuration>

The various image bundles are located at /opt/pf9/airctl/images/.

Stopping The Management Plane

The management plane can be cleaned up using:

Bash

airctl stop --config /opt/pf9/airctl/conf/airctl-config.yaml

Bash

airctl unconfigure-du [--force] --config /opt/pf9/airctl/conf/airctl-config.yaml

To delete the management cluster, run:

Bash

airctl advanced-ddu delete-mgmt --config /opt/pf9/airctl/conf/airctl-config.yaml

Management Plane Health

You may check the health of the management plane at any time using the status command.

Bash

airctl --config /opt/pf9/airctl/conf/airctl-config.yaml status

This reports the number of services deployed and how many are healthy, as well as the overall status of the Management Server. If you do see that the expected number of services are not healthy, you can look at each individual service via kubectl.

Bash

kubectl get pods -A

Host Management

Once the management plane has been deployed, you may now add nodes to it. The following fields in the airctl config file help with that

Bash

# nodeHostnames points to the list of nodes manage. This can be either IP addresses or DNS names.
# This list does not need to include all nodes on the Management Server. It can point to just the current list of nodes that need to be operated on.
nodeHostnames:
  - 10.128.240.189
  - 10.128.240.190
  - 10.128.240.191
  - host1.myorganization.localnet
  - host2.myorganization.localnet

# sshUser reflects the user on the nodes used by airctl to login.
# Airctl expects a key based password login on these nodes.
sshUser: centos

# path to ssh key files
# sshPublicKeyFile: /home/centos/.ssh/id_rsa.pub
# sshPrivateKeyFile: /home/centos/.ssh/id_rsa

# The following fields points to the path of the packages that need to be installed on the nodes.
# We install 2 packages and their dependencies, hostagent and docker.
# By default, we expect to install packages on Centos7.9.
hostAgentRepo: /opt/pf9/airctl/hostagent-v-5.2.0-1549281.tar.gz
dockerRepo: /opt/pf9/airctl/docker-v-5.2.0-1549281.tar.gz

Info

The hostagent package for centos, RHEL, rocky and ubuntu are named as hostagent-.tar.gz, hostagent-8-.tar.gz, hostagent-9-.tar.gz and hostagent-ubuntu-.tar.gz. Similarly for the docker package.

For example, the hostagent and docker packages are debian packages on Ubuntu Linux as shown in the following example.

Bash

hostAgentRepo: /opt/pf9/airctl/hostagent-ubuntu-v-5.10.0-3240704.tar.gz
dockerRepo: /opt/pf9/airctl/docker-ubuntu-v-5.10.0-3240704.tar.gz

Once you have the above fields populated correctly in the airctl config, run the following command to copy the packages to the nodes, as well as authorise those nodes in the airctl management plane.

Info

Before running the configure-hosts command, ensure that all other external YUM/ APT repositories are disabled for all the OSes.

Bash

airctl --config /opt/pf9/airctl/conf/airctl-config.yaml configure-hosts

The configure-hosts command has various flags that may be set to suit your specific environment. Please look at the help text to determine what’s appropriate for you.

The above command is idempotent and can be run any number of times. Nodes already authorized in the past are untouched, even if they are missing from the config above.

Host Health

You can look at the status of the hosts with:

Bash

airctl --config /opt/pf9/airctl/conf/airctl-config.yaml host-status

Obtain Credentials

Now we acquire the credential of the newly created Management Server.

None

airctl get-creds --config /opt/pf9/airctl/conf/airctl-config.yaml
email:     [email protected]
password:  somesecretpassword

Note: Please update the duUser and the duPassword entries in the config file with the above values before proceeding. If the default admin password is unchanged, then the duPassword field is optional.

Note: If a new password is passed during airctl start command in this way, then this new password needs to be passed to get-creds command to acquire the admin credentials of the Management Server, otherwise there will be an error to get the admin credentials. Do keep in mind to store this new password in a file, as the admin credentials cannot be acquired if this password is forgotten/lost.

YAML

airctl start --config /opt/pf9/airctl/conf/airctl-config.yaml --password $newPasswd

airctl get-creds --config /opt/pf9/airctl/conf/airctl-config.yaml
fatal error: failed to getting admin credentials: failed to validate airctl password: crypto/bcrypt: hashedPassword is not the hash of the given password

airctl get-creds --config /opt/pf9/airctl/conf/airctl-config.yaml --password $newPasswd
email:     [email protected]
password:  somesecretpassword

Accessing the UI/CLI

Accessing the UI or CLI is not possible using the IP address. You will either need to update your DNS settings to create an A record or have the FQDN and IP of the physical host's Management Server where the management plane runs. For testing purposes, you can create an /etc/hosts entry on your local workstation.

None

sudo cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
​
127.0.1.1    test-pf9-airgap-c7-du-host-1676888-110.novalocal test-pf9-airgap-c7-du-host-1676888-110
10.128.147.50 airctl-1-1676888-275.pf9.localnet

Configure Hosts

This section discusses how to prepare multiple hosts to be added to the cluster when you have direct SSH access. Your Platform9 solutions architect can assist you in working through methods of onboarding hosts. At this point, we are ready to onboard new hosts.

There are multiple methods to onboard hosts, especially if you do not have direct SSH access. The airctl command has a call named configure-hosts which aids in configuring multiple hosts and prepares them, along with the Platform9 agent that is already running and ready to add to a cluster. The command also helps set up the /etc/hosts entry to point to the Platform9 Management Server, and can optionally install docker. Additionally, it has the ability to pre-cache docker images as needed. The airctl page can reference more completed details on these tasks. The airctl command uses the nodeHostnames option to specify which hosts should be processed.

YAML

nodeHostnames:
 - 10.128.147.73
 - 10.128.147.185
 - test-pf9-airgap-c7-1676888-610-1.novalocal

Bash

airctl configure-hosts --help
configure the hosts for agent installation, docker installation and image pre-install
​
Usage:
 airctl configure-hosts [flags]
​
Flags:
 -h, --help                            help for configure-hosts
     --reset-pf9-managed-docker-conf   remove PF9_MANAGED_DOCKER=false from  /etc/pf9/kube_override.env, this would enable PMK to completely manage(install, upgrade) docker as part of the cluster creation
     --skip-docker-img-import          skip import the docker images on each hosts
     --skip-docker-install             skip docker install on each hosts
     --skip-image-renaming             skip image renaming to point to the privateRegistryBase, for e.g. gcr.io/hyperkube will become <privateRegistryBase>/hyperkube
     --skip-yum-repo-install           skip yum repo installation, default to false
​
Global Flags:
     --config string   config file (default is $HOME/airctl-config.yaml)
     --json            json output for commands (configure-hosts only currently)
     --verbose         print verbose logs to the console

Behind the scene, configure-host is going to accomplish the following:

• Create a Yum or APT Repo (unless --skip-yum-repo-install) is specified to install Platform9 agents. Additionally, this disables the existing yum repos on the node.

• The same flag --skip-yum-repo-install works on Ubuntu too

• Install docker (unless --skip-docker-install) is specified

• Push all the images needed by Platform9 (unless --skip-image-import is specified)

As mentioned earlier, the newer version moves to a central registry and needs you to have up-to-date YUM/ APT repositories, in which case some --skip-xxx can be applied, and the process would be much faster.