# Docker Image Management

### Install Base Artifacts

To begin the installation of SMCP, run the downloaded `install.sh` script.

{% hint style="info" %}
**Info**

The execution of the *install.sh* script requires root-level privileges, but users holding sudo-level permissions are also capable of carrying out these commands.
{% endhint %}

{% tabs %}
{% tab title="None" %}

```none
bash ./install.sh v-5.10.0-xxxxxxx
```

{% endtab %}
{% endtabs %}

The files are installed under the */opt/pf9/airctl* directory. See the [auto$](https://github.com/platform9/pcd-docs-gitbook/blob/main/PEC/artifacts/README.md) page for more details on the directory structure.

{% tabs %}
{% tab title="none" %}

```none
echo 'export PATH=$PATH:/opt/pf9/airctl' >> ~/.bashrc
source ~/.bashrc
```

{% endtab %}
{% endtabs %}

This command will append the directory containing *airctl* to your PATH environment variable, allowing you to run the command without specifying the full path each time. The source command ensures that the changes take effect immediately in the current shell session.

{% hint style="info" %}
**Info**

Please note that, in SMCP, we support online and airgapped mode. The following commands: `airctl configure` and `airctl artifacts download` are specific to online mode only.
{% endhint %}

### Configuration

Airctl relies on two configuration files to define the management stack. Both of these configuration files can be generated along with downloading any other artifacts using the command:

{% hint style="info" %}
**Info**

During the airctl configure command, users get the prompt `Download all artifacts(y/N)`. If artifacts are earlier downloaded as part of `./download.sh <secret-key> <release-version>` prerequisite, then one can skip downloading it now(by pressing `N`).
{% endhint %}

{% tabs %}
{% tab title="Bash" %}

```bash
$ /opt/pf9/airctl/airctl configure --config /opt/pf9/airctl/conf/airctl-config.yaml

$ /opt/pf9/airctl/airctl configure --help
generates the configuration for the management plane, including configuration options to create the management cluster and the management plane

Usage:
  airctl configure [flags]

Flags:
  -f, --du-fqdn duFQDN                             specify the management plane (DU) FQDN. Defaults to airctl-1.platform9.localnet
  -e, --external-ip4 ip                            set the external IPv4 for the DU
  -x, --external-ip6 ip                            set the external IPv6 for the DU
  -h, --help                                       help for configure
  -4, --ipv4-enabled                               set this flag to enable ipv4 network. To enable dual stack networks, set this flag and "ipv6-enabled"
  -6, --ipv6-enabled                               set this flag to enable ipv6 network. To enable dual stack networks, set this flag and "ipv4-enabled"
  -i, --master-ips ipSlice                         a set of master node IP(s), separated by commas (default [])
      --master-vip4 ip                             the VIP of type IPv4 for management cluster
      --master-vip6 ip                             the VIP of type IPv6 for management cluster
  -m, --num-masters int                            set the number of master nodes, only inputs supported are 1,3 and 5. Defaults to 1. (default 1)
      --private-reg-base privateRegistryBasePath   set the private registry base url if the registry for container images is set to "custom"
      --registry-type privateRegistryType          type of container image registry to use. Possible values: DU, custom, none. Defaults to DU
  -u, --unattended                                 set the unattended flag to disable prompts for configuration options
  -w, --worker-ips ipSlice                         a set of worker node IP(s), separated by commas (default [])

Global Flags:
      --config string   config file (default is $HOME/airctl-config.yaml)
      --json            json output for commands (configure-hosts only currently)
      --quiet           disable spinners
      --verbose         print verbose logs to the console
```

{% endtab %}
{% endtabs %}

This is an interactive CLI that takes in various user inputs and generates the two necessary configuration files at */opt/pf9/airctl/conf* location:

{% tabs %}
{% tab title="Bash" %}

```bash
/opt/pf9/airctl/conf/airctl-config.yaml
/opt/pf9/airctl/conf/nodelet-bootstrap-config.yaml
```

{% endtab %}
{% endtabs %}

You may edit these options by hand if necessary. Please refer [here](https://github.com/platform9/pcd-docs-gitbook/blob/main/PEC/airctl-reference/README.md) here for more information.

{% hint style="info" %}
**Info**

The files generated by this command contains data as per the input provided by the user. It is advisable to again go through the file and make necessary changes according to your environment.
{% endhint %}

The command also lets you download other required and optional artifacts once the config files are generated. You may also download these artifacts at a later point using the command:

{% tabs %}
{% tab title="Bash" %}

```bash
$ /opt/pf9/airctl/airctl artifacts download --config /opt/pf9/airctl/conf/airctl-config.yaml

$ /opt/pf9/airctl/airctl artifacts download --help
download airctl artifacts

Usage:
  airctl artifacts download <secret-key> [flags]

Examples:
airctl artifacts download <secret-key> [--kube-versions, --all-kube-versions, --download-dir, --kubevirt]

Flags:
      --all-kube-versions      download all kubernetes versions
      --download-dir string    download directory (default "/opt/pf9/airctl/artifacts")
  -h, --help                   help for download
      --kube-versions string   download only specific kube versions and corresponding addons. Use double quotes to pass in the kube versions and list comma separated versions. Example: "1.21, 1.23"
      --kubevirt               download kubevirt addon
```

{% endtab %}
{% endtabs %}

### Creating the Management Cluster

Once your configuration files are generated and the required artifacts downloaded, we are ready to create the management cluster that will host the management plane. Run the following command:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl advanced-ddu create-mgmt --config /opt/pf9/airctl/conf/airctl-config.yaml
```

{% endtab %}
{% endtabs %}

This will create a head-less Platform9 based Kubernetes cluster. It is considered head-less because there is no accompanying management plane to monitor and manage it.

{% hint style="warning" %}
**Warning**

If running in airgapped mode, the cluster creation can take considerable time to upload and import the required container images
{% endhint %}

Once the management cluster is created, please validate that the cluster is healthy by running:

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl get nodes
```

{% endtab %}
{% endtabs %}

You should see all attached nodes here and that they are marked `Ready`. You may find the required kubeconfig at `/etc/nodelet/airctl-mgmt/certs/admin.kubeconfig` on the node that you ran the `airctl create-mgmt` command on.

Make sure that on all nodes of the management cluster, the `/etc/resolv.conf` file must have a `nameserver` entry that is an IP reachable from pods running on the nodes.

In addition, On Ubuntu OS, the `systemd-resolved` service should be disabled and stopped so that it does not overwrite the contents of `/etc/resolv.conf`

{% tabs %}
{% tab title="Bash" %}

```bash
sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved
```

{% endtab %}
{% endtabs %}

### Starting the Management Plane

Update *airctl-config.yaml* with any values you'd like to change. It has reasonable defaults, but we all have our preferences.

With the config file updated, we need to prep the system. This involves installing required dependencies like helm, etc.

{% tabs %}
{% tab title="Bash" %}

```bash
sudo airctl init --config /opt/pf9/airctl/conf/airctl-config.yaml
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Info**

The SMCP management plane deployment contains nginx pod. It refers to a s3 URL that cannot be resolved in an airgapped system where the URL cannot be reached/ DNS cannot reach it. As a workaround, we add a fake DNS entry in the coredns section of the nodelet bootstrap configuration so that coredns can resolve the s3 URL and nginx will not crash.

While attaching nodes to the management plane, the airctl command gives the following error:

error preparing node Error: Unable to install hostagent. Invalid status code when identifiying hostagent type: 502

In the nginx pod, we see this issue:

host not found in upstream "s3-us-west-1.amazonaws.com" in /etc/nginx/conf.d/pf9/s3.conf

nginx crashes if it cannot resolve the S3 URL. To resolve this, make sure that there is a dummy dns entry for S3 like below in the nodelet-config yaml file.
{% endhint %}

{% tabs %}
{% tab title="Bash" %}

```bash
$ grep -i dns -A3 /opt/pf9/airctl/conf/nodelet-bootstrap-config.yaml 
dns: 
  corednsHosts: 
    - 172.29.21.134 smcp510validation.platform9.localnet 
    - 34.35.69.42  s3-us-west-1.amazonaws.com
```

{% endtab %}
{% endtabs %}

Start the management plane using:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl start --config /opt/pf9/airctl/conf/airctl-config.yaml
```

{% endtab %}
{% endtabs %}

Wait for Kube Management Server services to be up

The `airctl status` command reports the state of the Management Server. Wait until the status command reports the task state as `ready` .

{% tabs %}
{% tab title="Bash" %}

```bash
[centos@airctl-host ~]$ airctl --config /opt/pf9/airctl/conf/airctl-config.yaml status
------------- deployment details ---------------
fqdn:                airctl-1.pf9.localnet
cluster:             airctl-1.pf9.localnet
task state:          ready
-------- region service status ----------
task state:        ready
desired services:  42
ready services:    42
```

{% endtab %}
{% endtabs %}

### Management Server Private Registry (Airgapped)

If the workload clusters managed by the airctl Management Server are airgapped, there is a private registry service deployed. This behavior has not changed from the Management Server. To push an image bundle to the Management Server registry:

{% tabs %}
{% tab title="Bash" %}

```bash
sudo airctl advanced-du push-images --img-tar <path to image .tar.gz> --config <path to airctl configuration>
```

{% endtab %}
{% endtabs %}

The various image bundles are located at `/opt/pf9/airctl/images/`.

### Stopping The Management Plane

The management plane can be cleaned up using:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl stop --config /opt/pf9/airctl/conf/airctl-config.yaml
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Bash" %}

```bash
airctl unconfigure-du [--force] --config /opt/pf9/airctl/conf/airctl-config.yaml
```

{% endtab %}
{% endtabs %}

To delete the management cluster, run:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl advanced-ddu delete-mgmt --config /opt/pf9/airctl/conf/airctl-config.yaml
```

{% endtab %}
{% endtabs %}

### Management Plane Health

You may check the health of the management plane at any time using the status command.

{% tabs %}
{% tab title="Bash" %}

```bash
airctl --config /opt/pf9/airctl/conf/airctl-config.yaml status
```

{% endtab %}
{% endtabs %}

This reports the number of services deployed and how many are healthy, as well as the overall status of the Management Server. If you do see that the expected number of services are not healthy, you can look at each individual service via kubectl.

{% tabs %}
{% tab title="Bash" %}

```bash
kubectl get pods -A
```

{% endtab %}
{% endtabs %}

## Host Management

Once the management plane has been deployed, you may now add nodes to it. The following fields in the airctl config file help with that

{% tabs %}
{% tab title="Bash" %}

```bash
# nodeHostnames points to the list of nodes manage. This can be either IP addresses or DNS names.
# This list does not need to include all nodes on the Management Server. It can point to just the current list of nodes that need to be operated on.
nodeHostnames:
  - 10.128.240.189
  - 10.128.240.190
  - 10.128.240.191
  - host1.myorganization.localnet
  - host2.myorganization.localnet

# sshUser reflects the user on the nodes used by airctl to login.
# Airctl expects a key based password login on these nodes.
sshUser: centos

# path to ssh key files
# sshPublicKeyFile: /home/centos/.ssh/id_rsa.pub
# sshPrivateKeyFile: /home/centos/.ssh/id_rsa

# The following fields points to the path of the packages that need to be installed on the nodes.
# We install 2 packages and their dependencies, hostagent and docker.
# By default, we expect to install packages on Centos7.9.
hostAgentRepo: /opt/pf9/airctl/hostagent-v-5.2.0-1549281.tar.gz
dockerRepo: /opt/pf9/airctl/docker-v-5.2.0-1549281.tar.gz
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**Info**

The hostagent package for centos, RHEL, rocky and ubuntu are named as hostagent-.tar.gz, hostagent-8-.tar.gz, hostagent-9-.tar.gz and hostagent-ubuntu-.tar.gz. Similarly for the docker package.

For example, the hostagent and docker packages are debian packages on Ubuntu Linux as shown in the following example.
{% endhint %}

{% tabs %}
{% tab title="Bash" %}

```bash
hostAgentRepo: /opt/pf9/airctl/hostagent-ubuntu-v-5.10.0-3240704.tar.gz
dockerRepo: /opt/pf9/airctl/docker-ubuntu-v-5.10.0-3240704.tar.gz
```

{% endtab %}
{% endtabs %}

Once you have the above fields populated correctly in the airctl config, run the following command to copy the packages to the nodes, as well as authorise those nodes in the airctl management plane.

{% hint style="info" %}
**Info**

Before running the configure-hosts command, ensure that all other external YUM/ APT repositories are disabled for all the OSes.
{% endhint %}

{% tabs %}
{% tab title="Bash" %}

```bash
airctl --config /opt/pf9/airctl/conf/airctl-config.yaml configure-hosts
```

{% endtab %}
{% endtabs %}

The configure-hosts command has various flags that may be set to suit your specific environment. Please look at the help text to determine what’s appropriate for you.

The above command is idempotent and can be run any number of times. Nodes already authorized in the past are untouched, even if they are missing from the config above.

### Host Health

You can look at the status of the hosts with:

{% tabs %}
{% tab title="Bash" %}

```bash
airctl --config /opt/pf9/airctl/conf/airctl-config.yaml host-status
```

{% endtab %}
{% endtabs %}

### Obtain Credentials

Now we acquire the credential of the newly created Management Server.

{% tabs %}
{% tab title="None" %}

```none
airctl get-creds --config /opt/pf9/airctl/conf/airctl-config.yaml
email:     admin@airctl.localnet
password:  somesecretpassword
```

{% endtab %}
{% endtabs %}

**Note**: Please update the duUser and the duPassword entries in the config file with the above values before proceeding. If the default admin password is unchanged, then the duPassword field is optional.

**Note**: If a new password is passed during airctl start command in this way, then this new password needs to be passed to get-creds command to acquire the admin credentials of the Management Server, otherwise there will be an error to get the admin credentials. Do keep in mind to store this new password in a file, as the admin credentials cannot be acquired if this password is forgotten/lost.

{% tabs %}
{% tab title="YAML" %}

```yaml
airctl start --config /opt/pf9/airctl/conf/airctl-config.yaml --password $newPasswd

airctl get-creds --config /opt/pf9/airctl/conf/airctl-config.yaml
fatal error: failed to getting admin credentials: failed to validate airctl password: crypto/bcrypt: hashedPassword is not the hash of the given password

airctl get-creds --config /opt/pf9/airctl/conf/airctl-config.yaml --password $newPasswd
email:     admin@airctl.localnet
password:  somesecretpassword
```

{% endtab %}
{% endtabs %}

### Accessing the UI/CLI

Accessing the UI or CLI is not possible using the IP address. You will either need to update your DNS settings to create an A record or have the FQDN and IP of the physical host's Management Server where the management plane runs. For testing purposes, you can create an */etc/hosts* entry on your local workstation.

{% tabs %}
{% tab title="None" %}

```none
sudo cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
​
127.0.1.1    test-pf9-airgap-c7-du-host-1676888-110.novalocal test-pf9-airgap-c7-du-host-1676888-110
10.128.147.50 airctl-1-1676888-275.pf9.localnet
```

{% endtab %}
{% endtabs %}

### Configure Hosts

This section discusses how to prepare multiple hosts to be added to the cluster when you have direct SSH access. Your Platform9 solutions architect can assist you in working through methods of onboarding hosts. At this point, we are ready to onboard new hosts.

There are multiple methods to onboard hosts, especially if you do not have direct SSH access. The `airctl` command has a call named *configure-hosts* which aids in configuring multiple hosts and prepares them, along with the Platform9 agent that is already running and ready to add to a cluster. The command also helps set up the */etc/hosts* entry to point to the Platform9 Management Server, and can optionally install docker. Additionally, it has the ability to pre-cache docker images as needed. The airctl page can reference more completed details on these tasks. The airctl command uses the *nodeHostnames* option to specify which hosts should be processed.

{% tabs %}
{% tab title="YAML" %}

```yaml
nodeHostnames:
 - 10.128.147.73
 - 10.128.147.185
 - test-pf9-airgap-c7-1676888-610-1.novalocal
```

{% endtab %}
{% endtabs %}

{% tabs %}
{% tab title="Bash" %}

```bash
airctl configure-hosts --help
configure the hosts for agent installation, docker installation and image pre-install
​
Usage:
 airctl configure-hosts [flags]
​
Flags:
 -h, --help                            help for configure-hosts
     --reset-pf9-managed-docker-conf   remove PF9_MANAGED_DOCKER=false from  /etc/pf9/kube_override.env, this would enable PMK to completely manage(install, upgrade) docker as part of the cluster creation
     --skip-docker-img-import          skip import the docker images on each hosts
     --skip-docker-install             skip docker install on each hosts
     --skip-image-renaming             skip image renaming to point to the privateRegistryBase, for e.g. gcr.io/hyperkube will become <privateRegistryBase>/hyperkube
     --skip-yum-repo-install           skip yum repo installation, default to false
​
Global Flags:
     --config string   config file (default is $HOME/airctl-config.yaml)
     --json            json output for commands (configure-hosts only currently)
     --verbose         print verbose logs to the console
```

{% endtab %}
{% endtabs %}

Behind the scene, configure-host is going to accomplish the following:

* Create a Yum or APT Repo (unless --skip-yum-repo-install) is specified to install Platform9 agents. Additionally, this disables the existing yum repos on the node.
* The same flag `--skip-yum-repo-install` works on Ubuntu too
* Install docker (unless --skip-docker-install) is specified
* Push all the images needed by Platform9 (unless --skip-image-import is specified)

As mentioned earlier, the newer version moves to a central registry and needs you to have up-to-date YUM/ APT repositories, in which case some --skip-xxx can be applied, and the process would be much faster.