Bare Metal Support

Private Cloud Director supports creation of Kubernetes clusters using both virtualized as well as physical nodes.

This guide will walk you through the process of creating a Kubernetes cluster using physical (bare metal) machines.

Info

• Bare metal support in Private Cloud Director is currently in beta

• Private Cloud Director UI currently does not support creation of bare metal clusters. You need to use a separate CLI to create bare metal clusters today.

Prerequisites

Before starting, make sure you have:

• One or more Ubuntu physical machines that you want to use as Kubernetes nodes

• Network connectivity between all your machines and the Private Cloud Director management cluster

• Administrator (sudo/root) access on all machines you plan to use

Setup Process

Setting up a bare metal Kubernetes cluster involves four main steps:

1. Creating a cluster definition that describes your desired Kubernetes setup

2. Onboarding your machines using the byohctl CLI tool

3. Verifying that your cluster is working properly

Let's go through each step in detail.

Step 1: Onboard Your Machines

On each physical server that you want to add to your Kubernetes cluster as a node, follow these steps:

Download the byohctl CLI:

Bash

bash <(curl -s https://byohctl.s3.us-west-2.amazonaws.com/byohctl-setup)

This command downloads a setup script and runs it immediately. The script will install the byohctl CLI on your machine.

Onboard the machine to your Private Cloud Director environment:

Bash

sudo ./byohctl onboard \
  -u <URL> \
  -e <USERNAME/EMAIL> \
  -p <PASSWORD> \
  -c <CLIENT_TOKEN> \
  -d <DOMAIN> \
  -t <TENANT>

Replace the placeholders with your actual information:

<URL>: Your Private Cloud Director management plane URL (e.g., example.platform9.com)

<USERNAME/EMAIL>: Your Private Cloud Director username or email address

<PASSWORD>: Your Private Cloud Director password

<CLIENT_TOKEN>: Your Private Cloud Director client token (secret)

<DOMAIN>: Your Private Cloud Director domain (default is "default")

<TENANT>: Your Private Cloud Director tenant (default is "service")

For example:

Bash

byohctl onboard -u sam-dev.app.dev-pcd.platform9.com -e [email protected] -p your_password -c client_secret -d default -t service

Verify that the machine has been successfully onboarded:

Check that the required agent service is running:

Bash

systemctl status pf9-byohost-agent.service

You should see that the service is active and running.

Check the agent logs to confirm successful registration:

Bash

cat /var/log/pf9/byoh/byoh-agent.log

Look for messages indicating successful registration with the Private Cloud Director management plane.

From your management cluster, verify that the host appears as a resource:

Bash

kubectl get byohost -A

Your machine should appear in the list of available hosts.

Step 2: Create Your Kubernetes Cluster

Now you'll create a cluster definition that tells Private Cloud Director how to build your Kubernetes cluster using the physical servers you've onboarded.

Create a file named cluster.yaml with your cluster definition. Here's an explanation of the key components:

• Cluster: Defines the basic properties of your Kubernetes cluster

• ByoCluster: Configures the infrastructure-specific settings

• HostedControlPlane: Sets up the control plane components

• ByoMachineTemplate: Defines the template for worker nodes

• KubeadmConfigTemplate: Configures how kubeadm will bootstrap nodes

• MachineDeployment: Specifies how many worker nodes to create

• K8sInstallerConfigTemplate: Determines how Kubernetes will be installed

The sample cluster definition in the guide includes all these components with appropriate settings for a Private Cloud Director environment. You'll need to modify the following items to match your specific setup:

• Cluster name (replace byo-22-13 with your preferred name)

• Namespace (replace sam-dev-default-service with your namespace)

• Control plane endpoint hostname (update to match your environment)

• Kubernetes version (adjust if needed)

• Number of replicas (set based on how many worker nodes you want)

YAML

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cas-capi-version: v1.30
    core-addons: enabled
    metrics-server: 0.6.4
    proxy-addons: enabled
  name: byo-22-13
  namespace: sam-dev-default-service
spec:
  clusterNetwork:
    apiServerPort: 443
    pods:
      cidrBlocks:
      - 10.244.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.platform9.io/v1alpha1
    kind: HostedControlPlane
    name: byo-22-13cp
    namespace: sam-dev-default-service
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: ByoCluster
    name: byo-22-13
    namespace: sam-dev-default-service
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: ByoCluster
metadata:
  name: byo-22-13
  namespace: sam-dev-default-service
spec:
  bundleLookupBaseRegistry: quay.io/platform9
  controlPlaneEndpoint:
    host: byo-22-13cp.sam-dev-default-service.app.dev-pcd.platform9.com
    port: 443
---
apiVersion: controlplane.platform9.io/v1alpha1
kind: HostedControlPlane
metadata:
  name: byo-22-13cp
  namespace: sam-dev-default-service
spec:
  addons:
    coreDNS: {}
    konnectivity:
      agent:
        extraArgs:
        - --proxy-server-host=konnectivity-byo-22-13cp.sam-dev-default-service.app.dev-pcd.platform9.com
        - --proxy-server-port=443
        image: registry.k8s.io/kas-network-proxy/proxy-agent
        tolerations:
        - key: CriticalAddonsOnly
          operator: Exists
        version: v0.0.32
      server:
        image: registry.k8s.io/kas-network-proxy/proxy-server
        port: 8132
        version: v0.0.32
  apiServer:
    extraArgs:
    - --cloud-provider=external
    - --cors-allowed-origins=http://localhost:3000,http://localhost:3000/,https://sam-dev-default-service.app.dev-pcd.platform9.com,https://sam-dev-default-service.app.dev-pcd.platform9.com/,https://sam-dev-default-service-scadrial.app.dev-pcd.platform9.com,https://sam-dev-default-service-scadrial.app.dev-pcd.platform9.com/
    - --advertise-address=10.96.0.40
    - --authentication-config=/etc/kubernetes/oidc-auth/oidc.yaml
    extraVolumeMounts:
    - mountPath: /etc/ssl/host-certs
      name: etc-ssl-certs-from-host
      readOnly: true
    - mountPath: /etc/kubernetes/oidc-auth
      name: oidc-auth
      readOnly: true
    resources: {}
  controllerManager:
    extraArgs:
    - --cloud-provider=external
    resources: {}
  extraCertSANs:
  - konnectivity-byo-22-13cp.sam-dev-default-service.app.dev-pcd.platform9.com
  - byo-22-13cp.sam-dev-default-service.app.dev-pcd.platform9.com
  extraVolumes:
  - configMap:
      name: oidc-auth
    name: oidc-auth
  hcpClass: default
  hostname: byo-22-13cp.sam-dev-default-service.app.dev-pcd.platform9.com
  kubelet:
    cgroupfs: systemd
    preferredAddressTypes:
    - InternalIP
    - ExternalIP
    - Hostname
  scheduler:
    resources: {}
  version: 1.31.2
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: ByoMachineTemplate
metadata:
  name: byo-22-13
  namespace: sam-dev-default-service
spec:
  template:
    spec:
      installerRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: K8sInstallerConfigTemplate
        name: byo-22-13
        namespace: sam-dev-default-service
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: byo-22-13
  namespace: sam-dev-default-service
spec:
  template:
    spec: {}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: byo-22-13
  namespace: sam-dev-default-service
spec:
  clusterName: byo-22-13
  replicas: 1
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: byo-22-13
      clusterName: byo-22-13
      failureDomain: nova
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: ByoMachineTemplate
        name: byo-22-13
      version: v1.32.2
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: K8sInstallerConfigTemplate
metadata:
  name: byo-22-13
  namespace: sam-dev-default-service
spec:
  template:
    spec:
      bundleRepo: quay.io/platform9
      bundleType: k8s

Apply the cluster configuration:

Bash

kubectl apply -f cluster.yaml

This command sends your cluster definition to the Private Cloud Director management plane, which then begins creating your cluster.

Step 3: Monitor Cluster Creation

You can track the progress of your cluster creation with the following commands:

Check the TenantControlPlane (TCP) status:

Bash

kubectl get tcp -A

Check the HostedControlPlane (HCP) status:

Bash

kubectl get hcp -A

Monitor the Machine provisioning:

YAML

kubectl get machine -A

Once your machines show a status of "Running" with NODENAME and PROVIDERID values set, your cluster nodes have successfully joined the cluster.

Step 4: Access Your Cluster

Get the kubeconfig file for your new cluster:

Bash

kubectl get secret -n <NAMESPACE> <CLUSTER_NAME>cp-admin-kubeconfig -o json | \
  jq '.data["super-admin.conf"]' | xargs | base64 -d > cluster-kubeconfig.yaml

export KUBECONFIG=cluster-kubeconfig.yaml

Replace <NAMESPACE> with your namespace and <CLUSTER_NAME> with your cluster name.

Alternatively, you can download the kubeconfig file from the Private Cloud Director web interface.

Verify that your cluster is working properly:

Bash

kubectl get nodes -A
kubectl get ns
kubectl get pods -A

These commands show the nodes in your cluster, the namespaces, and all running pods.

Important Consideration

• The Kubernetes version specified in the HostedControlPlane definition must be supported by your management plane.

• The version specified in MachineDeployment should match the version available in the Private Cloud Director BYOH bundle repository.

• Make sure to use the correct bundle repository in your cluster definition.

• The Private Cloud Director service agent logs are stored at /var/log/pf9/byoh/byoh-agent.log and are useful for troubleshooting.