Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Set Up PXE Boot for VMs

Run a PXE server on a tenant network in Private Cloud Director and use VM rescue mode with an iPXE image to PXE boot other VMs from it.

Overview

Private Cloud Director supports PXE booting VMs by running a PXE server on a VM that you operate. The PXE server VM provides DHCP, TFTP, and (optionally) HTTP on a tenant network where the PCD managed DHCP service is disabled, and serves boot files to other VMs on the same network. To PXE boot a target VM, you launch it on that same network and put it into rescue mode with an iPXE image. iPXE then chains to the PXE server and the target VM boots the installer or operating system delivered by the PXE server.

You can also use a provider network with a specific VLAN instead of a tenant network.

In this guide, you will create a network with DHCP disabled, launch a PXE server VM on it, and PXE boot a target VM from that server using rescue mode.

Prerequisites

Before you start, make sure you have the following:

  • A PXE server image or a running VM that you have already configured with:

    • A DHCP server (for example, isc-dhcp-server) set to serve IPs on the network's subnet range and to advertise itself as the next-server.

    • A TFTP server (for example, tftpd-hpa) hosting the bootloader (pxelinux.0) and the OS kernel and initrd.

    • An HTTP server (for example, Apache) hosting the OS installer files, if you are using HTTP-based installation.

  • An iPXE ISO. You can download it from https://boot.ipxe.org/ipxe.iso.

  • Admin access to the PCD UI, or pcdctl installed and configured. See PCD CLI - pcdctl.

  • The pcdctl examples in this guide all include --insecure, because the default self-signed certificate otherwise causes an SSL verification failure on every API call (for example, pcdctl network list fails outright without it). Omit the flag if your environment uses a CA-signed certificate.

Step 1: Create the Tenant Network with DHCP Disabled

The PCD managed DHCP service and the PXE server's DHCP cannot both serve the same network. Disable DHCP on the subnet so that PXE clients receive their IPs from the PXE server VM.

Using the UI

  1. In the PCD UI, choose Networks and Security, then Networks.

  2. Choose Create Network and enter a name, for example pxe-network.

  3. In the subnet section, enter a CIDR for the subnet, for example 192.168.48.0/24.

  4. Select the Disable Gateway checkbox (it sits right above Enable DHCP). An isolated PXE network has no router, so the subnet's default gateway address does not exist; a guest that installs a route to it loses outbound connectivity.

  5. Clear the Enable DHCP checkbox.

  6. Choose Create to create the network.

Using pcdctl

If your environment uses a provider network with a specific VLAN, you can use that instead. Disable DHCP on the subnet the same way.

Even with DHCP disabled, the Networking Service still allocates each port a fixed IP from the subnet's full allocation pool (192.168.48.2 to 192.168.48.254 by default for this example CIDR), independent of the PXE server's own DHCP range. If the two ranges overlap, the Networking Service can hand a port an address that the PXE server also tries to lease. Narrow the Networking Service allocation pool with --allocation-pool start=<start-ip>,end=<end-ip> on pcdctl subnet create, and keep the PXE server's DHCP range outside that pool.

Step 2: Upload the PXE Server Image

Upload the operating system image that you have prepared as the PXE server. The image must already include the DHCP, TFTP, and (if used) HTTP services and their configuration.

Using the UI

  1. Choose Virtual Machines, then Images.

  2. Choose Add Image and select the image file.

  3. Set the disk format to match the file you are uploading (for example, qcow2 for a customized cloud image).

  4. Choose Create to upload the image.

Using pcdctl

Step 3: Launch the PXE Server VM

Launch a VM from the PXE server image on the tenant network you created in Step 1.

Using the UI

  1. Choose Virtual Machines, then Virtual Machines.

  2. Choose Create VM.

  3. Select the pxe-server image and a flavor with enough resources to run DHCP, TFTP, and HTTP.

  4. Attach the VM to pxe-network.

  5. Launch the VM.

  6. On the Virtual Machines page, locate pxe-server-vm, open its actions menu (the three-dot icon on the right), choose Network Actions, then choose Edit Port Security.

  7. In the Edit Port Security dialog, select the VM's port from the VM Port dropdown, clear the Enable Port Security Groups checkbox, and choose Update Port Security. This single control both disables port security and removes any security groups from the port. With port security enabled, PCD drops DHCP server responses coming from a VM port.

You can also disable port security from the Networks and Security, then Ports page: open the port and clear Apply Port Security Groups in the Security Groups section of the port form. It is the same single checkbox as Enable Port Security Groups in the Edit Port Security dialog.

Using pcdctl

If the VM was created through the UI (attached with --network instead of a pre-created port), disable port security on its existing port:

After the VM boots, log in and configure the VM's network interface with the same static IP as the port's fixed IP (192.168.48.10 in this example), for example using Netplan. The IP configured inside the guest must match the port's fixed IP; a mismatched address is dropped by anti-spoofing rules. Other VMs reach this IP as the next-server advertised in DHCP responses.

Because this network has DHCP disabled, the PXE server VM has no address on it until you configure one, and the PCD metadata service is not reachable on the network either. You cannot log in to the VM over the network to perform that initial configuration: cloud-init cannot inject an SSH key without metadata access (guests report a Datasource DataSourceNone message and "No active metadata service found"), and boot is also delayed by roughly two to three minutes while systemd-networkd-wait-online times out on the address-less interface. Use one of the following instead:

  • Log in through the VM console and configure the static IP manually.

  • Bake the static IP and credentials into the PXE server image ahead of time, as described in Step 2.

  • Temporarily attach a second NIC on a DHCP-enabled network so you can reach the VM over SSH. When a second DHCP-enabled NIC is attached, cloud-init picks up that port's fixed IP from metadata and configures it automatically, so you may not need to run the manual Netplan step at all; only the PXE interface's static IP needs to be configured by hand.

Disabling port security removes security-group protection from the PXE server VM. Port security must also be disabled on each target VM's port (see Step 5); otherwise the DHCP handshake between the target VM and the PXE server never completes. Because both VMs lose security-group protection, keep the whole PXE network isolated and do not attach any VM on it to networks with untrusted workloads.

Step 4: Upload the iPXE Image and Enable It for Rescue Operations

The PCD rescue dialog only accepts images that have the hw_rescue_device and hw_rescue_bus properties set. The image upload form has an option that sets these properties for you.

Using the UI

  1. Choose Virtual Machines, then Images.

  2. Choose Add Image and select the ipxe.iso file you downloaded.

  3. Set Disk Format to iso.

  4. Enable the Use for rescuing VMs option. This sets hw_rescue_device=disk and hw_rescue_bus=scsi on the image automatically.

  5. Choose Create to upload the image.

Using pcdctl

Step 5: PXE Boot a Target VM with Rescue Mode

Launch the target VM on pxe-network. The target VM can use any base image; rescue mode overrides its boot source with the iPXE image for one boot cycle.

Launch the Target VM

Create the VM on pxe-network the same way as in Step 3, using your chosen base image and flavor, then disable port security on its port using the same Edit Port Security dialog (or the CLI) described in Step 3.

Port security must be off on the target VM's port too, not just the PXE server's. With it enabled on the target VM's port, the PXE server logs repeated DHCPDISCOVER / DHCPOFFER exchanges but never receives a DHCPREQUEST or sends a DHCPACK, so no TFTP transfer ever starts. Pinning the PXE server's DHCP reservation to the port's Networking Service-allocated fixed IP does not work around this; the fix is to disable port security on the port itself.

Put the VM into Rescue Mode

  1. On the Virtual Machines page, locate the target VM.

  2. In the target VM's row, choose the actions menu (the three-dot icon on the right), choose Power Actions, then choose Rescue.

  3. In the Rescue VM dialog:

    • For an image-backed VM, select Rescue with Different Image to reveal the image picker.

    • For a volume-backed VM, the image picker is shown by default.

  4. In Select a rescue image, select the ipxe image. The Properties column shows only one property by default; choose show more to confirm both hw_rescue_device and hw_rescue_bus are set.

  5. Choose Rescue VM. Note the one-time password the dialog displays for logging in to the rescued VM; it is not shown again.

The VM reboots into iPXE, which broadcasts a DHCP request on pxe-network. The PXE server VM responds with an IP and the location of the boot files, and the target VM boots the installer or operating system that the PXE server serves.

Using pcdctl

Step 6: Exit Rescue Mode

After the install or boot workflow on the target VM completes, exit rescue mode so the VM boots from its primary disk.

Using the UI

  1. On the Virtual Machines page, locate the target VM.

  2. In the target VM's row, choose the actions menu (the three-dot icon on the right), choose Power Actions, then choose Unrescue.

  3. In the Unrescue VM dialog, confirm that you want to exit rescue mode and return the VM to normal operation, booting from its original disk image.

  4. Choose Unrescue VM.

Using pcdctl

Get Installer Files onto the PXE Server Without External Connectivity

Because the PXE network's subnet has no gateway (Step 1), the PXE server VM has no route to the internet by design. Deliver the OS installer ISO to it through a volume instead of downloading it on the VM:

  1. Upload the OS ISO as an image:

  2. In the PCD UI, create a volume from the ubuntu-iso image and attach it to the PXE server VM.

  3. On the PXE server VM, identify the attached volume's device node:

  4. Mount the attached volume and copy the kernel and initrd into the TFTP directory. Replace /dev/vdb with the device node reported by lsblk, and replace /var/lib/tftpboot with whatever TFTP root your server is actually configured to use (check TFTP_DIRECTORY in /etc/default/tftpd-hpa; on Ubuntu 24.04, tftpd-hpa defaults to /srv/tftp, not /var/lib/tftpboot):

Troubleshooting

Symptom
Likely cause

The Rescue VM dialog shows the error `hw_rescue_device` property missing in the selected image.

The iPXE image was not uploaded with the rescue option enabled. Re-upload with the rescue option enabled, or set the hw_rescue_device=disk property on the existing image.

The Rescue VM dialog shows the error `hw_rescue_bus` property missing in the selected image.

Same as above for the hw_rescue_bus=scsi property.

The target VM does not receive an IP after entering rescue mode.

Verify that Enable DHCP is clear on the subnet, that the PXE server VM is running with a static IP that matches its port's fixed IP, that the DHCP service on the PXE server VM is active, and that port security is disabled on both the PXE server VM's port and the target VM's port. With port security enabled on either port, the DHCP handshake never completes (the PXE server logs repeated DHCPDISCOVER / DHCPOFFER pairs with no DHCPREQUEST / DHCPACK).

The target VM receives an IP but does not start the boot workflow.

Verify that the next-server value in the PXE server's DHCP configuration matches the PXE server VM's static IP, and that the TFTP service on the PXE server VM is reachable on the subnet.

A rescued VM does not respond to pcdctl server reboot.

The VM is still in rescue state; rebooting a rescued VM returns an HTTP 409 error. Run pcdctl server unrescue, then pcdctl server rescue --image ipxe again to retry the PXE boot.

The PXE server VM now serves boot files to any VM launched on pxe-network, and you can repeat Step 5 for each target VM you want to PXE boot.

Related Pages

  • Virtual Network: network, subnet, and port management in the PCD UI and CLI.

  • Layer 2 Networking: running an external DHCP or IPAM service on a segment, including MAC-to-IP reservations.

  • Virtual Machine Actions: rescue and unrescue reference, including the hw_rescue_device and hw_rescue_bus image properties.

PreviousLayer 2 Networking
NextRouter

Last updated

Was this helpful?