BGP Support

Introduction

BGP (Border Gateway Protocol) is the internet's core routing protocol that exchanges routing and reachability information between different networks on the internet, making it essential for large-scale networks. BGP support in Private Cloud Director provides dynamic, scalable network routing for your resources, allowing self-service networks to advertise their private IP prefixes to the physical network, bypassing static routes and enabling pure Layer 3 data centers with high availability

Use this guide to stand up the components, advertise routes, and validate connectivity for BGP network with Private Cloud Director.

Pre-requisites

Following are the pre-requisites that you must satisfy before you can configure the Networking Service to support BGP.

  • All host must run on Ubuntu 22.04 or 24.04

  • Install a generic kernel (if not already):

sudo apt update sudo apt install -y linux-generic

  • The virtual routing and forwarding package (VRF) is required to enable BGP. Verify that VRF support enabled in current kernel:

grep CONFIG_NET_VRF /boot/config-$(uname -r) 
# Expect one of: # CONFIG_NET_VRF=m (module) # CONFIG_NET_VRF=y (built-in)

  • Check VRF module is loaded (if built as a module):

lsmod | grep -w vrf || echo "vrf module not loaded"
sudo modprobe vrf
lsmod | grep -w vrf

  • Persist module across reboots (if needed):

echo vrf | sudo tee /etc/modules-load.d/vrf.conf

  • Optional: Create a dummy external interface for testing:

2) Install FRR and ovn-bgp-agent

  • Install from Ubuntu repositories (simple path):

Host Onboarding

Now create the required host network configuration as part of your cluster blueprint. Then onboard your hosts and authorize with hypervisor role .

Information to Gather

Before starting, collect the following information:

Parameter

Description

Example

AS Number

BGP Autonomous System number

64999

Router Interface IP

IP of interface connected to router

192.168.10.30

Router Interface Name

Name of interface connected to router

ens192

Router BGP Peer IP

IP of upstream router

192.168.10.152

Router AS Number

AS number of upstream router

64999 (same for iBGP)

Configuration Steps

Step 1: Create Address Scope

Address scopes control which networks are eligible for BGP advertisement. Only networks within this scope will be advertised.

1.1 Create IPv4 Address Scope

pcdctl address scope create \ --share \ --ip-version 4 \ bgp-scope-v4

Expected Output:

1.2 Save the Address Scope UUID

# Save the ID for later use ADDRESS_SCOPE_ID="f71c958f-dbe8-49a2-8fb9-19c5f52a37f1" echo "Address Scope ID: $ADDRESS_SCOPE_ID"

⚠️ Important: Save this UUID - you'll need it for BGP agent configuration.

Step 2: Configure FRR (Free Range Routing)

FRR acts as the BGP speaker that advertises routes to the upstream router.

2.1 Verify FRR Installation

# Check if FRR is installed which vtysh # Should output: /usr/bin/vtysh # Check FRR service status systemctl status frr

2.2 Configure FRR

Edit /etc/frr/frr.conf:

vi /etc/frr/frr.conf

Add the following configuration (replace placeholders with your values):

2.3 Example Configuration

Here's a complete example with actual values:

2.4 Restart FRR

systemctl restart frr # Wait for startup sleep 5 # Verify FRR is running systemctl status frr

2.5 Verify BGP Peering

# Check BGP summary sudo vtysh -c 'show bgp summary'

Expected Output (after router is configured):

Status Indicators:

  • State/PfxRcd shows a number: Peering established

  • State shows "Idle": Peering not established

  • State shows "Active": Trying to connect

  • State shows "Connect": TCP connection issues

Step 3: Configure OVN BGP Agent

The OVN BGP Agent monitors OVN and creates kernel routes for FRR to advertise.

3.1 Verify OVN BGP Agent Installation

# Check if agent is installed which ovn-bgp-agent # Should output: /usr/bin/ovn-bgp-agent # Check if service file exists ls -l /usr/lib/systemd/system/ovn-bgp-agent.service

3.2 Configure Systemd Service

Edit /usr/lib/systemd/system/ovn-bgp-agent.service:

vi /usr/lib/systemd/system/ovn-bgp-agent.service

Service Configuration:

[Unit] Description=OpenStack Neutron OVN BGP Agent After=network.target frr.service ovn-controller.service Wants=frr.service Documentation=man:ovn-bgp-agent(1) [Service] User=root Type=simple CacheDirectory=neutron ExecStart=/usr/bin/ovn-bgp-agent --config-file /etc/ovn-bgp-agent/bgp-agent.conf --log-file=/var/log/ovn/ovn_bgp_agent.log Restart=on-failure LimitNOFILE=65535 TimeoutStopSec=15 [Install] WantedBy=multi-user.target

Key Points:

  • After=frr.service: Ensures FRR starts first

  • User=root: Required for kernel route manipulation

  • Restart=on-failure: Auto-restart on crashes

3.3 Configure OVN BGP Agent

Edit /etc/ovn-bgp-agent/bgp-agent.conf:

vi /etc/ovn-bgp-agent/bgp-agent.conf

Configuration Template:

# OVN BGP Agent Configuration # Replace the following: # <AS_NUMBER>: Your BGP AS number (e.g., 64999) # <ROUTER_INTERFACE_IP>: IP of interface connected to router (e.g., 192.168.10.30) # <ADDRESS_SCOPE_UUID>: UUID from Step 1 (e.g., f71c958f-dbe8-49a2-8fb9-19c5f52a37f1) [DEFAULT] # Enable debug logging debug = True # Reconcile interval (seconds) - how often to sync state reconcile_interval = 120 # Expose tenant networks (required for tenant network advertisement) expose_tenant_networks = True # Driver type driver = ovn_bgp_driver # BGP Configuration bgp_AS = <AS_NUMBER> bgp_router_id = <ROUTER_INTERFACE_IP> # OVS Database connection ovsdb_connection = tcp:127.0.0.1:6640 # BGP NIC and VRF configuration bgp_nic = bgp-nic bgp_vrf = bgp-vrf bgp_vrf_table_id = 10 # Address scopes (comma-separated list of UUIDs) address_scopes = <ADDRESS_SCOPE_UUID> [ovn] # OVN Southbound DB connection ovn_sb_connection = tcp:127.0.0.1:6642 # OVN Northbound DB connection ovn_nb_connection = tcp:127.0.0.1:6641 [bgp] # BGP speaker implementation bgp_speaker = frr

3.5 Example Configuration

[DEFAULT] debug = True reconcile_interval = 120 expose_tenant_networks = True driver = ovn_bgp_driver bgp_AS = 64999 bgp_router_id = 192.168.10.30 ovsdb_connection = tcp:127.0.0.1:6640 bgp_nic = bgp-nic bgp_vrf = bgp-vrf bgp_vrf_table_id = 10 address_scopes = f71c958f-dbe8-49a2-8fb9-19c5f52a37f1 [ovn] ovn_sb_connection = tcp:127.0.0.1:6642 ovn_nb_connection = tcp:127.0.0.1:6641 [bgp] bgp_speaker = frr

3.6 Enable and Start OVN BGP Agent

# Reload systemd systemctl daemon-reload # Enable service (start on boot) systemctl enable ovn-bgp-agent # Start service systemctl start ovn-bgp-agent # Wait for startup sleep 5 # Check status systemctl status ovn-bgp-agent

Step 4: Enable Proxy ARP

Proxy ARP allows the OpenStack host to respond to ARP requests for VM IPs, enabling traffic forwarding.

4.1 Identify Provider Network Bridge

# List OVS bridges ovs-vsctl list-br # Look for bridges starting with "br-phy" # Example: br-phy5, br-phy-provider, etc.

4.2 Enable Proxy ARP on Provider Bridge

Replace br-phy5 with your actual bridge name:

# Enable proxy ARP sudo sysctl -w net.ipv4.conf.br-phy5.proxy_arp=1 # Disable reverse path filtering (required for asymmetric routing) sudo sysctl -w net.ipv4.conf.br-phy5.rp_filter=0 # Enable IP forwarding sudo sysctl -w net.ipv4.ip_forward=1

4.3 Make Changes Persistent

Add to /etc/sysctl.conf:

cat >> /etc/sysctl.conf << 'EOF' # BGP OVN Configuration net.ipv4.conf.br-phy5.proxy_arp = 1 net.ipv4.conf.br-phy5.rp_filter = 0 net.ipv4.ip_forward = 1 EOF # Apply changes sysctl -p

4.4 Verify Settings

# Check proxy ARP sysctl net.ipv4.conf.br-phy5.proxy_arp # Should output: net.ipv4.conf.br-phy5.proxy_arp = 1 # Check rp_filter sysctl net.ipv4.conf.br-phy5.rp_filter # Should output: net.ipv4.conf.br-phy5.rp_filter = 0 # Check IP forwarding sysctl net.ipv4.ip_forward # Should output: net.ipv4.ip_forward = 1

Step 5: Create Subnet Pools

Subnet pools with address scopes ensure only authorized subnets are advertised via BGP.

5.1 Create subnet Pools

You can create multiple pools for different purposes:

# Provider network pool openstack subnet pool create bgp-pool-provider \ --address-scope bgp-scope-v4 \ --prefix 172.200.0.0/16 \ --default-prefix-length 24 # tenant network pool openstack subnet pool create bgp-pool-tenant \ --address-scope bgp-scope-v4 \ --prefix 100.10.0.0/16 \ --default-prefix-length 24

5.3 Verify Subnet Pools

openstack subnet pool list # Check details openstack subnet pool show bgp-pool-v4

Step 6: Create Networks

Create tenant networks using the subnet pools to enable BGP advertisement.

6.1 Create Tenant Network

openstack network create tenant-bgp-net

6.2 Create Subnet from Pool

openstack subnet create tenant-bgp-subnet \ --network tenant-bgp-net \ --subnet-pool bgp-pool-tenant \ --subnet-range 10.100.10.0/24

Expected Output:

+----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | allocation_pools | 10.100.10.2-10.100.10.254 | | cidr | 10.100.10.0/24 | | created_at | 2025-11-28T14:00:00Z | | description | | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 10.100.10.1 | | host_routes | | | id | 9b4d8f0a-2345-6789-01bc-def123456789 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | tenant-bgp-subnet | | network_id | 7c5e9d1b-3456-7890-12cd-ef1234567890 | | project_id | 86acdbd1d72745fd8e8320edd7543400 | | subnetpool_id | 8a3c7e9f-1234-5678-90ab-cdef12345678 | | updated_at | 2025-11-28T14:00:00Z | +----------------------+--------------------------------------+

6.3 Create Provider (External / Public) Network

openstack network create provider-bgp-net --provider-network-type vlan \ --provider-physical-network phy-net \ --provider-segment 100 \ --external

6.4 Create Subnet from Pool

openstack subnet create provider-bgp-subnet \ --network provider-bgp-net \ --subnet-pool bgp-pool-provider \ --subnet-range 172.200.10.0/24

6.5 Create Router with External Gateway

# Create router openstack router create tenant-bgp-router # Set external gateway (provider network) openstack router set \ --external-gateway provider-bgp-net \ tenant-bgp-router # Add interface to tenant network openstack router add subnet \ tenant-bgp-router \ tenant-bgp-subnet

6.6 Create VM on Tenant Network

openstack server create \ --flavor m1.small \ --image ubuntu-20.04 \ --network tenant-bgp-net \ --key-name mykey \ test-vm-bgp

6.7 (Optional) Assign Floating IP

# Create floating IP openstack floating ip create provider-network # Assign to VM openstack server add floating ip test-vm-bgp <FLOATING_IP>

Verification

Step 1: Verify BGP Peering Status

sudo vtysh -c "show bgp summary"

Expected Output:

IPv4 Unicast Summary (VRF default): BGP router identifier 192.168.10.30, local AS number 64999 vrf-id 0 BGP table version 13 RIB entries 19, using 3648 bytes of memory Peers 1, using 724 KiB of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc 192.168.10.152 4 64999 1339 1345 0 0 0 22:15:38 1 9 N/A Total number of neighbors 1

Check:

  • ✅ State/PfxRcd: Should show number of received prefixes

  • ✅ Up/Down: Should show uptime (e.g., 22:15:38)

  • ✅ PfxSnt: Should show number of sent prefixes (>0 if advertising)

Step 2: Verify Advertised Routes

sudo vtysh -c "show bgp ipv4 unicast"

Expected Output like:

BGP table version is 13, local router ID is 192.168.10.30, vrf id 0 Default local pref 100, local AS 64999 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *> 10.100.10.5/32 0.0.0.0@10< 0 32768 ? *> 10.100.10.10/32 0.0.0.0@10< 0 32768 ? *> 172.200.0.50/32 0.0.0.0@10< 0 32768 ? *> 192.168.10.0/24 0.0.0.0 100 0 32768 ? Displayed 4 routes and 4 total paths

Check:

  • ✅ VM IPs (/32): Should see individual VM IPs advertised

  • ✅ **Next Hop 0.0.0.0

    10**: Indicates routes from bgp-vrf (table 10)

  • ✅ Path "32768 ?": Indicates locally originated routes**

Step 3: Verify Kernel Routes

# Check main routing table ip route show # Check bgp-vrf routing table (table 10) ip route show table 10

Expected in table 10:

10.100.10.5 dev bgp-nic scope link 10.100.10.10 dev bgp-nic scope link 172.200.0.50 dev bgp-nic scope link

Troubleshooting

Issue 1: BGP Peering Not Established

Symptoms:

Neighbor State 192.168.10.152 Idle

Diagnosis:

# Check BGP logs journalctl -u frr -n 100 | grep -i bgp # Check connectivity to router ping 192.168.10.152 # Check if BGP port is open telnet 192.168.10.152 179 # Check FRR BGP status sudo vtysh -c "show bgp neighbors 192.168.10.152"

Solutions:

  1. Check L2 connectivity

    # Verify interface is up ip link show ens192 # Check ARP table ip neigh show

  2. Check firewall rules

    # Allow BGP port 179 iptables -I INPUT -p tcp --dport 179 -j ACCEPT iptables -I OUTPUT -p tcp --sport 179 -j ACCEPT

  3. Verify AS numbers match

    • For iBGP: Both sides must have same AS number

    • For eBGP: AS numbers must be different

  4. Check router configuration

    • Ensure router has matching BGP configuration

    • Verify router is listening on BGP port

Issue 2: Routes Not Advertised

Symptoms:

  • BGP peering established

  • No routes shown in show bgp ipv4 unicast

Diagnosis:

# Check if routes exist in kernel ip route show table 10 # Check OVN BGP agent logs journalctl -u ovn-bgp-agent -n 100 # Check if VMs have IPs openstack server list --long

Solutions:

  1. Verify address scope configuration

    # Check subnet pool has correct address scope openstack subnet pool show bgp-pool-v4 -c address_scope_id # Check subnet is from correct pool openstack subnet show tenant-bgp-subnet -c subnetpool_id

  2. Restart OVN BGP agent

    systemctl restart ovn-bgp-agent # Wait and check logs journalctl -u ovn-bgp-agent -f

  3. Check OVN SB database connectivity

    # Test connection ovn-sbctl --db=tcp:127.0.0.1:6642 show

  4. Verify bgp-nic and bgp-vrf exist

    ip link show bgp-nic ip link show bgp-vrf # If missing, restart agent systemctl restart ovn-bgp-agent

Issue 3: VMs Not Reachable from External Network

Symptoms:

  • Routes advertised correctly

  • Cannot ping VM IPs from external network

Diagnosis:

# Check proxy ARP settings sysctl net.ipv4.conf.br-phy5.proxy_arp sysctl net.ipv4.conf.br-phy5.rp_filter # Check IP forwarding sysctl net.ipv4.ip_forward # Check OVS flows ovs-ofctl dump-flows br-int | grep <VM_IP>

Solutions:

  1. Enable proxy ARP (if not enabled)

    sudo sysctl -w net.ipv4.conf.br-phy5.proxy_arp=1 sudo sysctl -w net.ipv4.conf.br-phy5.rp_filter=0 sudo sysctl -w net.ipv4.ip_forward=1

  2. Check security groups

    # Allow ICMP openstack security group rule create \ --protocol icmp \ --ingress \ default

  3. Verify OVN flows

    # Check logical flows ovn-sbctl lflow-list | grep <VM_IP>

  4. Check router gateway

    # Ensure router has external gateway openstack router show tenant-bgp-router -c external_gateway_info

Issue 4: FRR Not Starting

Symptoms:

systemctl status frr Active: failed

Diagnosis:

# Check FRR logs journalctl -u frr -n 100 # Check configuration syntax sudo vtysh -c "show running-config" # Check for port conflicts netstat -tlnp | grep 179

Solutions:

  1. Fix configuration syntax errors

    # Validate config sudo vtysh -f /etc/frr/frr.conf # Check for errors in output

  2. Enable BGP daemon

    # Edit /etc/frr/daemons vi /etc/frr/daemons # Set bgpd=yes bgpd=yes # Restart systemctl restart frr

  3. Check file permissions

    ls -l /etc/frr/frr.conf # Should be: -rw-r----- 1 frr frr chown frr:frr /etc/frr/frr.conf chmod 640 /etc/frr/frr.conf

PreviousDPU / SmartNIC Integration & Hardware OffloadingNextNetworking Quality Of Service QoS Configuration

Last updated

Was this helpful?