# BGP Support ### Introduction BGP ([Border Gateway Protocol](https://www.google.com/search?q=Border+Gateway+Protocol\&sca_esv=e2fb2748a0591637\&rlz=1C5OZZY_enUS1161US1161\&ei=O007afqiIPmfkPIPmu-soAU\&ved=2ahUKEwigxc6A0raRAxVvl-4BHW9VCAoQgK4QegQIARAC\&uact=5\&oq=what+is+bgp+and+bgp+support+in+openstack\&gs_lp=Egxnd3Mtd2l6LXNlcnAiKHdoYXQgaXMgYmdwIGFuZCBiZ3Agc3VwcG9ydCBpbiBvcGVuc3RhY2syBRAhGKABMgUQIRigATIFECEYoAFIlx5QkAFYvRxwAXgBkAEAmAGWAaAB1hKqAQQyNy4yuAEDyAEA-AEBmAIeoALjE8ICChAAGLADGNYEGEfCAg0QABiABBiwAxhDGIoFwgIFEAAYgATCAgoQABiABBhDGIoFwgILEAAYgAQYkQIYigXCAgYQABgWGB7CAgsQABiABBiGAxiKBcICBRAhGKsCmAMAiAYBkAYKkgcEMjYuNKAHt40BsgcEMjUuNLgH3hPCBwcwLjE3LjEzyAdTgAgA\&sclient=gws-wiz-serp\&mstk=AUtExfCeya9ttb1DKpd3wP-MGkt9zuFfNrJ9bD8wQhCOLncdnbLc-ZjCQbcjniyA8BKJF4MDOd-ZaMHaPYG4R73Z_GOalwCmnzB2dh6YyEmKcAE_MdTxQ6IbNZ4b0Mph77wgzgsNdLb8r1wJ-6C3fCRl4IUS64rzeE7SlfSJ63cGvki2ybOn5Pje4gdYy5gsn9nf_Ega\&csui=3)) is the internet's core routing protocol that exchanges routing and reachability information between different networks on the internet, making it essential for large-scale networks. BGP support in space.vars.product\_name provides dynamic, scalable network routing for your resources, allowing self-service networks to advertise their private IP prefixes to the physical network, bypassing static routes and enabling pure Layer 3 data centers with high availability Use this guide to stand up the components, advertise routes, and validate connectivity for BGP network with space.vars.product\_name. ### Pre-requisites Following are the pre-requisites that you must satisfy before you can configure the Networking Service to support BGP. * All host must run on Ubuntu 22.04 or 24.04 * **Install a generic kernel (if not already):** `sudo apt update sudo apt install -y linux-generic` * The virtual routing and forwarding package ([VRF](https://docs.kernel.org/networking/vrf.html)) is required to enable BGP. Verify that VRF support enabled in current kernel: ```bash grep CONFIG_NET_VRF /boot/config-$(uname -r) # Expect one of: # CONFIG_NET_VRF=m (module) # CONFIG_NET_VRF=y (built-in) ``` * Check VRF module is loaded (if built as a module): ```bash lsmod | grep -w vrf || echo "vrf module not loaded" sudo modprobe vrf lsmod | grep -w vrf ``` * Persist module across reboots (if needed): `echo vrf | sudo tee /etc/modules-load.d/vrf.conf` * Optional: Create a dummy external interface for testing: ```bash sudo modprobe dummy echo dummy | sudo tee /etc/modules-load.d/dummy.conf sudo ip link add ext-nic type dummy sudo ip link set ext-nic up sudo ip a a 1.1.1.1/32 dev ext-nic ip -d link show ext-nic ``` #### 2) Install FRR and ovn-bgp-agent * **Install from Ubuntu repositories (simple path):** ```bash sudo apt update sudo apt install -y frr frr-pythontools ovn-bgp-agent sed -i 's/zebra=no/zebra=yes/' /etc/frr/daemons sed -i 's/bgpd=no/bgpd=yes/' /etc/frr/daemons ``` ### Host Onboarding Now create the required host network configuration as part of your cluster blueprint. Then onboard your hosts and authorize with hypervisor role . ### **Information to Gather** Before starting, collect the following information: | Parameter | Description | Example | | ------------------------- | ------------------------------------- | ----------------------- | | **AS Number** | BGP Autonomous System number | `64999` | | **Router Interface IP** | IP of interface connected to router | `192.168.10.30` | | **Router Interface Name** | Name of interface connected to router | `ens192` | | **Router BGP Peer IP** | IP of upstream router | `192.168.10.152` | | **Router AS Number** | AS number of upstream router | `64999` (same for iBGP) | ### **Configuration Steps** #### **Step 1: Create Address Scope** Address scopes control which networks are eligible for BGP advertisement. Only networks within this scope will be advertised. **1.1 Create IPv4 Address Scope** `pcdctl address scope create \ --share \ --ip-version 4 \ bgp-scope-v4` **Expected Output:** ```bash +------------+--------------------------------------+ | Field | Value | +------------+--------------------------------------+ | id | f71c958f-dbe8-49a2-8fb9-19c5f52a37f1 | | ip_version | 4 | | name | bgp-scope-v4 | | project_id | 86acdbd1d72745fd8e8320edd7543400 | | shared | True | +------------+--------------------------------------+ ``` **1.2 Save the Address Scope UUID** `# Save the ID for later use ADDRESS_SCOPE_ID="f71c958f-dbe8-49a2-8fb9-19c5f52a37f1" echo "Address Scope ID: $ADDRESS_SCOPE_ID"` ⚠️ **Important:** Save this UUID - you'll need it for BGP agent configuration. #### **Step 2: Configure FRR (Free Range Routing)** FRR acts as the BGP speaker that advertises routes to the upstream router. **2.1 Verify FRR Installation** `# Check if FRR is installed which vtysh # Should output: /usr/bin/vtysh # Check FRR service status systemctl status frr` **2.2 Configure FRR** Edit `/etc/frr/frr.conf`: `vi /etc/frr/frr.conf` **Add the following configuration** (replace placeholders with your values): **2.3 Example Configuration** Here's a complete example with actual values: **2.4 Restart FRR** `systemctl restart frr # Wait for startup sleep 5 # Verify FRR is running systemctl status frr` **2.5 Verify BGP Peering** `# Check BGP summary sudo vtysh -c 'show bgp summary'` **Expected Output (after router is configured):** **Status Indicators:** * **State/PfxRcd shows a number**: Peering established * **State shows "Idle"**: Peering not established * **State shows "Active"**: Trying to connect * **State shows "Connect"**: TCP connection issues #### **Step 3: Configure OVN BGP Agent** The OVN BGP Agent monitors OVN and creates kernel routes for FRR to advertise. **3.1 Verify OVN BGP Agent Installation** `# Check if agent is installed which ovn-bgp-agent # Should output: /usr/bin/ovn-bgp-agent # Check if service file exists ls -l /usr/lib/systemd/system/ovn-bgp-agent.service` **3.2 Configure Systemd Service** Edit `/usr/lib/systemd/system/ovn-bgp-agent.service`: `vi /usr/lib/systemd/system/ovn-bgp-agent.service` **Service Configuration:** `[Unit] Description=OpenStack Neutron OVN BGP Agent After=network.target frr.service ovn-controller.service Wants=frr.service Documentation=man:ovn-bgp-agent(1) [Service] User=root Type=simple CacheDirectory=neutron ExecStart=/usr/bin/ovn-bgp-agent --config-file /etc/ovn-bgp-agent/bgp-agent.conf --log-file=/var/log/ovn/ovn_bgp_agent.log Restart=on-failure LimitNOFILE=65535 TimeoutStopSec=15 [Install] WantedBy=multi-user.target` **Key Points:** * `After=frr.service`: Ensures FRR starts first * `User=root`: Required for kernel route manipulation * `Restart=on-failure`: Auto-restart on crashes **3.3 Configure OVN BGP Agent** Edit `/etc/ovn-bgp-agent/bgp-agent.conf`: `vi /etc/ovn-bgp-agent/bgp-agent.conf` **Configuration Template:** `# OVN BGP Agent Configuration # Replace the following: # : Your BGP AS number (e.g., 64999) # : IP of interface connected to router (e.g., 192.168.10.30) # : UUID from Step 1 (e.g., f71c958f-dbe8-49a2-8fb9-19c5f52a37f1) [DEFAULT] # Enable debug logging debug = True # Reconcile interval (seconds) - how often to sync state reconcile_interval = 120 # Expose tenant networks (required for tenant network advertisement) expose_tenant_networks = True # Driver type driver = ovn_bgp_driver # BGP Configuration bgp_AS = bgp_router_id = # OVS Database connection ovsdb_connection = tcp:127.0.0.1:6640 # BGP NIC and VRF configuration bgp_nic = bgp-nic bgp_vrf = bgp-vrf bgp_vrf_table_id = 10 # Address scopes (comma-separated list of UUIDs) address_scopes = [ovn] # OVN Southbound DB connection ovn_sb_connection = tcp:127.0.0.1:6642 # OVN Northbound DB connection ovn_nb_connection = tcp:127.0.0.1:6641 [bgp] # BGP speaker implementation bgp_speaker = frr` **3.5 Example Configuration** `[DEFAULT] debug = True reconcile_interval = 120 expose_tenant_networks = True driver = ovn_bgp_driver bgp_AS = 64999 bgp_router_id = 192.168.10.30 ovsdb_connection = tcp:127.0.0.1:6640 bgp_nic = bgp-nic bgp_vrf = bgp-vrf bgp_vrf_table_id = 10 address_scopes = f71c958f-dbe8-49a2-8fb9-19c5f52a37f1 [ovn] ovn_sb_connection = tcp:127.0.0.1:6642 ovn_nb_connection = tcp:127.0.0.1:6641 [bgp] bgp_speaker = frr` **3.6 Enable and Start OVN BGP Agent** `# Reload systemd systemctl daemon-reload # Enable service (start on boot) systemctl enable ovn-bgp-agent # Start service systemctl start ovn-bgp-agent # Wait for startup sleep 5 # Check status systemctl status ovn-bgp-agent` #### **Step 4: Enable Proxy ARP** Proxy ARP allows the OpenStack host to respond to ARP requests for VM IPs, enabling traffic forwarding. **4.1 Identify Provider Network Bridge** `# List OVS bridges ovs-vsctl list-br # Look for bridges starting with "br-phy" # Example: br-phy5, br-phy-provider, etc.` **4.2 Enable Proxy ARP on Provider Bridge** Replace `br-phy5` with your actual bridge name: `# Enable proxy ARP sudo sysctl -w net.ipv4.conf.br-phy5.proxy_arp=1 # Disable reverse path filtering (required for asymmetric routing) sudo sysctl -w net.ipv4.conf.br-phy5.rp_filter=0 # Enable IP forwarding sudo sysctl -w net.ipv4.ip_forward=1` **4.3 Make Changes Persistent** Add to `/etc/sysctl.conf`: `cat >> /etc/sysctl.conf << 'EOF' # BGP OVN Configuration net.ipv4.conf.br-phy5.proxy_arp = 1 net.ipv4.conf.br-phy5.rp_filter = 0 net.ipv4.ip_forward = 1 EOF # Apply changes sysctl -p` **4.4 Verify Settings** `# Check proxy ARP sysctl net.ipv4.conf.br-phy5.proxy_arp # Should output: net.ipv4.conf.br-phy5.proxy_arp = 1 # Check rp_filter sysctl net.ipv4.conf.br-phy5.rp_filter # Should output: net.ipv4.conf.br-phy5.rp_filter = 0 # Check IP forwarding sysctl net.ipv4.ip_forward # Should output: net.ipv4.ip_forward = 1` #### **Step 5: Create Subnet Pools** Subnet pools with address scopes ensure only authorized subnets are advertised via BGP. **5.1 Create subnet Pools** You can create multiple pools for different purposes: `# Provider network pool openstack subnet pool create bgp-pool-provider \ --address-scope bgp-scope-v4 \ --prefix 172.200.0.0/16 \ --default-prefix-length 24 # tenant network pool openstack subnet pool create bgp-pool-tenant \ --address-scope bgp-scope-v4 \ --prefix 100.10.0.0/16 \ --default-prefix-length 24` **5.3 Verify Subnet Pools** `openstack subnet pool list # Check details openstack subnet pool show bgp-pool-v4` *** #### **Step 6: Create Networks** Create tenant networks using the subnet pools to enable BGP advertisement. **6.1 Create Tenant Network** `openstack network create tenant-bgp-net` **6.2 Create Subnet from Pool** `openstack subnet create tenant-bgp-subnet \ --network tenant-bgp-net \ --subnet-pool bgp-pool-tenant \ --subnet-range 10.100.10.0/24` **Expected Output:** `+----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | allocation_pools | 10.100.10.2-10.100.10.254 | | cidr | 10.100.10.0/24 | | created_at | 2025-11-28T14:00:00Z | | description | | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 10.100.10.1 | | host_routes | | | id | 9b4d8f0a-2345-6789-01bc-def123456789 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | tenant-bgp-subnet | | network_id | 7c5e9d1b-3456-7890-12cd-ef1234567890 | | project_id | 86acdbd1d72745fd8e8320edd7543400 | | subnetpool_id | 8a3c7e9f-1234-5678-90ab-cdef12345678 | | updated_at | 2025-11-28T14:00:00Z | +----------------------+--------------------------------------+` **6.3 Create Provider (External / Public) Network** `openstack network create provider-bgp-net --provider-network-type vlan \ --provider-physical-network phy-net \ --provider-segment 100 \ --external` **6.4 Create Subnet from Pool** `openstack subnet create provider-bgp-subnet \ --network provider-bgp-net \ --subnet-pool bgp-pool-provider \ --subnet-range 172.200.10.0/24` **6.5 Create Router with External Gateway** `# Create router openstack router create tenant-bgp-router # Set external gateway (provider network) openstack router set \ --external-gateway provider-bgp-net \ tenant-bgp-router # Add interface to tenant network openstack router add subnet \ tenant-bgp-router \ tenant-bgp-subnet` **6.6 Create VM on Tenant Network** `openstack server create \ --flavor m1.small \ --image ubuntu-20.04 \ --network tenant-bgp-net \ --key-name mykey \ test-vm-bgp` **6.7 (Optional) Assign Floating IP** `# Create floating IP openstack floating ip create provider-network # Assign to VM openstack server add floating ip test-vm-bgp ` *** ### **Verification** #### **Step 1: Verify BGP Peering Status** `sudo vtysh -c "show bgp summary"` **Expected Output:** `IPv4 Unicast Summary (VRF default): BGP router identifier 192.168.10.30, local AS number 64999 vrf-id 0 BGP table version 13 RIB entries 19, using 3648 bytes of memory Peers 1, using 724 KiB of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc 192.168.10.152 4 64999 1339 1345 0 0 0 22:15:38 1 9 N/A Total number of neighbors 1` **Check:** * ✅ **State/PfxRcd**: Should show number of received prefixes * ✅ **Up/Down**: Should show uptime (e.g., 22:15:38) * ✅ **PfxSnt**: Should show number of sent prefixes (>0 if advertising) #### **Step 2: Verify Advertised Routes** `sudo vtysh -c "show bgp ipv4 unicast"` **Expected Output like:** `BGP table version is 13, local router ID is 192.168.10.30, vrf id 0 Default local pref 100, local AS 64999 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *> 10.100.10.5/32 0.0.0.0@10< 0 32768 ? *> 10.100.10.10/32 0.0.0.0@10< 0 32768 ? *> 172.200.0.50/32 0.0.0.0@10< 0 32768 ? *> 192.168.10.0/24 0.0.0.0 100 0 32768 ? Displayed 4 routes and 4 total paths` **Check:** * ✅ **VM IPs (/32)**: Should see individual VM IPs advertised * ✅ \*\*Next Hop 0.0.0.0 `10**:` Indicates routes from bgp-vrf (table 10) * ✅ **Path "32768 ?"**: Indicates locally originated routes\*\* #### **Step 3: Verify Kernel Routes** `# Check main routing table ip route show # Check bgp-vrf routing table (table 10) ip route show table 10` **Expected in table 10:** `10.100.10.5 dev bgp-nic scope link 10.100.10.10 dev bgp-nic scope link 172.200.0.50 dev bgp-nic scope link` ### **Troubleshooting** #### **Issue 1: BGP Peering Not Established** **Symptoms:** `Neighbor State 192.168.10.152 Idle` **Diagnosis:** `# Check BGP logs journalctl -u frr -n 100 | grep -i bgp # Check connectivity to router ping 192.168.10.152 # Check if BGP port is open telnet 192.168.10.152 179 # Check FRR BGP status sudo vtysh -c "show bgp neighbors 192.168.10.152"` **Solutions:** 1. **Check L2 connectivity** `# Verify interface is up ip link show ens192 # Check ARP table ip neigh show` 2. **Check firewall rules** `# Allow BGP port 179 iptables -I INPUT -p tcp --dport 179 -j ACCEPT iptables -I OUTPUT -p tcp --sport 179 -j ACCEPT` 3. **Verify AS numbers match** * For iBGP: Both sides must have same AS number * For eBGP: AS numbers must be different 4. **Check router configuration** * Ensure router has matching BGP configuration * Verify router is listening on BGP port #### **Issue 2: Routes Not Advertised** **Symptoms:** * BGP peering established * No routes shown in `show bgp ipv4 unicast` **Diagnosis:** `# Check if routes exist in kernel ip route show table 10 # Check OVN BGP agent logs journalctl -u ovn-bgp-agent -n 100 # Check if VMs have IPs openstack server list --long` **Solutions:** 1. **Verify address scope configuration** `# Check subnet pool has correct address scope openstack subnet pool show bgp-pool-v4 -c address_scope_id # Check subnet is from correct pool openstack subnet show tenant-bgp-subnet -c subnetpool_id` 2. **Restart OVN BGP agent** `systemctl restart ovn-bgp-agent # Wait and check logs journalctl -u ovn-bgp-agent -f` 3. **Check OVN SB database connectivity** `# Test connection ovn-sbctl --db=tcp:127.0.0.1:6642 show` 4. **Verify bgp-nic and bgp-vrf exist** `ip link show bgp-nic ip link show bgp-vrf # If missing, restart agent systemctl restart ovn-bgp-agent` #### **Issue 3: VMs Not Reachable from External Network** **Symptoms:** * Routes advertised correctly * Cannot ping VM IPs from external network **Diagnosis:** `# Check proxy ARP settings sysctl net.ipv4.conf.br-phy5.proxy_arp sysctl net.ipv4.conf.br-phy5.rp_filter # Check IP forwarding sysctl net.ipv4.ip_forward # Check OVS flows ovs-ofctl dump-flows br-int | grep ` **Solutions:** 1. **Enable proxy ARP** (if not enabled) `sudo sysctl -w net.ipv4.conf.br-phy5.proxy_arp=1 sudo sysctl -w net.ipv4.conf.br-phy5.rp_filter=0 sudo sysctl -w net.ipv4.ip_forward=1` 2. **Check security groups** `# Allow ICMP openstack security group rule create \ --protocol icmp \ --ingress \ default` 3. **Verify OVN flows** `# Check logical flows ovn-sbctl lflow-list | grep ` 4. **Check router gateway** `# Ensure router has external gateway openstack router show tenant-bgp-router -c external_gateway_info` #### **Issue 4: FRR Not Starting** **Symptoms:** `systemctl status frr Active: failed` **Diagnosis:** `# Check FRR logs journalctl -u frr -n 100 # Check configuration syntax sudo vtysh -c "show running-config" # Check for port conflicts netstat -tlnp | grep 179` **Solutions:** 1. **Fix configuration syntax errors** `# Validate config sudo vtysh -f /etc/frr/frr.conf # Check for errors in output` 2. **Enable BGP daemon** `# Edit /etc/frr/daemons vi /etc/frr/daemons # Set bgpd=yes bgpd=yes # Restart systemctl restart frr` 3. **Check file permissions** `ls -l /etc/frr/frr.conf # Should be: -rw-r----- 1 frr frr chown frr:frr /etc/frr/frr.conf chmod 640 /etc/frr/frr.conf`