--- title: Example route advertisements setup --- # Example route advertisements setup {#example-route-advertisements-setup} To learn how to implement a route reflection setup on bare-metal infrastructure, you can follow this sample configuration. This example demonstrates how to enable the necessary feature gates and configure objects to advertise pod and egress IP routes. As a cluster administrator, you can configure the following example route advertisements setup for your cluster. This configuration is intended as a sample that demonstrates how to configure route advertisements. ## Sample route advertisements setup {#nw-route-advertisements-setup_example-route-advertisements-setup} You can implement Border Gateway Protocol (BGP) routing by using a sample configuration to set up route advertisements for your cluster. This configuration demonstrates how to configure route reflection on bare-metal infrastructure to share pod and egress IP routes. As a cluster administrator, you can enable Border Gateway Protocol (BGP) routing support for your cluster. The configuration uses route reflection rather than a full mesh setup. > [!NOTE] > BGP routing is supported only on bare-metal infrastructure. **Prerequisites** - You installed the OpenShift CLI (`oc`). - You are logged in to the cluster as a user with `cluster-admin` privileges. - The cluster is installed on bare-metal infrastructure. - You have a bare-metal system with access to the cluster where you plan to run the FRR daemon container. **Procedure** 1. Confirm that the `RouteAdvertisements` feature gate is enabled by running the following command: ```terminal $ oc get featuregate -oyaml | grep -i routeadvertisement ``` ```yaml {title="Example output"} - name: RouteAdvertisements ``` 2. Configure the Cluster Network Operator (CNO) by running the following command: ```terminal $ oc patch Network.operator.openshift.io cluster --type=merge \ -p=' {"spec":{ "additionalRoutingCapabilities": { "providers": ["FRR"]}, "defaultNetwork":{"ovnKubernetesConfig"{ "routeAdvertisements":"Enabled" }}}}' ``` It might take a few minutes for the CNO to restart all nodes. 3. Get the IP addresses of the nodes by running the following command: ```terminal $ oc get node -owide ``` ```text {title="Example output"} NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master-0 Ready control-plane,master 27h v1.31.3 192.168.111.20 Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9 master-1 Ready control-plane,master 27h v1.31.3 192.168.111.21 Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9 master-2 Ready control-plane,master 27h v1.31.3 192.168.111.22 Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9 worker-0 Ready worker 27h v1.31.3 192.168.111.23 Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9 worker-1 Ready worker 27h v1.31.3 192.168.111.24 Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9 worker-2 Ready worker 27h v1.31.3 192.168.111.25 Red Hat Enterprise Linux CoreOS 418.94.202501062026-0 5.14.0-427.50.1.el9_4.x86_64 cri-o://1.31.4-2.rhaos4.18.git33d7598.el9 ``` 4. Get the default pod network of each node by running the following command: ```terminal $ oc get node -o=jsonpath={.metadata.annotations.k8s\\.ovn\\.org/node-subnets} ``` ```text {title="Example output"} {"default":["10.129.0.0/23"],"ns1.udn-network-primary-layer3":["10.150.6.0/24"]} ``` 5. On the bare-metal hypervisor, get the IP address for the external FRR container to use by running the following command: ```terminal $ ip -j -d route get | jq -r '.[] | .dev' | xargs ip -d -j address show | jq -r '.[] | .addr_info[0].local' ``` 6. Create a `frr.conf` file for FRR that includes each node’s IP address, as shown in the following example: ```text {title="Example frr.conf configuration file"} router bgp 64512 no bgp default ipv4-unicast no bgp default ipv6-unicast no bgp network import-check neighbor 192.168.111.20 remote-as 64512 neighbor 192.168.111.20 route-reflector-client neighbor 192.168.111.21 remote-as 64512 neighbor 192.168.111.21 route-reflector-client neighbor 192.168.111.22 remote-as 64512 neighbor 192.168.111.22 route-reflector-client neighbor 192.168.111.40 remote-as 64512 neighbor 192.168.111.40 route-reflector-client neighbor 192.168.111.47 remote-as 64512 neighbor 192.168.111.47 route-reflector-client neighbor 192.168.111.23 remote-as 64512 neighbor 192.168.111.23 route-reflector-client neighbor 192.168.111.24 remote-as 64512 neighbor 192.168.111.24 route-reflector-client neighbor 192.168.111.25 remote-as 64512 neighbor 192.168.111.25 route-reflector-client address-family ipv4 unicast network 192.168.1.0/24 network 192.169.1.1/32 exit-address-family address-family ipv4 unicast neighbor 192.168.111.20 activate neighbor 192.168.111.20 next-hop-self neighbor 192.168.111.21 activate neighbor 192.168.111.21 next-hop-self neighbor 192.168.111.22 activate neighbor 192.168.111.22 next-hop-self neighbor 192.168.111.40 activate neighbor 192.168.111.40 next-hop-self neighbor 192.168.111.47 activate neighbor 192.168.111.47 next-hop-self neighbor 192.168.111.23 activate neighbor 192.168.111.23 next-hop-self neighbor 192.168.111.24 activate neighbor 192.168.111.24 next-hop-self neighbor 192.168.111.25 activate neighbor 192.168.111.25 next-hop-self exit-address-family neighbor remote-as 64512 neighbor route-reflector-client address-family ipv6 unicast network 2001:db8::/128 exit-address-family address-family ipv6 unicast neighbor activate neighbor next-hop-self exit-address-family ``` 7. Create a file named `daemons` that includes the following content: ```text {title="Example daemons configuration file"} # This file tells the frr package which daemons to start. # # Sample configurations for these daemons can be found in # /usr/share/doc/frr/examples/. # # ATTENTION: # # When activating a daemon for the first time, a config file, even if it is # empty, has to be present *and* be owned by the user and group "frr", else # the daemon will not be started by /etc/init.d/frr. The permissions should # be u=rw,g=r,o=. # When using "vtysh" such a config file is also needed. It should be owned by # group "frrvty" and set to ug=rw,o= though. Check /etc/pam.d/frr, too. # # The watchfrr and zebra daemons are always started. # bgpd=yes ospfd=no ospf6d=no ripd=no ripngd=no isisd=no pimd=no ldpd=no nhrpd=no eigrpd=no babeld=no sharpd=no pbrd=no bfdd=yes fabricd=no vrrpd=no # # If this option is set the /etc/init.d/frr script automatically loads # the config via "vtysh -b" when the servers are started. # Check /etc/pam.d/frr if you intend to use "vtysh"! # vtysh_enable=yes zebra_options=" -A 127.0.0.1 -s 90000000" bgpd_options=" -A 127.0.0.1" ospfd_options=" -A 127.0.0.1" ospf6d_options=" -A ::1" ripd_options=" -A 127.0.0.1" ripngd_options=" -A ::1" isisd_options=" -A 127.0.0.1" pimd_options=" -A 127.0.0.1" ldpd_options=" -A 127.0.0.1" nhrpd_options=" -A 127.0.0.1" eigrpd_options=" -A 127.0.0.1" babeld_options=" -A 127.0.0.1" sharpd_options=" -A 127.0.0.1" pbrd_options=" -A 127.0.0.1" staticd_options="-A 127.0.0.1" bfdd_options=" -A 127.0.0.1" fabricd_options="-A 127.0.0.1" vrrpd_options=" -A 127.0.0.1" # configuration profile # #frr_profile="traditional" #frr_profile="datacenter" # # This is the maximum number of FD's that will be available. # Upon startup this is read by the control files and ulimit # is called. Uncomment and use a reasonable value for your # setup if you are expecting a large number of peers in # say BGP. #MAX_FDS=1024 # The list of daemons to watch is automatically generated by the init script. #watchfrr_options="" # for debugging purposes, you can specify a "wrap" command to start instead # of starting the daemon directly, e.g. to use valgrind on ospfd: # ospfd_wrap="/usr/bin/valgrind" # or you can use "all_wrap" for all daemons, e.g. to use perf record: # all_wrap="/usr/bin/perf record --call-graph -" # the normal daemon command is added to this at the end. ``` 8. Save both the `frr.conf` and `daemons` files in the same directory, such as `/tmp/frr`. 9. Create an external FRR container by running the following command: ```terminal $ sudo podman run -d --privileged --network host --rm --ulimit core=-1 --name frr --volume /tmp/frr:/etc/frr quay.io/frrouting/frr:9.1.0 ``` 10. Create the following `FRRConfiguration` and `RouteAdvertisements` configurations: 1. Create a `receive_all.yaml` file that includes the following content: ```yaml {title="Example receive_all.yaml configuration file"} apiVersion: frrk8s.metallb.io/v1beta1 kind: FRRConfiguration metadata: name: receive-all namespace: openshift-frr-k8s spec: bgp: routers: - asn: 64512 neighbors: - address: 192.168.111.1 asn: 64512 toReceive: allowed: mode: all ``` 2. Create a `ra.yaml` file that includes the following content: ```yaml {title="Example ra.yaml configuration file"} apiVersion: k8s.ovn.org/v1 kind: RouteAdvertisements metadata: name: default spec: nodeSelector: {} frrConfigurationSelector: {} networkSelectors: - networkSelectionType: DefaultNetwork advertisements: - "PodNetwork" - "EgressIP" ``` 11. Apply the `receive_all.yaml` and `ra.yaml` files by running the following command: ```terminal $ for f in receive_all.yaml ra.yaml; do oc apply -f $f; done ``` **Verification** 1. Verify that the configurations were applied: 1. Verify that the `FRRConfiguration` configurations were created by running the following command: ```terminal $ oc get frrconfiguration -A ``` ```text {title="Example output"} NAMESPACE NAME AGE openshift-frr-k8s ovnk-generated-6lmfb 4h47m openshift-frr-k8s ovnk-generated-bhmnm 4h47m openshift-frr-k8s ovnk-generated-d2rf5 4h47m openshift-frr-k8s ovnk-generated-f958l 4h47m openshift-frr-k8s ovnk-generated-gmsmw 4h47m openshift-frr-k8s ovnk-generated-kmnqg 4h47m openshift-frr-k8s ovnk-generated-wpvgb 4h47m openshift-frr-k8s ovnk-generated-xq7v6 4h47m openshift-frr-k8s receive-all 4h47m ``` 2. Verify that the `RouteAdvertisements` configurations were created by running the following command: ```terminal $ oc get ra -A ``` ```text {title="Example output"} NAME STATUS default Accepted ``` 2. Get the external FRR container ID by running the following command: ```terminal $ sudo podman ps | grep frr ``` ```text {title="Example output"} 22cfc713890e quay.io/frrouting/frr:9.1.0 /usr/lib/frr/dock... 5 hours ago Up 5 hours ago frr ``` 3. Use the container ID that you obtained in the previous step to check the BGP neighbor and routes in the external FRR container’s `vtysh` session. Run the following command: ```terminal $ sudo podman exec -it vtysh -c "show ip bgp" ``` ```text {title="Example output"} BGP table version is 10, local router ID is 192.168.111.1, vrf id 0 Default local pref 100, local AS 64512 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *>i10.128.0.0/23 192.168.111.22 0 100 0 i *>i10.128.2.0/23 192.168.111.23 0 100 0 i *>i10.129.0.0/23 192.168.111.20 0 100 0 i *>i10.129.2.0/23 192.168.111.24 0 100 0 i *>i10.130.0.0/23 192.168.111.21 0 100 0 i *>i10.130.2.0/23 192.168.111.40 0 100 0 i *>i10.131.0.0/23 192.168.111.25 0 100 0 i *>i10.131.2.0/23 192.168.111.47 0 100 0 i *> 192.168.1.0/24 0.0.0.0 0 32768 i *> 192.169.1.1/32 0.0.0.0 0 32768 i ``` 4. Find the `frr-k8s` pod for each cluster node by running the following command: ```terminal $ oc -n openshift-frr-k8s get pod -owide ``` ```text {title="Example output"} NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES frr-k8s-86wmq 6/6 Running 0 25h 192.168.111.20 master-0 frr-k8s-h2wl6 6/6 Running 0 25h 192.168.111.21 master-1 frr-k8s-jlbgs 6/6 Running 0 25h 192.168.111.40 node1.example.com frr-k8s-qc6l5 6/6 Running 0 25h 192.168.111.25 worker-2 frr-k8s-qtxdc 6/6 Running 0 25h 192.168.111.47 node2.example.com frr-k8s-s5bxh 6/6 Running 0 25h 192.168.111.24 worker-1 frr-k8s-szgj9 6/6 Running 0 25h 192.168.111.22 master-2 frr-k8s-webhook-server-6cd8b8d769-kmctw 1/1 Running 0 25h 10.131.2.9 node3.example.com frr-k8s-zwmgh 6/6 Running 0 25h 192.168.111.23 worker-0 ``` 5. From the OpenShift Container Platform cluster, check BGP routes on the cluster node’s `frr-k8s` pod in the FRR container by running the following command: ```terminal $ oc -n openshift-frr-k8s -c frr rsh frr-k8s-86wmq ``` 6. Check the IP routes from the cluster node by running the following command: ```terminal sh-5.1# vtysh ``` ```text {title="Example output"} Hello, this is FRRouting (version 8.5.3). Copyright 1996-2005 Kunihiro Ishiguro, et al. ``` 7. Check the IP routes by running the following command: ```terminal worker-2# show ip bgp ``` ```text {title="Example output"} BGP table version is 10, local router ID is 192.168.111.25, vrf id 0 Default local pref 100, local AS 64512 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *>i10.128.0.0/23 192.168.111.22 0 100 0 i *>i10.128.2.0/23 192.168.111.23 0 100 0 i *>i10.129.0.0/23 192.168.111.20 0 100 0 i *>i10.129.2.0/23 192.168.111.24 0 100 0 i *>i10.130.0.0/23 192.168.111.21 0 100 0 i *>i10.130.2.0/23 192.168.111.40 0 100 0 i *> 10.131.0.0/23 0.0.0.0 0 32768 i *>i10.131.2.0/23 192.168.111.47 0 100 0 i *>i192.168.1.0/24 192.168.111.1 0 100 0 i *>i192.169.1.1/32 192.168.111.1 0 100 0 i Displayed 10 routes and 10 total paths ``` 8. From the OpenShift Container Platform cluster, debug the node by running the following command: ```terminal $ oc debug node/ ``` ```text {title="Example output"} Temporary namespace openshift-debug-lbtgh is created for debugging node... Starting pod/worker-2-debug-zrg4v ... To use host binaries, run `chroot /host` Pod IP: 192.168.111.25 If you don't see a command prompt, try pressing enter. ``` 9. Confirm that the BGP routes are being advertised by running the following command: ```terminal sh-5.1# ip route show | grep bgp ``` ```text {title="Example output"} 10.128.0.0/23 nhid 268 via 192.168.111.22 dev br-ex proto bgp metric 20 10.128.2.0/23 nhid 259 via 192.168.111.23 dev br-ex proto bgp metric 20 10.129.0.0/23 nhid 260 via 192.168.111.20 dev br-ex proto bgp metric 20 10.129.2.0/23 nhid 261 via 192.168.111.24 dev br-ex proto bgp metric 20 10.130.0.0/23 nhid 266 via 192.168.111.21 dev br-ex proto bgp metric 20 10.130.2.0/23 nhid 262 via 192.168.111.40 dev br-ex proto bgp metric 20 10.131.2.0/23 nhid 263 via 192.168.111.47 dev br-ex proto bgp metric 20 192.168.1.0/24 nhid 264 via 192.168.111.1 dev br-ex proto bgp metric 20 192.169.1.1 nhid 264 via 192.168.111.1 dev br-ex proto bgp metric 20 ```