Skip to main content

Example route advertisements setup

To learn how to implement a route reflection setup on bare-metal infrastructure, you can follow this sample configuration. This example demonstrates how to enable the necessary feature gates and configure objects to advertise pod and egress IP routes.

As a cluster administrator, you can configure the following example route advertisements setup for your cluster. This configuration is intended as a sample that demonstrates how to configure route advertisements.

Sample route advertisements setup​

You can implement Border Gateway Protocol (BGP) routing by using a sample configuration to set up route advertisements for your cluster. This configuration demonstrates how to configure route reflection on bare-metal infrastructure to share pod and egress IP routes.

As a cluster administrator, you can enable Border Gateway Protocol (BGP) routing support for your cluster. The configuration uses route reflection rather than a full mesh setup.

note

BGP routing is supported only on bare-metal infrastructure.

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You are logged in to the cluster as a user with cluster-admin privileges.
  • The cluster is installed on bare-metal infrastructure.
  • You have a bare-metal system with access to the cluster where you plan to run the FRR daemon container.

Procedure

  1. Confirm that the RouteAdvertisements feature gate is enabled by running the following command:

    $ oc get featuregate -oyaml | grep -i routeadvertisement
    Example output
          - name: RouteAdvertisements

  2. Configure the Cluster Network Operator (CNO) by running the following command:

    $ oc patch Network.operator.openshift.io cluster --type=merge \
      -p='
        {"spec":{
          "additionalRoutingCapabilities": {
            "providers": ["FRR"]},
            "defaultNetwork":{"ovnKubernetesConfig"{
              "routeAdvertisements":"Enabled"
              }}}}'

    It might take a few minutes for the CNO to restart all nodes.

  3. Get the IP addresses of the nodes by running the following command:

    $ oc get node -owide
    Example output
    NAME                                       STATUS   ROLES                  AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                                                KERNEL-VERSION                 CONTAINER-RUNTIME
    master-0                                   Ready    control-plane,master   27h   v1.31.3   192.168.111.20   <none>        Red Hat Enterprise Linux CoreOS 418.94.202501062026-0   5.14.0-427.50.1.el9_4.x86_64   cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
    master-1                                   Ready    control-plane,master   27h   v1.31.3   192.168.111.21   <none>        Red Hat Enterprise Linux CoreOS 418.94.202501062026-0   5.14.0-427.50.1.el9_4.x86_64   cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
    master-2                                   Ready    control-plane,master   27h   v1.31.3   192.168.111.22   <none>        Red Hat Enterprise Linux CoreOS 418.94.202501062026-0   5.14.0-427.50.1.el9_4.x86_64   cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
    worker-0                                   Ready    worker                 27h   v1.31.3   192.168.111.23   <none>        Red Hat Enterprise Linux CoreOS 418.94.202501062026-0   5.14.0-427.50.1.el9_4.x86_64   cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
    worker-1                                   Ready    worker                 27h   v1.31.3   192.168.111.24   <none>        Red Hat Enterprise Linux CoreOS 418.94.202501062026-0   5.14.0-427.50.1.el9_4.x86_64   cri-o://1.31.4-2.rhaos4.18.git33d7598.el9
    worker-2                                   Ready    worker                 27h   v1.31.3   192.168.111.25   <none>        Red Hat Enterprise Linux CoreOS 418.94.202501062026-0   5.14.0-427.50.1.el9_4.x86_64   cri-o://1.31.4-2.rhaos4.18.git33d7598.el9

  4. Get the default pod network of each node by running the following command:

    $ oc get node <node_name> -o=jsonpath={.metadata.annotations.k8s\\.ovn\\.org/node-subnets}
    Example output
    {"default":["10.129.0.0/23"],"ns1.udn-network-primary-layer3":["10.150.6.0/24"]}

  5. On the bare-metal hypervisor, get the IP address for the external FRR container to use by running the following command:

    $ ip -j -d route get <a cluster node's IP> | jq -r '.[] | .dev' | xargs ip -d -j address show | jq -r '.[] | .addr_info[0].local'

  6. Create a frr.conf file for FRR that includes each node’s IP address, as shown in the following example:

    Example frr.conf configuration file
    router bgp 64512
     no bgp default ipv4-unicast
     no bgp default ipv6-unicast
     no bgp network import-check
     neighbor 192.168.111.20 remote-as 64512
     neighbor 192.168.111.20 route-reflector-client
     neighbor 192.168.111.21 remote-as 64512
     neighbor 192.168.111.21 route-reflector-client
     neighbor 192.168.111.22 remote-as 64512
     neighbor 192.168.111.22 route-reflector-client
     neighbor 192.168.111.40 remote-as 64512
     neighbor 192.168.111.40 route-reflector-client
     neighbor 192.168.111.47 remote-as 64512
     neighbor 192.168.111.47 route-reflector-client
     neighbor 192.168.111.23 remote-as 64512
     neighbor 192.168.111.23 route-reflector-client
     neighbor 192.168.111.24 remote-as 64512
     neighbor 192.168.111.24 route-reflector-client
     neighbor 192.168.111.25 remote-as 64512
     neighbor 192.168.111.25 route-reflector-client
     address-family ipv4 unicast
      network 192.168.1.0/24
      network 192.169.1.1/32
     exit-address-family
     address-family ipv4 unicast
      neighbor 192.168.111.20 activate
      neighbor 192.168.111.20 next-hop-self
      neighbor 192.168.111.21 activate
      neighbor 192.168.111.21 next-hop-self
      neighbor 192.168.111.22 activate
      neighbor 192.168.111.22 next-hop-self
      neighbor 192.168.111.40 activate
      neighbor 192.168.111.40 next-hop-self
      neighbor 192.168.111.47 activate
      neighbor 192.168.111.47 next-hop-self
      neighbor 192.168.111.23 activate
      neighbor 192.168.111.23 next-hop-self
      neighbor 192.168.111.24 activate
      neighbor 192.168.111.24 next-hop-self
      neighbor 192.168.111.25 activate
      neighbor 192.168.111.25 next-hop-self
     exit-address-family
     neighbor  remote-as 64512
     neighbor  route-reflector-client
     address-family ipv6 unicast
      network 2001:db8::/128
     exit-address-family
     address-family ipv6 unicast
      neighbor  activate
      neighbor  next-hop-self
     exit-address-family

  7. Create a file named daemons that includes the following content:

    Example daemons configuration file
    # This file tells the frr package which daemons to start.
    #
    # Sample configurations for these daemons can be found in
    # /usr/share/doc/frr/examples/.
    #
    # ATTENTION:
    #
    # When activating a daemon for the first time, a config file, even if it is
    # empty, has to be present *and* be owned by the user and group "frr", else
    # the daemon will not be started by /etc/init.d/frr. The permissions should
    # be u=rw,g=r,o=.
    # When using "vtysh" such a config file is also needed. It should be owned by
    # group "frrvty" and set to ug=rw,o= though. Check /etc/pam.d/frr, too.
    #
    # The watchfrr and zebra daemons are always started.
    #
    bgpd=yes
    ospfd=no
    ospf6d=no
    ripd=no
    ripngd=no
    isisd=no
    pimd=no
    ldpd=no
    nhrpd=no
    eigrpd=no
    babeld=no
    sharpd=no
    pbrd=no
    bfdd=yes
    fabricd=no
    vrrpd=no

    #
    # If this option is set the /etc/init.d/frr script automatically loads
    # the config via "vtysh -b" when the servers are started.
    # Check /etc/pam.d/frr if you intend to use "vtysh"!
    #
    vtysh_enable=yes
    zebra_options="  -A 127.0.0.1 -s 90000000"
    bgpd_options="   -A 127.0.0.1"
    ospfd_options="  -A 127.0.0.1"
    ospf6d_options=" -A ::1"
    ripd_options="   -A 127.0.0.1"
    ripngd_options=" -A ::1"
    isisd_options="  -A 127.0.0.1"
    pimd_options="   -A 127.0.0.1"
    ldpd_options="   -A 127.0.0.1"
    nhrpd_options="  -A 127.0.0.1"
    eigrpd_options=" -A 127.0.0.1"
    babeld_options=" -A 127.0.0.1"
    sharpd_options=" -A 127.0.0.1"
    pbrd_options="   -A 127.0.0.1"
    staticd_options="-A 127.0.0.1"
    bfdd_options="   -A 127.0.0.1"
    fabricd_options="-A 127.0.0.1"
    vrrpd_options="  -A 127.0.0.1"

    # configuration profile
    #
    #frr_profile="traditional"
    #frr_profile="datacenter"

    #
    # This is the maximum number of FD's that will be available.
    # Upon startup this is read by the control files and ulimit
    # is called. Uncomment and use a reasonable value for your
    # setup if you are expecting a large number of peers in
    # say BGP.
    #MAX_FDS=1024

    # The list of daemons to watch is automatically generated by the init script.
    #watchfrr_options=""

    # for debugging purposes, you can specify a "wrap" command to start instead
    # of starting the daemon directly, e.g. to use valgrind on ospfd:
    #   ospfd_wrap="/usr/bin/valgrind"
    # or you can use "all_wrap" for all daemons, e.g. to use perf record:
    #   all_wrap="/usr/bin/perf record --call-graph -"
    # the normal daemon command is added to this at the end.

  8. Save both the frr.conf and daemons files in the same directory, such as /tmp/frr.

  9. Create an external FRR container by running the following command:

    $ sudo podman run -d --privileged --network host --rm --ulimit core=-1 --name frr --volume /tmp/frr:/etc/frr quay.io/frrouting/frr:9.1.0

  10. Create the following FRRConfiguration and RouteAdvertisements configurations:

    1. Create a receive_all.yaml file that includes the following content:
      Example receive_all.yaml configuration file
      apiVersion: frrk8s.metallb.io/v1beta1
      kind: FRRConfiguration
      metadata:
        name: receive-all
        namespace: openshift-frr-k8s
      spec:
        bgp:
          routers:
          - asn: 64512
            neighbors:
            - address: 192.168.111.1
              asn: 64512
              toReceive:
                allowed:
                  mode: all
    2. Create a ra.yaml file that includes the following content:
      Example ra.yaml configuration file
      apiVersion: k8s.ovn.org/v1
      kind: RouteAdvertisements
      metadata:
        name: default
      spec:
        nodeSelector: {}
        frrConfigurationSelector: {}
        networkSelectors:
        - networkSelectionType: DefaultNetwork
        advertisements:
        - "PodNetwork"
        - "EgressIP"

  11. Apply the receive_all.yaml and ra.yaml files by running the following command:

    $ for f in receive_all.yaml ra.yaml; do oc apply -f $f; done

Verification

  1. Verify that the configurations were applied:

    1. Verify that the FRRConfiguration configurations were created by running the following command:

      $ oc get frrconfiguration -A
      Example output
      NAMESPACE           NAME                   AGE
      openshift-frr-k8s   ovnk-generated-6lmfb   4h47m
      openshift-frr-k8s   ovnk-generated-bhmnm   4h47m
      openshift-frr-k8s   ovnk-generated-d2rf5   4h47m
      openshift-frr-k8s   ovnk-generated-f958l   4h47m
      openshift-frr-k8s   ovnk-generated-gmsmw   4h47m
      openshift-frr-k8s   ovnk-generated-kmnqg   4h47m
      openshift-frr-k8s   ovnk-generated-wpvgb   4h47m
      openshift-frr-k8s   ovnk-generated-xq7v6   4h47m
      openshift-frr-k8s   receive-all            4h47m

    2. Verify that the RouteAdvertisements configurations were created by running the following command:

      $ oc get ra -A
      Example output
      NAME      STATUS
      default   Accepted

  2. Get the external FRR container ID by running the following command:

    $ sudo podman ps | grep frr
    Example output
    22cfc713890e  quay.io/frrouting/frr:9.1.0              /usr/lib/frr/dock...  5 hours ago   Up 5 hours ago               frr

  3. Use the container ID that you obtained in the previous step to check the BGP neighbor and routes in the external FRR container’s vtysh session. Run the following command:

    $ sudo podman exec -it <container_id> vtysh -c "show ip bgp"
    Example output
    BGP table version is 10, local router ID is 192.168.111.1, vrf id 0
    Default local pref 100, local AS 64512
    Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                   i internal, r RIB-failure, S Stale, R Removed
    Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
    Origin codes:  i - IGP, e - EGP, ? - incomplete
    RPKI validation codes: V valid, I invalid, N Not found

        Network          Next Hop            Metric LocPrf Weight Path
     *>i10.128.0.0/23    192.168.111.22           0    100      0 i
     *>i10.128.2.0/23    192.168.111.23           0    100      0 i
     *>i10.129.0.0/23    192.168.111.20           0    100      0 i
     *>i10.129.2.0/23    192.168.111.24           0    100      0 i
     *>i10.130.0.0/23    192.168.111.21           0    100      0 i
     *>i10.130.2.0/23    192.168.111.40           0    100      0 i
     *>i10.131.0.0/23    192.168.111.25           0    100      0 i
     *>i10.131.2.0/23    192.168.111.47           0    100      0 i
     *> 192.168.1.0/24   0.0.0.0                  0         32768 i
     *> 192.169.1.1/32   0.0.0.0                  0         32768 i

  4. Find the frr-k8s pod for each cluster node by running the following command:

    $ oc -n openshift-frr-k8s get pod -owide
    Example output
    NAME                                      READY   STATUS    RESTARTS   AGE   IP               NODE                                       NOMINATED NODE   READINESS GATES
    frr-k8s-86wmq                             6/6     Running   0          25h   192.168.111.20   master-0                                   <none>           <none>
    frr-k8s-h2wl6                             6/6     Running   0          25h   192.168.111.21   master-1                                   <none>           <none>
    frr-k8s-jlbgs                             6/6     Running   0          25h   192.168.111.40   node1.example.com   <none>           <none>
    frr-k8s-qc6l5                             6/6     Running   0          25h   192.168.111.25   worker-2                                   <none>           <none>
    frr-k8s-qtxdc                             6/6     Running   0          25h   192.168.111.47   node2.example.com   <none>           <none>
    frr-k8s-s5bxh                             6/6     Running   0          25h   192.168.111.24   worker-1                                   <none>           <none>
    frr-k8s-szgj9                             6/6     Running   0          25h   192.168.111.22   master-2                                   <none>           <none>
    frr-k8s-webhook-server-6cd8b8d769-kmctw   1/1     Running   0          25h   10.131.2.9       node3.example.com   <none>           <none>
    frr-k8s-zwmgh                             6/6     Running   0          25h   192.168.111.23   worker-0                                   <none>           <none>

  5. From the OpenShift Container Platform cluster, check BGP routes on the cluster node’s frr-k8s pod in the FRR container by running the following command:

    $ oc -n openshift-frr-k8s -c frr rsh frr-k8s-86wmq

  6. Check the IP routes from the cluster node by running the following command:

    sh-5.1# vtysh
    Example output
    Hello, this is FRRouting (version 8.5.3).
    Copyright 1996-2005 Kunihiro Ishiguro, et al.

  7. Check the IP routes by running the following command:

    worker-2# show ip bgp
    Example output
    BGP table version is 10, local router ID is 192.168.111.25, vrf id 0
    Default local pref 100, local AS 64512
    Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
                   i internal, r RIB-failure, S Stale, R Removed
    Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
    Origin codes:  i - IGP, e - EGP, ? - incomplete
    RPKI validation codes: V valid, I invalid, N Not found

        Network          Next Hop            Metric LocPrf Weight Path
     *>i10.128.0.0/23    192.168.111.22           0    100      0 i
     *>i10.128.2.0/23    192.168.111.23           0    100      0 i
     *>i10.129.0.0/23    192.168.111.20           0    100      0 i
     *>i10.129.2.0/23    192.168.111.24           0    100      0 i
     *>i10.130.0.0/23    192.168.111.21           0    100      0 i
     *>i10.130.2.0/23    192.168.111.40           0    100      0 i
     *> 10.131.0.0/23    0.0.0.0                  0         32768 i
     *>i10.131.2.0/23    192.168.111.47           0    100      0 i
     *>i192.168.1.0/24   192.168.111.1            0    100      0 i
     *>i192.169.1.1/32   192.168.111.1            0    100      0 i

    Displayed  10 routes and 10 total paths

  8. From the OpenShift Container Platform cluster, debug the node by running the following command:

    $ oc debug node/<node_name>
    Example output
    Temporary namespace openshift-debug-lbtgh is created for debugging node...
    Starting pod/worker-2-debug-zrg4v ...
    To use host binaries, run `chroot /host`
    Pod IP: 192.168.111.25
    If you don't see a command prompt, try pressing enter.

  9. Confirm that the BGP routes are being advertised by running the following command:

    sh-5.1# ip route show | grep bgp
    Example output
    10.128.0.0/23 nhid 268 via 192.168.111.22 dev br-ex proto bgp metric 20
    10.128.2.0/23 nhid 259 via 192.168.111.23 dev br-ex proto bgp metric 20
    10.129.0.0/23 nhid 260 via 192.168.111.20 dev br-ex proto bgp metric 20
    10.129.2.0/23 nhid 261 via 192.168.111.24 dev br-ex proto bgp metric 20
    10.130.0.0/23 nhid 266 via 192.168.111.21 dev br-ex proto bgp metric 20
    10.130.2.0/23 nhid 262 via 192.168.111.40 dev br-ex proto bgp metric 20
    10.131.2.0/23 nhid 263 via 192.168.111.47 dev br-ex proto bgp metric 20
    192.168.1.0/24 nhid 264 via 192.168.111.1 dev br-ex proto bgp metric 20
    192.169.1.1 nhid 264 via 192.168.111.1 dev br-ex proto bgp metric 20