13 Replies Latest reply: Dec 6, 2018 10:18 PM by Marwan Alshawi RSS

    BGP Multisite Design

    Luke

      I've been thinking about BGP design lately, especially around iBGP, and I would like to get your thoughts.

      I'm having trouble articulating the design problem here, so please bear with me.

       

      Imagine that we have three small sites, of 2 internet edge routers (running BGP), and a few core routers/L3 switches.

      Site 1 connects to site 2, site 2 connects to site 3. There is no direct connection between site 1 and site 3.

      Each site has two internet providers, with BGP peering in both cases. A separate router for each provider, iBGP between the pair in each site.

      There is only one public AS number for all sites. Each site advertises a /24 network.

       

       

      The result of this topology is that the iBGP network is partitioned. There is no peering between sites.

      I'm trying to fully understand the problems that this could present. Ideally there would be a full mesh (or RR's) between them. Is this necessary in this topology?

      Should I be thinking about route reflectors? If so, would they need to be in each site? I can just see as many route reflectors as active BGP speakers in this case.

       

      Now to throw in a tricky twist... What if there are L3 switches in each site running BGP/EVPN on the same AS? They're in a different address family/VRF, so would that affect the design at all?

       

       

      I realise this is a bit scattered, so thankyou for bearing with me. I'm trying to talk it out to better understand the problem.

       

      Thank you for your time.

         
        • 1. Re: BGP Multisite Design
          Martin

          one book comes to mind they may help you is Internet Routing Architectures from cisco press

          • 2. Re: BGP Multisite Design
            Luke

            Thanks, I'll have a look

            • 3. Re: BGP Multisite Design
              Kevin Santillan

              Imagine that we have three small sites, of 2 internet edge routers (running BGP), and a few core routers/L3 switches.

              Site 1 connects to site 2, site 2 connects to site 3. There is no direct connection between site 1 and site 3.

               

              How does one site connect to the other? Do you have a backdoor link between each pair or are you just describing how the sites should be connected?

               

              If your goal is to enable one site to accept the /24 blocks of the other two, you can configure allowas-in facing your ISP so that the CEs wouldn't ignore the update from the other two sites that originate NLRI with the same AS. Then just filter the site's own public block along with other bogon networks inbound. Is this what you're trying to accomplish?

               

              Now to throw in a tricky twist... What if there are L3 switches in each site running BGP/EVPN on the same AS? They're in a different address family/VRF, so would that affect the design at all?

               

              It might affect the design depending on what you are intending to advertise to the INET CEs since you will definitely have an RR for the fabric. But why not just use Private ASes and have your Border Leaf/Leaves form EBGP with the INET CEs if needed?

              • 4. Re: BGP Multisite Design
                Luke

                A lot is changing soon, so much of this is not yet set in stone. There will be two 1G DCI's between site-1/2 and again between site-2/3.

                 

                We already have site 2 and 3 advertising their own prefixes (but not each others). Site 1 does not yet have any BGP, but it will soon.

                I'm mainly focusing on site 1 and 2, while trying not to ignore the impact to site 3. Just writing that down gives me a headache.

                 

                I've never used allowas-in. I'll read up on it, and see if that will work.

                 

                We're basically a multitenanted environment, where tenants are business units (they are essentially customers though). One of these BU's has VXLAN already, so changing the AS is not an option. The SLA's are extremely tight with them, so very few changes get past change management.

                 

                I've also just counted up all the routers that we have that do or may run BGP (some managed by us, some third-party; Mostly WANs to BU offices), and I've realise that the original scope is growing. I think I'm going to need to think about route reflector clusters, or something like that.

                • 5. Re: BGP Multisite Design
                  Kevin Santillan

                  I thought these were just remote sites. Anyway, do you have L2 or L3 DCIs in between? Also, I'm assuming running VXLAN in your DCs isn't feasible at the moment which is why you're resorting to BGP?

                  • 6. Re: BGP Multisite Design
                    Juergen Ilse CCNA R&S

                    Luke schrieb:

                     

                    I've been thinking about BGP design lately, especially around iBGP, and I would like to get your thoughts.

                    I'm having trouble articulating the design problem here, so please bear with me.

                     

                    Imagine that we have three small sites, of 2 internet edge routers (running BGP), and a few core routers/L3 switches.

                    Site 1 connects to site 2, site 2 connects to site 3. There is no direct connection between site 1 and site 3.

                    Each site has two internet providers, with BGP peering in both cases. A separate router for each provider, iBGP between the pair in each site.

                    There is only one public AS number for all sites. Each site advertises a /24 network.

                     

                     

                    The result of this topology is that the iBGP network is partitioned. There is no peering between sites.

                    I'm trying to fully understand the problems that this could present. Ideally there would be a full mesh (or RR's) between them. Is this necessary in this topology?

                    Should I be thinking about route reflectors? If so, would they need to be in each site? I can just see as many route reflectors as active BGP speakers in this case.

                     

                    Now to throw in a tricky twist... What if there are L3 switches in each site running BGP/EVPN on the same AS? They're in a different address family/VRF, so would that affect the design at all?

                    Why do you want that? I would simply run all sites standalone with the same AS. There is nothing wromg about running "split AS". But in this case you should not forget to configure the neighbors from other site but with same as with "neighbor ... allowas-in" within the address-family. Otherwise you would not have connectivity between the different sites.

                    • 7. Re: BGP Multisite Design
                      waninae39

                      have you gone through chapter 5 of the CCDP book. it has many details on scaleable BGP and route reflector deployment

                       

                      maybe this summary of the chapter can help

                       

                      chapter5-1.PNG

                      chapter5-2.PNG

                      • 8. Re: BGP Multisite Design
                        Juergen Ilse CCNA R&S

                        waninae39 schrieb:

                         

                        have you gone through chapter 5 of the CCDP book.

                        No.

                        it has many details on scaleable BGP and route reflector deployment

                        maybe this summary of the chapter can help

                        Sorry, i don't see, how this may help for a "splitted AS" (what the main point in Luke's question seems to be). In my opinion, the best solution would be a configuration with "allowas-in".

                        • 9. Re: BGP Multisite Design
                          Luke

                          L3 DCI's...

                           

                          I'm using BGP for peering with internet and WAN providers. I also do some MP-BGP (but its a bit outside the scope of this)

                          • 10. Re: BGP Multisite Design
                            Luke

                            Split-AS! That's the term I'm looking for. Yes, I have been trying to decide if Split-AS is suitable in this environment, or if I need to have an end to end iBGP deployment.

                            I'm new to the AllowAS-In configuration, but it looks like it could be helpful

                            • 11. Re: BGP Multisite Design
                              Luke

                              Yeah, I'm not so much after technical details from a book, as I'm after design ideas, and how to make it work in this environment

                              • 12. Re: BGP Multisite Design
                                Kevin Santillan

                                My take on this is don't overcomplicate the design. If you had L2 DCIs then that's when you might need redundant RRs in Site 2 since there would be no intermediate AS between sites that would stop IBGP split-horizon. But since you have L3 DCIs and are already EBGP peering with the SP, similar to my initial suggestion, just do allowas-in to accept NLRI with the same AS and only advertise what's needed between the sites.

                                 

                                The only possible limitation is within Site 2. How many hops away is it from your CEs in Site 2 facing Site 1 to your CEs in Site 2 facing Site 3? Or are you using the same pair of CEs in Site 2 for all DCIs?

                                • 13. Re: BGP Multisite Design
                                  Marwan Alshawi

                                  The main point here before you look into BGP, is how the applications are designed to service external users

                                  do you have same exact applications and DB etc. replicated in each DC in which if a user request land in any DC will be able to be served locally by that DC?

                                  if the answer is yes, then I wouldn't even extend the iBGP among the DC sites, because users will be routed to the closest DC based on the internet routing, this design called anycast, some cloud providers they use it >> assuming applications and storage/DB replication is in place

                                   

                                  if the the above not applicable, you need again to understand and discuss with the application team how the application should serve users, which DC is the primary for which application etc. then after that you can find out which iBGP/eBGP peering model you can have among your DC sites

                                  hope this helps