by Micheline Murphy, with guest writer, A.J. Murray[i]
A few weeks ago, I had a great opportunity to collaborate with fellow everyday engineer and Cisco Champion, A.J. Murray on an ACI migration. It proved to be such a great experience that we decided to team up and share our adventures in ACI migration...Just for Fun!
To begin, I’ll have AJ set up the design considerations for our readers. I’ll cover the constraints and challenges of moving from a traditional network model to ACI, and we will round it out with how we met those challenges.
- AJ Murray, here.
We’re working with a customer that is moving to from a traditional three-tier network architecture to Cisco ACI. The customer, we’re going to call them Acme Bowling[ii], is a large global company with many sites and multiple data centers. For this deployment, only the Boston data center is targeted for the migration to ACI.
The Data Center located in Boston MA uses a subnet of 10.100.0.0/16. There are also two other data centers, one on the West Coast of the United States and using a subnet of 10.101.0.0/16 and another in the United Kingdom using a subnet of 10.102.0.0/16.
The company has office locations around the world and, depending upon the size of the site, they use a subnet within either the 172.16.0.0/12 or 192.168.0.0/16 RFC 1918 address spaces. The IT administrative team is located mostly within New England. IT administrators sit in Burlington VT, Portland ME, and New Haven CT. The IT admin’s workstations have been assigned static IPs within the subnet for their respective sites. The IT admins are not given special subnets restricted to administrative access. So, there are other clients within these same subnets with differing security permissions.
Customer policy requires that the network blocks access to administrative ports for SSH, HTTPS, and RDP (ports 22, 443, 3329) by end user subnets. However, the IT administrative team is located within these same subnets and must have remote access to these administrative ports. Additionally, the client subnets need to have access to all other ports to gain access to services hosted within the data center.
Here is a list of static IP addresses in use by the IT Administrative team, and the sites at which they are located.
Burlington, VT Portland, ME New Haven, CT
- 172.19.20.45 172.20.18.15 192.168.32.10
- 172.19.20.46 172.20.18.20 192.168.32.12
We need ACI to be able to:
- Allow access to administrative ports (SSH, HTTPS, and RDP) from the administrator IPs and the other data centers.
- Block access to administrative ports from all other client subnets.
- Allow access to all other ports from all Client Subnets so they may utilize services hosted at the DC.
ACI Constraints and Challenges
Hello all. This is Micheline Murphy.
ACI has a couple of big things that might take a little while to wrap your noodle around. One of those things is the shift from traditional networking’s blacklist model to ACI’s whitelist only model. In An Offer You Can't Refuse: ACI Contracts...Just for Fun I talked about the nuts and bolts of configuring ACI contracts, but didn’t really touch on what they mean from a higher, design level.
So, what does ACI’s whitelist model mean from a higher design perspective? Quite a bit, actually. In the clean environment of lab learning, it’s kind of easy to overlook. You configure contracts to permit traffic between such-and-such EPG to thus-and-so EPG. Everything not permitted is denied. No big deal, right?
But what happens when you want to migrate an existing network to ACI? Quite often, existing networks are crusted over with layers of access control lists (ACLs) that deny certain types of traffic, for example, no Telnet. Everything not specifically denied gets through. How do you translate from one mindset to the other?
The trick to remember is that ACI still uses ACLs, but it just breaks them down to two logical parts. The first part is a definition of the source and destination, that is, the consumer and provider EPGs. The second part is the contract between the provider and consumer EPG, that defines allowed or forbidden traffic. From the combination of the two parts, ACI can create its ACL.
If you’ll recall, a contract is nothing more than a collection of rules that all get applied together. A contract has at least one subject that groups one or more filters together. A filter identifies a single traffic type. A subject defines the action taken on the traffic type—permit, deny, or redirect. By combining EPGs and the contracts between them, you can impose the same traffic regulation as your old ACLs.
Let’s take a look at a few ACLs to see how they would translate into ACI.
ip access-list Example_1
10 permit ip 10.1.0.0/24 10.2.0.0/24
20 permit ip 10.2.0.0/24 10.1.0.0/24[iii]
This ACL can be translated very easily into ACI. One EPG is associated to the subnet 10.1.0.0/24, another EPG is associated to the subnet 10.2.0.0/24. Then there is a contract between the two EPGs that allows all IP traffic. Easy?
ACI’s topology view of a contract named All_Traffic permitting all IP traffic between EPG-A and EPG-B.
What does this ACL translate to in ACI?
ip access-list Example_2
10 deny ip 10.1.0.0/24 10.2.0.0/24
20 deny ip 10.2.0.0/24 10.1.0.0/24
30 deny tcp any any eq telnet
This ACL is more typical of what you would find in a traditional blacklist model network. We can see that the first two lines of the ACL forbid traffic between 10.1.0.0/24 and 10.2.0.0/24 in either direction. The third line denies a type of traffic. In this example, no Telnet is allowed.
Like the previous ACL, this second ACL translates easily into ACI. The first two lines of the ACL are actually unnecessary in ACI, except in one corner case.[iv] The third line of the ACL can be easily configured as a filter and a subject for a contract, identifying telnet as denied traffic. Like so:
Policy details of a contract that denies telnet traffic. Notice how the telnet filter identifies traffic going to and from port 23. You can achieve the same result by using the “Reverse Filter Ports” checkbox, above (which is True by default), but for the sake of clarity, I configured both filters.
Now that our brains are warmed up, let’s get complicated. Let’s say you wanted to allow all traffic between EPG-A and EPG-B, except telnet. And let’s also say that you wanted HTTP traffic between the two EPGs to go through a firewall. How would we structure all of this into a single contract? Like this:
- Subject name: permitAll
- Filter: IP traffic
- Subject action: permit
- Subject name: denyTelnet
- Filter: TCP port 23
- Subject action: deny
- Subject name: httpRedirect
- Filter: TCP port 80
- subject action: redirect[v]
Now here’s a question. How does ACI decide which rule trumps? Is it top-down as in traditional CLI-based ACLs? The short answer is no. What rule trumps is governed by priority. When you configure a subject, each filter has a field for priority. The field accepts four different values—default, lowest, medium, and highest. Priority is an easy way to for the network engineer to explicitly configure which rules she wants to be used first, without the hassle of a physical “top down.”
Building a filter with a configured priority.
Often, network engineers will leave the priority at default level. In the event that filters have the same priority, the following two rules apply.
- Deny wins over permit.
- Redirect wins over permit.[vi]
In our example contract above, we can see that ACI would apply the denyTelnet rules first, then the httpRedirect rules, and lastly the permitAll. If we left off the PBR, our Contract_Example1 would generally result in the same behavior as this ACL:
ip access-list example_3
10 deny tcp any any eq telnet
20 permit ip any any
One last thing. There’s an implicit last line for every ACL. That is, deny any any. The implicit deny any any means that for any ACL to pass traffic, it has to have at least one permit statement. So, ACLs written like Example_2, with no permit statements will, in fact, not pass any traffic.
The same is true in ACI.
This has considerable impact if you are trying to translate between blacklist and whitelist models. In a migration from a traditional network designed to identify forbidden traffic and permit everything else, it’s very important that ACI contracts be built with a last filter/subject that permits anything that hasn’t been previously denied. If you go back and take a look at the Example_2 contract denying telnet, you’ll see that permit any any is the second filter on the subject. Did you catch that?
Next, AJ and I will put it all together.
Putting It All Together: Taming an ACL Beast
As with the previous section, let’s start with the ACL. As often happens, ACLs can grow out of hand pretty quick, and Acme Bowling’ ACL was no exception. Here it is. It starts out allowing access from each admin /32 address to ports 3389 (for RDP), 22 (for SSH), and 443 (for HTTPS) into each DC network. Then, it denies traffic on those ports from all other addresses in the RFC space, except 10.0/8. Finally, it permits any other traffic it sees.
IP access list RESTRICT-ADMIN-PORTS
permit tcp 172.19.20.45/32 10.100.0.0/16 eq 3389
permit tcp 172.19.20.46/32 10.100.0.0/16 eq 3389
permit tcp 172.19.20.47/32 10.100.0.0/16 eq 3389
permit tcp 172.20.18.15/32 10.100.0.0/16 eq 3389
permit tcp 172.20.18.20/32 10.100.0.0/16 eq 3389
permit tcp 192.168.32.10/32 10.100.0.0/16 eq 3389
permit tcp 192.168.32.12/32 10.100.0.0/16 eq 3389
permit tcp 172.19.20.45/32 10.101.0.0/16 eq 3389
permit tcp 172.19.20.46/32 10.101.0.0/16 eq 3389
permit tcp 172.19.20.47/32 10.101.0.0/16 eq 3389
permit tcp 172.20.18.15/32 10.101.0.0/16 eq 3389
permit tcp 172.20.18.20/32 10.101.0.0/16 eq 3389
permit tcp 192.168.32.10/32 10.101.0.0/16 eq 3389
permit tcp 192.168.32.12/32 10.101.0.0/16 eq 3389
permit tcp 172.19.20.45/32 10.102.0.0/16 eq 3389
permit tcp 172.19.20.46/32 10.102.0.0/16 eq 3389
permit tcp 172.19.20.47/32 10.102.0.0/16 eq 3389
permit tcp 172.20.18.15/32 10.102.0.0/16 eq 3389
permit tcp 172.20.18.20/32 10.102.0.0/16 eq 3389
permit tcp 192.168.32.10/32 10.102.0.0/16 eq 3389
permit tcp 192.168.32.12/32 10.102.0.0/16 eq 3389
permit tcp 172.19.20.45/32 10.100.0.0/16 eq 22
permit tcp 172.19.20.46/32 10.100.0.0/16 eq 22
permit tcp 172.19.20.47/32 10.100.0.0/16 eq 22
permit tcp 172.20.18.15/32 10.100.0.0/16 eq 22
permit tcp 172.20.18.20/32 10.100.0.0/16 eq 22
permit tcp 192.168.32.10/32 10.100.0.0/16 eq 22
permit tcp 192.168.32.12/32 10.100.0.0/16 eq 22
permit tcp 172.19.20.45/32 10.101.0.0/16 eq 22
permit tcp 172.19.20.46/32 10.101.0.0/16 eq 22
permit tcp 172.19.20.47/32 10.101.0.0/16 eq 22
permit tcp 172.20.18.15/32 10.101.0.0/16 eq 22
permit tcp 172.20.18.20/32 10.101.0.0/16 eq 22
permit tcp 192.168.32.10/32 10.101.0.0/16 eq 22
permit tcp 192.168.32.12/32 10.101.0.0/16 eq 22
permit tcp 172.19.20.45/32 10.102.0.0/16 eq 22
permit tcp 172.19.20.46/32 10.102.0.0/16 eq 22
permit tcp 172.19.20.47/32 10.102.0.0/16 eq 22
permit tcp 172.20.18.15/32 10.102.0.0/16 eq 22
permit tcp 172.20.18.20/32 10.102.0.0/16 eq 22
permit tcp 192.168.32.10/32 10.102.0.0/16 eq 22
permit tcp 192.168.32.12/32 10.102.0.0/16 eq 22
permit tcp 172.19.20.45/32 10.100.0.0/16 eq 443
permit tcp 172.19.20.46/32 10.100.0.0/16 eq 443
permit tcp 172.19.20.47/32 10.100.0.0/16 eq 443
permit tcp 172.20.18.15/32 10.100.0.0/16 eq 443
permit tcp 172.20.18.20/32 10.100.0.0/16 eq 443
permit tcp 192.168.32.10/32 10.100.0.0/16 eq 443
permit tcp 192.168.32.12/32 10.100.0.0/16 eq 443
permit tcp 172.19.20.45/32 10.101.0.0/16 eq 443
permit tcp 172.19.20.46/32 10.101.0.0/16 eq 443
permit tcp 172.19.20.47/32 10.101.0.0/16 eq 443
permit tcp 172.20.18.15/32 10.101.0.0/16 eq 443
permit tcp 172.20.18.20/32 10.101.0.0/16 eq 443
permit tcp 192.168.32.10/32 10.101.0.0/16 eq 443
permit tcp 192.168.32.12/32 10.101.0.0/16 eq 443
permit tcp 172.19.20.45/32 10.102.0.0/16 eq 443
permit tcp 172.19.20.46/32 10.102.0.0/16 eq 443
permit tcp 172.19.20.47/32 10.102.0.0/16 eq 443
permit tcp 172.20.18.15/32 10.102.0.0/16 eq 443
permit tcp 172.20.18.20/32 10.102.0.0/16 eq 443
permit tcp 192.168.32.10/32 10.102.0.0/16 eq 443
permit tcp 192.168.32.12/32 10.102.0.0/16 eq 443
deny tcp 172.16.0.0/12 10.100.0.0/16 eq 3389
deny tcp 172.16.0.0/12 10.101.0.0/16 eq 3389
deny tcp 172.16.0.0/12 10.102.0.0/16 eq 3389
deny tcp 172.16.0.0/12 10.100.0.0/16 eq 22
deny tcp 172.16.0.0/12 10.101.0.0/16 eq 22
deny tcp 172.16.0.0/12 10.102.0.0/16 eq 22
deny tcp 172.16.0.0/12 10.100.0.0/16 eq 443
deny tcp 172.16.0.0/12 10.101.0.0/16 eq 443
deny tcp 172.16.0.0/12 10.102.0.0/16 eq 443
deny tcp 192.168.0.0/16 10.100.0.0/16 eq 3389
deny tcp 192.168.0.0/16 10.101.0.0/16 eq 3389
deny tcp 192.168.0.0/16 10.102.0.0/16 eq 3389
deny tcp 192.168.0.0/16 10.100.0.0/16 eq 22
deny tcp 192.168.0.0/16 10.101.0.0/16 eq 22
deny tcp 192.168.0.0/16 10.102.0.0/16 eq 22
deny tcp 192.168.0.0/16 10.100.0.0/16 eq 443
deny tcp 192.168.0.0/16 10.101.0.0/16 eq 443
deny tcp 192.168.0.0/16 10.102.0.0/16 eq 443
permit tcp 172.16.0.0/12 any
permit tcp 192.168.0.0/16 any
In all, this beastie is over eighty lines long. Translating it into ACI will require two parts:
- Making sure that admin addresses are in the same EPG, and
- Writing a contract that identifies the forbidden traffic for everyone else.
Putting all the admin addresses into the same EPG ended up being pretty easy, because of the decision to not migrate them into the fabric. That decision meant that they would belong to an external EPG, which can be configured with a /32 address.
Here, you can see an external EPG named ADMIN-TEAM-BURLINGTON containing the three /32 addresses designated for the Burlington team. Notice that they each have the flag “External Subnet for the External EPG.” This flag designates each hosts’ membership in the EPG ADMIN-TEAM-BURLINGTON.[vii]
The external EPG for the Burlington admins with two of the admin addresses showing.
The contract for the ADMIN-TEAM-BURLINGTON EPG should permit all traffic between the Burlington team’s EPG and the EPGs representing each data center.[viii]
The second part of the equation is building the proper contract for everyone else. As we discussed in the last section, it will need to identify all forbidden traffic AND permit everything else. In this case, several filters were built, identifying SSH, RDP, and HTTPS traffic by port. A fourth filter was also built, that identified IP traffic. A subject, named NO-ADMIN-TRAFFIC was built, denying the SSH, RDP, and HTTPS traffic, but permitting all IP traffic. If you’ll remember, by operation of the priority rules, the APIC will test traffic against each deny first, before testing the traffic for permit. Below, is the final subject.
The subject NO-ADMIN-TRAFFIC, showing the four filters used and the actions taken upon match. Remember, the order doesn’t matter. Since all of the filters are the same priority, the deny filters are applied first before any permit filter.
Verification that the contract works as intended is via ping and SSH. First, can an admin connect to a data center address? Below is a ping from 172.19.20.45, an admin IP in Burlington to a device in the Boston data center. We expect that ping to go through.
Ping from an admin IP in Burlington to a device in the Boston data center.
Next is an SSH attempt from an admin IP into a device in the Boston data center. Again, we expect to be able to connect.
SSH into a device in the Boston data center from an admin IP in Burlington.
The last test of the ACI configuration is to make sure non-admin users cannot SSH. This test was conducted using a user IP from the Hartford office, 192.168.55.127.
A non-admin IP address from the Hartford office...
...no connection allowed.
And that concludes this latest installment of ...Just for Fun! We both hope you enjoyed our adventures in ACLs and ACI. As always, I'm available on CLN for questions or comments. AJ can be found on Twitter at @noblinkyblinky. His blog is https://blog.noblinkyblinky.com/
Please share your own ACL or ACI stories.
[i] AJ Murray is a Network Deployment Engineer on the Enterprise Networking and Security Team at Cisco partner Red River, and a current Cisco Champion. When he’s not helping customers deploy new solutions, he likes to hike with his kids and eat Peanut Butter Pie ice cream from the local ice cream shop.
[ii] The name and the IP addresses of the actual customer were changed to mask identity.
[iii] Yes, I know that iOS requires a wildcard mask. For the sake of human-readability, I have substituted the wildcard mask for the slash notation.
[iv] That corner case is the event that 10.1.0.0/24 and 10.2.0.0/24 subnets are configured on a single bridge domain and endpoints in both subnets belong to the same EPG. In this corner case, endpoints in different subnets will be permitted to communicate. If you want to maintain segregation along subnet lines, you could configure micro-segmentation (uSeg) EPGs, or better yet, just different EPGs all together.
[v] Policy based redirect is beyond the scope of this article, but if you are really burning to know about ACI PBR, check out Smoke and Mirrors: ACI L4-L7 Service Insertion with Policy Based Redirect.
[vii] If you are burning to know more about external EPG flags, check out We Don't Need No Stinkin' Flags: ACI External Subnet Flags... Just for Fun.
[viii] If Acme Bowling eventually decides to pull the admin addresses into the fabric, security policy can be maintained by placing all the admin IPs in a uSeg EPG. Or, better yet, re-IP all of the admins, place them in their own subnet, and create an EPG just for admins.