Smoke and Mirrors: ACI L4-L7 Service Insertion with Policy-Based Redirect...Just for Fun!

Smoke and Mirrors: ACI L4-L7 Service Insertion with Policy-Based Redirect...Just for Fun!

By Micheline Murphy


Traffic in a data center is a lot like an irrigation system. You need to make sure that there are no dry spots, of course.  But just delivering water might not be enough. Sometimes you might need to make sure that the volume in a channel don't exceed capacity. You might need to make sure that you don't have gators swimming in your channels gobbling up what they shouldn't. Enter the wonderous world of L4-L7 service insertion!


It used to be that in order to make the most of an L4-L7 device, such as a load balancer or security device, you would have to engage some crazy network sorcery in order to direct traffic through the device you wanted to insert. And an ACI L4-L7 service insertion might seem like some Gandalf level spellcasting. In this latest chapter of ...Just for Fun, we will peel away the curtain to reveal the magic behind the mystery of ACI L4-L7 service insertion with Policy-Based Redirect.


PBR... know it, love it, use it sparingly.

Policy based routing or policy-based redirect in ACI (PBR in both cases) is forwarding based on criteria other than the packet's destination IP address. And in both non-ACI and ACI environments, PBR can be used to direct traffic through a device that you want to insert into the traffic stream. Use cases for ACI PBR-based L4-L7 service insertion include:


  • Use PBR to insert firewalls or load balancers in the path between endpoints while keeping the default gateway on the Cisco ACI fabric to use distributed routing.
  • Use PBR to insert an L4-L7 device in the path between endpoints that are in the same subnet.
  • Use PBR to send traffic selectively to L4-L7 devices based on protocol and port filtering.
  • Use PBR to horizontally scale the performance of L4-L7 devices. [i]


For example, you can combine a contract that permits only web-based traffic with a PBR policy that redirects that traffic to an Intrusion Prevention System device (IPS) capable of doing deep packet inspection to weed out traffic to undesirable web sites... you know the ones.


Policy-based redirect does have a few limitations in ACI. So, let's keep those in mind:

  • ACI Release 2.0(1m) or later;
  • The ACI fabric is required to be the gateway for both attached servers and the L4-L7 device.
  • Deployment must be in routed mode (go-to).
  • The device interfaces must be in a bridge domain, versus a L3Out.
  • The PBR policy requires the device’s IP and MAC address.
  • The device requires a bridge domain with the following features:
    1. The bridge domain must be dedicated.  That is, it may not be either the provider or consumer bridge domain.[ii]
    2. Data plane learning must be disabled.[iii]
  • The bridge domain must belong to the same VRF as either the consumer or provider EPG/bridge domain.




As always, it’s a good idea to start with the topology.


Screen Shot 2019-08-16 at 2.37.00 PM.png

In our lab environment, we have two EPGs—EPG-A and EPG-C—with a contract named All_Traffic allowing ...well, all traffic between the two. That is, the bottom half of this topology is already built.  If you need a refresher on how to build ACI contracts, take a look An Offer You Can't Refuse:  ACI Contracts...Just for Fun!


We’re going to build the top half of this topology, and use PBR policies to logically insert the whole top-half right into the traffic stream created by All_Traffic.


“I cast the redirect spell.” Configuring a PBR-based L4-L7 Service Insertion

Now let’s get to making the magic happen. If you’re like me, I like to read my recipes before I start cooking... er, spellcasting. I mean, configuration.

  1. Create an L4-L7 Device in ACI.
  2. Create two device bridge domains.
  3. Configure the device.
  4. Create the PBR Policy.
  5. Create and apply the service graph.


OK, pre-flight done.  Let’s do this!

Step #1: Create an L4-L7 Device in ACI

Creating an L4-L7 device is done under tenant>Services>L4-L7>Devices. Right-click the folder to bring up the creation wizard. I’m going to create an unmanaged, virtual firewall named CSR2. And we need it to be in routed or go-to mode.


Screen Shot 2019-08-16 at 2.38.20 PM.png



Here you can see the device I made. It has two “physical” interfaces—Gi1, and Gi2—and you can see that I’ve designated Gi1 as the provider, or inside, interface, and Gi2 is the consumer, or outside, interface.


Step #2: Create the firewall bridge domains.

If you look back at the topology, we are going to need to create two more bridge domains—an inside domain, and an outside domain. We also need to make sure we meet some requirements. Do you remember the requirements?

  • Dedicated device bridge domain.
  • Disable data-plane learning.
  • VRF not different than either consumer or provider bridge domain.[iv]


So, let’s create some bridge domains. tenant>Networking>Bridge Domains. Right-click the folder to create. We’ll need to also enable ARP flooding.[v]  Your bridge domains’ L2 config should look like so:


Screen Shot 2019-08-16 at 2.39.44 PM.png


We’re also going to need some L3 configuration on the bridge domains. We’ll need a subnet so that the bridge domain can route traffic, and we’ll need to make sure unicast routing is enabled. Should end up looking something like this:


Screen Shot 2019-08-16 at 2.41.27 PM.png



What’s I’ve shown you are screenshots from BD-external. But don’t forget, you need a bridge domain for the inside as well. For folks following along at home, the gateway address for my BD-internal is


Step #3: Configure the device.

The specifics for device configuration will vary from device to device, but there are some essentials that are required. You will need IP addresses for the inside and outside interface. You’ll need some static routes so that your device knows how to get to your provider and consumer EPGs, and you’ll need to configure whatever manipulation you want your device to do.


Taking a look back at the topology for configuration specifics, my CSR’s inside interface (Gi1) needs an IP address in the BD-internal subnet, and the outside interface (Gi2) needs an IP address in the BD-external subnet. In this example, Gi1 is configured with the address and Gi2 is configured with address


Once that is done, CSR2 should be able to ping the gateway addresses configured on BD-internal and BD-external.



Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:


Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms



Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:


Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/2 ms


Now we need some static routes to EPG-A and EPG-C. The next hop should be the gateway address of the respective firewall bridge domain. That is, a static route to EPG-A (the provider) should use BD-internal’s gateway address, and a static route-EPG-C (the consumer) should use BD-external’s gateway address. Like so:


ip route

ip route



Once you configure the static routes, you should be able to ping the gateway addresses of EPG-A and EPG-C.



Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:


Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms



Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:


Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/2 ms


Lastly, you need to configure the device to do some sort of manipulation on the traffic. In this lab, I’m going to configure an access list to filter by traffic type, allowing TCP flows and ICMP flows, and denying all types.


Step #4: Create the PBR Policy

The PBR policy is configured in tenant>Policies>Protocol>L4-L7 Policy Based Redirect. You will need to configure two policies, one for each interface. The policy will need to be configured with each interface's IP and MAC address.


In my lab, I can use the command show interface to verify the interface’s IP and MAC. I just take that information and plug it right into the PBR Policy. Looks like this:


Screen Shot 2019-08-16 at 2.42.22 PM.png


Step #5: Configure and apply a service graph.

Creating the service graph.

The naming of this object is a little bit confusing. You're not actually creating a template, but an actual service graph. Navigate to tenant>Services>L4-L7>Service Graph Templates. Right-click to bring up the creation wizard for a Service Graph Template. This object doesn't demand too much information. You will to name the object, and specify Routed mode and route redirect. Looks like so:


Screen Shot 2019-08-16 at 2.43.31 PM.png


Applying the service graph.

To apply the new service graph template to this contract, navigate to the service graph template. It will be in tenant>L4-L7>Services>Service Graph Templates. Right-click the service graph you want to apply. You will get a drop-down menu that will, among other things, allow you to select to "Apply L4-L7 Service Graph Template".


Selecting that option will bring up a two-part wizard. In the first part, you will select a contract. In the second part, you will be asked to name the service graph template and configure the inside and outside interfaces. The second part is pretty critical. You want the PBR config to be like so.


Screen Shot 2019-08-16 at 2.44.19 PM.png


Do you see how I’ve got the consumer side lined up with the consumer interface, the external BD, and the external PBR policy? Same with the provider side. Internal BD, internal PBR policy, and provider interface.



“Does my spell go off?” Verifying PBR-based L4-L7 Service Insertion.

ACI does this easy-on-the-eyes topology that makes for easy verification. You go to tenant>Application Profile and select the Topology tab on the upper right side of the Work Pane.  It gives you a graphical display of the Application Profile. If you applied the service graph right, you’ll see a little cloud icon hanging out on the contract.



Screen Shot 2019-08-16 at 2.45.12 PM.png



If you mouse-over the contract, you’ll get a pop-up screen with important details like the name of the service graph, and the type of device.

Screen Shot 2019-08-16 at 2.46.08 PM.png


Another way of verifying that the service graph was applied is to take a look at show zoning-rule in the individual leaf CLI. Here’s an example of the show command.


Screen Shot 2019-08-16 at 2.46.46 PM.png



The scope is the number assigned to the VRF. EPG-A has a tag number of 49153 and EPG-C’s tag number is 16388.[i] You can see that I’ve highlighted two new rules between EPG-A and EPG-C with a redirect action.


Of course, the million-dollar question is “Is the device manipulating the traffic in the manner designed?”  In this example, we want to know if the access-list is filtering traffic. Since I used the log keyword on my access-list, show access-list will give me counters by match.


csr-2#show access-list

Extended IP access list 100

    10 permit tcp any any log (1295 matches)

    20 permit icmp any any log (39 matches)

    30 deny ip any any (4 matches)


Moving to the endpoints themselves, I expect ICMP and TCP flows to continue to pass between the endpoints, but UDP to fail.


[cisco@aci2-redfish-epgc-vm6 ~]$ ping -c 5

PING ( 56(84) bytes of data.

64 bytes from icmp_seq=1 ttl=59 time=1.32 ms

64 bytes from icmp_seq=2 ttl=60 time=0.684 ms

64 bytes from icmp_seq=3 ttl=60 time=0.498 ms

64 bytes from icmp_seq=4 ttl=60 time=0.564 ms

64 bytes from icmp_seq=5 ttl=60 time=0.531 ms


--- ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 4001ms

rtt min/avg/max/mdev = 0.498/0.720/1.325/0.309 ms



[cisco@aci2-redfish-epgc-vm6 ~]$ iperf3 -c [TCP]

Connecting to host, port 5201

[  4] local port 43562 connected to port 5201

[ ID] Interval Transfer    Bandwidth Retr  Cwnd

[  4]  0.00-1.00 sec  143 KBytes  1.17 Mbits/sec    1  24.0 KBytes     

[  4]  1.00-2.00 sec  0.00 Bytes  0.00 bits/sec    0  32.5 KBytes

[  4]  2.00-3.00 sec  86.3 KBytes  707 Kbits/sec    0  42.4 KBytes     

[  4]  3.00-4.00 sec  0.00 Bytes  0.00 bits/sec    0  42.4 KBytes     

[  4]  4.00-5.00 sec  0.00 Bytes  0.00 bits/sec    0  43.8 KBytes     

[  4]  5.00-6.00 sec  0.00 Bytes  0.00 bits/sec    0  46.7 KBytes     

[  4]  6.00-7.00 sec  154 KBytes  1.26 Mbits/sec    0  49.5 KBytes     

[  4]  7.00-8.00 sec  0.00 Bytes  0.00 bits/sec    0  53.7 KBytes     

[  4]  8.00-9.00 sec  0.00 Bytes  0.00 bits/sec    0  58.0 KBytes     


[  4] 9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0 60.8 KBytes     

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer    Bandwidth      Retr

[  4]  0.00-10.00 sec  383 KBytes  314 Kbits/sec    1 sender

[  4] 0.00-10.00  sec  170 KBytes  139 Kbits/sec receiver


iperf Done.



[cisco@aci2-redfish-epgc-vm6 ~]$ iperf3 -c -u  [UDP]

Connecting to host, port 5201

iperf3: error - unable to read from stream socket: Resource temporarily unavailable


I hope you’ve enjoyed this exploration into the mysteries of PBR-Based service insertion in ACI. As always, your comments and questions are welcome!


[i] Cisco Application Centric Infrastructure Policy Based Redirect Service Graph Design White Paper

[ii] In ACI Releases 3.1 or later, this requirement is relaxed and the device bridge domain may be either the provider or consumer bridge domain.

[iii] In ACI Releases 3.1 or later and on fabrics deployed on Nexus 9300 -EX or -FX switches, data plane learning may be disabled in the device EPG, rather than the entire bridge domain.

[iv] Are you looking for a good mnemonic?  “DD no D.”

[v] ACI won’t let you configure unknown unicast flood without ARP flooding, so when you slide the slider to flood, ACI will automatically check ARP flooding.

[vi] Where do you find such things?  Navigate to the respective object.  The tag number will be on the policy page, usually a few fields from the top.