8 Replies Latest reply: Jun 17, 2019 1:29 PM by Omar Ortiz Rodriguez RSS

    Output Drops

    Cisco335

      Hi Guys,

       

      I have started observing the Total Output Drops, on the MPLS link towards ISP, which are constantly incrementing. Any idea what could be the issue? This just started happening, otherwise it had been stable. I have got Speed 100, Duplex Full, and Bandwidth 100000 configured on the port.

       

       

      Router#sh int gig 0/0

      GigabitEthernet0/0 is up, line protocol is up

      MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

           reliability 255/255, txload 44/255, rxload 53/255

        Encapsulation ARPA, loopback not set

        Keepalive set (10 sec)

        Full Duplex, 100Mbps, media type is RJ45

        output flow-control is unsupported, input flow-control is unsupported

        ARP type: ARPA, ARP Timeout 04:00:00

        Last input 00:00:00, output 00:00:00, output hang never

        Last clearing of "show interface" counters 01:35:38

        Input queue: 0/150/0/17 (size/max/drops/flushes); Total output drops: 124726

        Queueing strategy: Class-based queueing

        Output queue: 0/2000/124726 (size/max total/drops)

        30 second input rate 21070000 bits/sec, 4626 packets/sec

        30 second output rate 17453000 bits/sec, 5002 packets/sec

           30148128 packets input, 15130454289 bytes, 0 no buffer

           Received 4 broadcasts (0 IP multicasts)

           0 runts, 0 giants, 0 throttles

           0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

           0 watchdog, 0 multicast, 0 pause input

           30196743 packets output, 12999722869 bytes, 0 underruns

           0 output errors, 0 collisions, 0 interface resets

       

       

      Regards

         
        • 1. Re: Output Drops
          Omar Ortiz Rodriguez

          Hi!!

           

          Can you share the verion of your router, the model, and the Qos Configuration

           

          you have observed some increase in traffic toward the ISP? !

          • 2. Re: Output Drops
            Samer

            apparently you got something to do with QoS causing this drop.

            • 3. Re: Output Drops
              Cisco335

              Its Cisco 3900, Version 15.5(3)M7. Here is the qos config. No the traffic towards the ISP has been consistent


              class-map match-any GOLD

              match ip dscp cs2

              match ip dscp af21

              match access-group name management

              class-map match-any BRONZE

              match ip dscp cs1

              class-map match-any PLATINUM

              match ip dscp cs3

              match ip dscp af31

              class-map match-all AVC_TCP_UDP_Apps

              match access-group name AVC_ipv4_tcp_udp

              class-map match-any AVC_media_app

              match protocol telepresence-media

              match protocol rtp

              class-map match-any VIDEOCONF

              match ip dscp cs4

              match ip dscp af41

              class-map match-any VOICE

              match ip dscp ef

              class-map match-any ROUTING

              match ip dscp cs6

              match ip dscp cs7

              match access-group name routing

              class-map match-any SCAVENGER

              match ip dscp cs1

              class-map match-all DROP-UDP

              match access-group name deny-udp

              class-map match-all AVC_Media

              match access-group name AVC_ipv4_udp

              match class-map AVC_media_app

              !

              policy-map WAN-COS-IN

              class ROUTING

              class VOICE

                police cir 30000000 conform-action transmit

              class PLATINUM

                police cir 7000000 conform-action transmit  exceed-action set-dscp-transmit af32

              class VIDEOCONF

                police cir 35000000 conform-action transmit  exceed-action set-dscp-transmit af42

              class GOLD

                police cir 22000000 conform-action transmit  exceed-action set-dscp-transmit af22

              class SCAVENGER

                police cir 2000000 conform-action transmit

              class class-default

              policy-map WAN-COS-OUT

              class ROUTING

                bandwidth 1500

              class VOICE

                priority 30000

                police cir 30000000 conform-action transmit  exceed-action drop

              class PLATINUM

                bandwidth 7000

                police cir 7000000 conform-action transmit  exceed-action set-dscp-transmit af32

              class VIDEOCONF

                bandwidth 35000

                police cir 35000000 conform-action transmit  exceed-action set-dscp-transmit af42

              class GOLD

                set ip dscp cs2

                bandwidth 22000

                police cir 22000000 conform-action transmit  exceed-action set-dscp-transmit af22

                queue-limit 2048 packets

              class class-default

                bandwidth 3500

                queue-limit 3072 packets

                random-detect dscp-based

              policy-map SHAPE-WAN-OUT

              class class-default

                shape average percent 99

                 service-policy WAN-COS-OUT

              policy-map POLICE-WAN-IN

              class class-default

                police cir 100000000 conform-action transmit  exceed-action drop

                 service-policy WAN-COS-IN

              policy-map type performance-monitor NETFLOW_AVC_NBAR

              class AVC_Media

                flow monitor AVC_NBAR_Media

              class AVC_TCP_UDP_Apps

                flow monitor AVC_NBAR_TCP_UDP

              class class-default

                flow monitor AVC_IP

              policy-map COPP-DROP-UDP

              class DROP-UDP

                drop





              • 4. Re: Output Drops
                jh

                They are a very small percentage, but if they are causing you problems you could try following the suggestions here:

                https://www.cisco.com/c/en/us/support/docs/routers/10000-series-routers/6343-queue-drops.html?dtid=pseggl000183&ccid=cc0…

                You could also do some traffic analysis to try to find out what has changed to reduce/control that.

                As the link says, "The ultimate solution to resolve the problem is to increase the line speed.". Isn't it always

                • 5. Re: Output Drops
                  Omar Ortiz Rodriguez

                  Please can you share the output of the command show policy-map interface XX command and the config of the interface

                  • 6. Re: Output Drops
                    Steven Davidson

                    The drops might very well be normal and just the result of QoS policy at work.  Seconding Omar's request.  We need the show policy-map interface output to help diagnosis the "problem" further.

                    • 7. Re: Output Drops
                      Cisco335

                      Here is the output;

                       

                       

                      #sh policy-map interface gigabitEthernet

                      GigabitEthernet

                        Service-policy input: POLICE-WAN-IN

                          Class-map: class-default (match-any)
                            256084513 packets, 104924344506 bytes
                            30 second offered rate 53940000 bps, drop rate 0000 bps
                            Match: any
                            police:
                                cir 100000000 bps, bc 3125000 bytes
                              conformed 256084514 packets, 104924345805 bytes; actions:
                                transmit
                              exceeded 0 packets, 0 bytes; actions:
                                drop
                              conformed 53941000 bps, exceeded 0000 bps

                            Service-policy : WAN-COS-IN

                              Class-map: ROUTING (match-any)
                                1224423 packets, 102021974 bytes
                                30 second offered rate 3000 bps
                                Match: ip dscp cs6 (48)
                                  1221357 packets, 100764389 bytes
                                  30 second rate 3000 bps
                                Match: ip dscp cs7 (56)
                                  0 packets, 0 bytes
                                  30 second rate 0 bps
                                Match: access-group name routing
                                  3066 packets, 1257585 bytes
                                  30 second rate 0 bps

                              Class-map: VOICE (match-any)
                                90351861 packets, 20126337346 bytes
                                30 second offered rate 3140000 bps, drop rate 0000 bps
                                Match: ip dscp ef (46)
                                  90351861 packets, 20126337560 bytes
                                  30 second rate 3140000 bps
                                police:
                                    cir 30000000 bps, bc 937500 bytes
                                  conformed 90351861 packets, 20126337560 bytes; actions:
                                    transmit
                                  exceeded 0 packets, 0 bytes; actions:
                                    drop
                                  conformed 3140000 bps, exceeded 0000 bps

                              Class-map: PLATINUM (match-any)
                                5515075 packets, 3209150064 bytes
                                30 second offered rate 609000 bps, drop rate 0000 bps
                                Match: ip dscp cs3 (24)
                                  5515075 packets, 3209150064 bytes
                                  30 second rate 609000 bps
                                Match: ip dscp af31 (26)
                                  0 packets, 0 bytes
                                  30 second rate 0 bps
                                police:
                                    cir 7000000 bps, bc 218750 bytes
                                  conformed 5514852 packets, 3208833179 bytes; actions:
                                    transmit
                                  exceeded 223 packets, 316885 bytes; actions:
                                    set-dscp-transmit af32
                                  conformed 609000 bps, exceeded 0000 bps

                              Class-map: VIDEOCONF (match-any)
                                9758625 packets, 9067757624 bytes
                                30 second offered rate 0000 bps, drop rate 0000 bps
                                Match: ip dscp cs4 (32)
                                  9741326 packets, 9059705765 bytes
                                  30 second rate 0 bps
                                Match: ip dscp af41 (34)
                                  17299 packets, 8051859 bytes
                                  30 second rate 0 bps
                                police:
                                    cir 35000000 bps, bc 1093750 bytes
                                  conformed 9758625 packets, 9067757624 bytes; actions:
                                    transmit
                                  exceeded 0 packets, 0 bytes; actions:
                                    set-dscp-transmit af42
                                  conformed 0000 bps, exceeded 0000 bps

                              Class-map: GOLD (match-any)
                                40620139 packets, 25526249861 bytes
                                30 second offered rate 11497000 bps, drop rate 0000 bps
                                Match: ip dscp cs2 (16)
                                  35825113 packets, 24035581710 bytes
                                  30 second rate 11455000 bps
                                Match: ip dscp af21 (18)
                                  0 packets, 0 bytes
                                  30 second rate 0 bps
                                Match: access-group name management
                                  4795026 packets, 1490668151 bytes
                                  30 second rate 42000 bps
                                police:
                                    cir 22000000 bps, bc 687500 bytes
                                  conformed 39403753 packets, 23839996449 bytes; actions:
                                    transmit
                                  exceeded 1216394 packets, 1686261238 bytes; actions:
                                    set-dscp-transmit af22
                                  conformed 11474000 bps, exceeded 23000 bps

                              Class-map: SCAVENGER (match-any)
                                0 packets, 0 bytes
                                30 second offered rate 0000 bps, drop rate 0000 bps
                                Match: ip dscp cs1 (8)
                                  0 packets, 0 bytes
                                  30 second rate 0 bps
                                police:
                                    cir 2000000 bps, bc 62500 bytes
                                  conformed 0 packets, 0 bytes; actions:
                                    transmit
                                  exceeded 0 packets, 0 bytes; actions:
                                    drop
                                  conformed 0000 bps, exceeded 0000 bps

                              Class-map: class-default (match-any)
                                108614390 packets, 46892827423 bytes
                                30 second offered rate 38689000 bps, drop rate 0000 bps
                                Match: any

                        Service-policy output: SHAPE-WAN-OUT

                          Class-map: class-default (match-any)
                            270277508 packets, 93458125295 bytes
                            30 second offered rate 13379000 bps, drop rate 208000 bps
                            Match: any
                            Queueing
                            queue limit 4096 packets
                            (queue depth/total drops/no-buffer drops) 0/1104717/0
                            (pkts output/bytes output) 272576383/95425227460
                            shape (average) cir 99000000, bc 990000, be 990000
                            target shape rate 99000000


                            Service-policy : WAN-COS-OUT

                              queue stats for all priority classes:
                                Queueing
                                queue limit 64 packets
                                (queue depth/total drops/no-buffer drops) 0/0/0
                                (pkts output/bytes output) 85032795/18326754263

                              Class-map: ROUTING (match-any)
                                3018481 packets, 1173068834 bytes
                                30 second offered rate 90000 bps, drop rate 0000 bps
                                Match: ip dscp cs6 (48)
                                  3007201 packets, 1170540816 bytes
                                  30 second rate 90000 bps
                                Match: ip dscp cs7 (56)
                                  0 packets, 0 bytes
                                  30 second rate 0 bps
                                Match: access-group name routing
                                  11280 packets, 2528018 bytes
                                  30 second rate 0 bps
                                Queueing
                                queue limit 64 packets
                                (queue depth/total drops/no-buffer drops) 0/0/0
                                (pkts output/bytes output) 3018481/1173068834
                                bandwidth 1500 kbps

                              Class-map: VOICE (match-any)
                                85032797 packets, 18326754713 bytes
                                30 second offered rate 1572000 bps, drop rate 0000 bps
                                Match: ip dscp ef (46)
                                  85032797 packets, 18326754713 bytes
                                  30 second rate 1572000 bps
                                Priority: 30000 kbps, burst bytes 750000, b/w exceed drops: 0

                                police:
                                    cir 30000000 bps, bc 937500 bytes
                                  conformed 85032798 packets, 18326754818 bytes; actions:
                                    transmit
                                  exceeded 0 packets, 0 bytes; actions:
                                    drop
                                  conformed 1572000 bps, exceeded 0000 bps

                              Class-map: PLATINUM (match-any)
                                4961141 packets, 1927659804 bytes
                                30 second offered rate 141000 bps, drop rate 0000 bps
                                Match: ip dscp cs3 (24)
                                  4961141 packets, 1927659804 bytes
                                  30 second rate 141000 bps
                                Match: ip dscp af31 (26)
                                  0 packets, 0 bytes
                                  30 second rate 0 bps
                                Queueing
                                queue limit 64 packets
                                (queue depth/total drops/no-buffer drops) 0/0/0
                                (pkts output/bytes output) 4961141/1927659804
                                bandwidth 7000 kbps
                                police:
                                    cir 7000000 bps, bc 218750 bytes
                                  conformed 4961141 packets, 1927659804 bytes; actions:
                                    transmit
                                  exceeded 0 packets, 0 bytes; actions:
                                    set-dscp-transmit af32
                                  conformed 141000 bps, exceeded 0000 bps

                              Class-map: VIDEOCONF (match-any)
                                11353483 packets, 10468472029 bytes
                                30 second offered rate 6000 bps, drop rate 0000 bps
                                Match: ip dscp cs4 (32)
                                  11353483 packets, 10468472029 bytes
                                  30 second rate 6000 bps
                                Match: ip dscp af41 (34)
                                  0 packets, 0 bytes
                                  30 second rate 0 bps
                                Queueing
                                queue limit 64 packets
                                (queue depth/total drops/no-buffer drops) 0/0/0
                                (pkts output/bytes output) 11353483/10468472029
                                bandwidth 35000 kbps
                                police:
                                    cir 35000000 bps, bc 1093750 bytes
                                  conformed 11353483 packets, 10468472029 bytes; actions:
                                    transmit
                                  exceeded 0 packets, 0 bytes; actions:
                                    set-dscp-transmit af42
                                  conformed 6000 bps, exceeded 0000 bps

                              Class-map: GOLD (match-any)
                                54733119 packets, 23141093320 bytes
                                30 second offered rate 6820000 bps, drop rate 0000 bps
                                Match: ip dscp cs2 (16)
                                  16985736 packets, 5891424373 bytes
                                  30 second rate 3365000 bps
                                Match: ip dscp af21 (18)
                                  1496238 packets, 1774363058 bytes
                                  30 second rate 117000 bps
                                Match: access-group name management
                                  36251145 packets, 15475305889 bytes
                                  30 second rate 3335000 bps
                                Queueing
                                queue limit 2048 packets
                                (queue depth/total drops/no-buffer drops) 0/0/0
                                (pkts output/bytes output) 54733119/23141093320
                                QoS Set
                                  dscp cs2
                                    Packets marked 54733120
                                bandwidth 22000 kbps
                                police:
                                    cir 22000000 bps, bc 687500 bytes
                                  conformed 54733120 packets, 23141093386 bytes; actions:
                                    transmit
                                  exceeded 0 packets, 0 bytes; actions:
                                    set-dscp-transmit af22
                                  conformed 6820000 bps, exceeded 0000 bps


                              Class-map: class-default (match-any)
                                111178487 packets, 38421076595 bytes
                                30 second offered rate 4746000 bps, drop rate 304000 bps
                                Match: any
                                Queueing
                                queue limit 3072 packets
                                (queue depth/total drops/no-buffer drops) 0/1104717/0
                                (pkts output/bytes output) 113477363/40388179144
                                bandwidth 3500 kbps

                                  Exp-weight-constant: 9 (1/512)
                                  Mean queue depth: 0 packets
                                  dscp       Transmitted         Random drop      Tail drop          Minimum        Maximum     Mark
                                          pkts/bytes            pkts/bytes       pkts/bytes          thresh         thresh     prob

                                  default109779333/37178556018  73664/72480874  1028350/968315995         20            40  1/10
                                  af21     2189146/2922182751     56/77558       2342/3212232           32            40  1/10
                                  cs5       273098/199878368       9/4835         279/219098            30            40  1/10
                                  cs6       949113/62954450        0/0             12/1002              32            40  1/10
                                  cs7       286674/24608276        0/0              5/406               34            40  1/10

                      • 8. Re: Output Drops
                        Omar Ortiz Rodriguez

                        Hi, i can see that the major parts of drops are in the default class, you need to verify the type of traffic and flows, you can use ip accounting, or netflow, or an packet analyzer using a span port, once time you iddentified the traffic, you can implement control forms to reduce it, if it's possible.

                         

                        Regards