5 Replies Latest reply: Dec 5, 2010 6:10 AM by Paul Stewart - CCIE Security RSS

    TCP ACKed Lost Segment.

    CISCOCCIE

      Dear All,

       

      I have a beneath setup --

       

       

      Branch Office                                                                                                          Head Office

       

      Client PC ---- Switch --- Bluecoat -- Packetshaper ---Router - WAN LINK -------**-Router --- ASA Firewall --- Packet Shaper --- IPS ------CORE SWITCH --- Server.

       

       

      After certain interval session between client and server is getting disconnected.

       

      Install and run wiresherk sniffer on both client as well as on server.

       

      On Server sniffer output is showing  TCP ACKed Lost segment.

       

      Please refer attached Wiresherk sniffer snaps on server and client.

       

      Thanks

        • 1. Re: TCP ACKed Lost Segment.
          Paul Stewart  -  CCIE Security

          That is very, very strange.   There is a frame 5 and 6 in the client capture that is not in the server capture.  I think frame 5 in the server capture is frame 7 in the client capture.  The wireshark interpretation in the server capture is correct because it is missing frames 5 and 6.  The ack number is too high.  Without those two missing frames, the ack should still be at 1.  With that being said, we see something very similar in the other direction.  If this is a highly used server and/or client, I would capture down to just this pair of addresses and disable the realtime view.  You might also make sure tcp checksum validation is disabled in the tcp protocol settings.  I've seen that filter packets out before.

           

          The other thing that this could be is the bluecoat screwing up the flow during optimization.  For example during the optimization, it is spoofing frames 5 and 6 to the client (in the client capture).  The client is acking with frame 7.  Frame 7 in the client capture is seen as frame 5 in the server capture.  I might actually try sniffing directly in front of and behind the bluecoat.  The captures won't be identical due to optimization.  However, there should not be protocol violations like this.  Bottom line is either your capture is incomplete on the server side (at least), or something injected packets in the stream, but didn't deal with the sequence/acks properly.  HTH.

          • 2. Re: TCP ACKed Lost Segment.
            CISCOCCIE

            Dear Paul,

             

            Thanks for your prompt.

             

            From your reply I understood that Bluecoat is causing this problem.

             

            Should i remove Bluecoat from the path and perform the same wiresherk test.

             

            Branch Office                                                                     Head Office

             

            Client PC ---- Switch --- Bluecoat -- Packetshaper ---Router - WAN LINK -------**-Router --- ASA Firewall --- Packet Shaper --- IPS ------CORE SWITCH --- Server.

             

            Thanks

            • 3. Re: TCP ACKed Lost Segment.
              Paul Stewart  -  CCIE Security

              If we assume that the client capture has nothing missing and the server has nothing missing, we can guarantee that something in the middle is inserting segments.  Then the acks that relate to those inserted segments are being passed through.  I would suggest an optimizer, but I cannot be sure.  The best case scenario would be to span directly in front of and directly behind the bluecoat with two captures.  If this is still happening, go to bluecoat support for recommendations.  You could remove the bluecoat and see if the issue goes away.  However, this is a less technical approach and could introduce new issues like bw utilization, etc.  The bottom line is that I suspect there is something going on and that optimizer is my first guess.  However, that is only a guess.

              • 4. Re: TCP ACKed Lost Segment.
                CISCOCCIE

                Hi Paul,

                 

                Thanks for your prompt reply.

                 

                Any comments on beneath links on wireshark .

                 

                http://blogs.technet.com/b/nettracer/archive/2010/10/05/bogus-ip-packets-and-wireshark.aspx

                 

                Thanks

                • 5. Re: TCP ACKed Lost Segment.
                  Paul Stewart  -  CCIE Security

                  That definitely looks a lot like what you are seeing.  I'd move the capture process to an independant host to eliminate any of the nic accelleration features from messing with your capture.  You could also try the fix they suggested.  Additionally, there are ways to disable the hw accelleration.  Bottom line is there is definitely something missing from your capture.  We need to get it to the point that it is an accurate representation of the wire.  An independant host can can do this on a span port.  If this is the case, we really don't yet have anything to go on for the root of the problem.