A long time ago, in a network far far away, PAT was in use. The old PAT router would track UDP and/or TCP port numbers with overload, to keep tabs on which sessions went with which inside devices.
Then along came IPSec, neither UDP or TCP, it is protocol #50. A really old PAT device, that only PATs on UDP or TCP would freak out, and say how do I track that, (no ports. New routers can track this with PAT just fine), and the ESP would fail due to a lack of translation.
So, when 2 VPN NAT-T compatible endpoints notice that they are connected via NAT, they will err on the side of caution, and use UDP 4500 as a shim in front of the ESP header, so that a really old PAT device, can still track the sessions, due to the UDP shim there.
Thanks for your response. That helps me understand how it functions now the only remaining question I have is why was it working when I was behind one router/firewall device with NAT-T disabled but not when I was behind two?
Once I went behind to router/firewalls that were NAT'ing I was not able to carry traffic across the tunnel.
In addition to that experience, there was a laptop connected to a tethered smartphone that would surf the internet and connect to the vpn just fine but was unable to reach any network behind the firewall (ie. it couldn't carry traffic through the tunnel) until I enabled NAT-T.
If this is only needed for old devices...what are modern SO/** router/firewalls failing to carry traffic and also modern smartphones?
In my story, I only intended to explain the basic purpose of NAT-T in the context of IPSec.
Regrading your current scenario:
Are one or both of your firewalls an ASA or PIX with ver 8.x or higer?
Are both firewalls attempting/performing NAT?
Is the ipsec passthrough option configured one of the firewalls.
There are many variables.