I haven't seen that specific error message before but in my experience the ASDM is very sensitive to Java updates. Have you updated or changed your Java client recently?
You might want to try removing Java completely and reinstalling a slightly older version. That's worked for me in the past...
I know that message! Used to get it regular on SDM, Neils fix should work, if not, go into Java control panel and setup the following in the runtime parameters area (java tab, click view);
This adjusts the resources Java has available, incidently, I've not had any problems with the latest Java available on ASDM....but then again I don't use IPS all that much.