There is Nothing wrong going directly for CCIE-V from a scratch, the only question is how fast you can bring yourself to the speed and depth of the CCIE level.
Some have done it quite well, with some good hardword and dedication, and some have gone through the path of CCVP and CCNA-V, either way my best wishes.
Also, please be tuned to the NetPro as it really offers valuable input to your CCIE-V voyage.
There's nothing wrong with doing it that way. I typically suggest that people do go throug the NA and VP first, just to develop a good core understanding. But if you've been doing VoIP and networking for a long time, then go for it.
Generally I don't think there are any problems with going straight for CCIE if you feel you have the necessary skills and knowledge. Determining what level you are at currently may be a more difficult process without the other levels to guide you. I would suggest that people following the full program CCNA-CCVP-CCIE probably have a very good idea of where their skill sets are, because they are regularly assessed. Without knowing your personal situation, I would suggest that perhaps a bootcamp or some mock labs would be a good idea to determine what your level of skill is. That way, you can get some sort of indication of the level of knowledge CCIE certification requires.