I bought a security book years ago called the Hacker's hand book. It tells the story of a hacking case and shows what both sides are seeing and why. One of the points was that a hacker will often perform some kind of reconn to learn about the target he or she is planning to attack. Among those reconn tools were snmp scannners, traceroute, wireless sniffers and yes... port scanners.
It's a great read and really shows security on various levels and why you need to secure your network, not just in networking but keeping up to date with OS and applications patches as well. It taught me that just about anything can be a security threat and to be very careful of who is trusted. So, being the head security geek where I work (one of many hats I wear) my motto is to trust no one and verify, verify, verify before I do trust them aka give or open access.
At times it is incovenient, but when users understand that I am just looking out for their digital well being, they don't mind it as much.