Its because they look sloppy. Fair or not, thats just the way it is. Just like a lot of nicer courses have rules requiring collared shirts. If you cant stand to give up your beloved cargo shorts, go play your local muni. Im sure theyd be more than happy to take your money.
These are not the shorts I own (the pocket on mine is actually far less subtle), but I might just buy these and post pictures of myself wearing them at private clubs and exclusive resort golf courses throughout Hawaii. Care to take a bet on how many places turn me away?