I almost always wear sun glasses when playing golf.
I pretty much don't to anything outdoors without my sun glasses.
I don't think I would survive in Florida with my shades... Quality sun glasses are a must IMO, and polarized lenses are worth the extra cost. No gas station sun glasses for me... Oakley and ray-ban all the way. I have found that quality lenses eliminate basically all distortion that may affect putting.
If I do take my glasses off, they go on top of my hat,and rest on the bill, not across the back where I am sure they would fall off.