To me, this is the crux of it. Not noise vs. silence, but rather, change. If it's totally silent, fine, and if it's totally loud, fine, but if somebody starts or stops during my backswing, that's distracting.
Same is true, I think, for free throw shooters. People behind the basket waving things the whole time and yelling doesn't faze them one bit. But if you could coordinate everybody to go from waving and screaming to dead stillness and silence immediately preceeding the shot - or vice versa? Then I think you'd start seeing some airballs. :)
The other part of it is expectance. If we're on 16 at Augusta we're not planning on somebody yelling during our shot, but at Phoenix we know it's coming. How many times have we seen Tiger Woods (or somebody else) glare or yell at a camera man for distracting him? Now, considering the fact that these tournaments are frequently played on what might as well be nature preserves, how many times have we seen somebody affected by a bird chirping during their swing? I'm going to say probably never.
[Side note to the Tiger thing: Don't know why I remember this so well but during the AT&T Pebble Beach tourney in 1995 I was standing behind the 9th tee watching Kirk Triplett (pretty sure it was him) hit a drive and then glare at a person standing by me with a crying baby. Seriously, somebody brought a baby to a golf tournament.]