If you look at the stats, the scratch guy is making 4-5 more pars and 1 or 2 more birdies than the guy shooting 80 on average. He is also having a lot fewer doubles. Birdies to some extend are a result of hitting another 4 GIR. If you have 4 more 15-20 ft putts, a couple go in.
Birdies are also worth more to the higher scoring player. Lets compare a guy shooting 81 and a guy shooting 72 (assume those are ~10 and 0 respectivily). The 81 gets an extra .5 strokes per hole. So every birdie saves 1 stroke while a bogey only costs .5. If that guy gets a birdie and a double, he is right on pace. The 72 guy on the other when he makes a double, he needs to get 2 birdies to make up for it. So the high scoring guy can negate a double with a birdie, while the scratch guy needs eagle for the same effect.
One other thing, I don't know if anyone else has thought about why it's misleading but my thought is that it has to do with the difference between pars and birdies. Hear me out. :) It's pretty easy to make pars after you get semi-decent at golf, you can hack it around and make pars. Birdies on the other hand are much much more difficult to consistently pull off.
I feel like this is a chasm between the birdie and the par that really creates that separation as you approach scratch. You need to be technically much more perfect to pull out even a handful of birdies (maybe just 3 a round) and this is where I think it really breaks down for people that are 5-6 handicaps.
If nothing else consider this, at some point to shoot even par you have to be just as likely to make a birdie as you are a bogey. (think about that, it's crazy, and I suppose just similarly make two birdies for every double). It's not necessarily about more fairways and greens. It's about a huge leap in the level of playing ability on several holes to jump that chasm between par and birdie. If you are technically very skilled and can hit a lot of shots to 10-12 feet and sink those putts than you can offset your mistakes and be scratch. The majority of people are not. You are not close with your 12 pars and 6 bogeys, not close at all. The enormous gap in the skill required to convert 3 of those bogies to birdies is ENORMOUS. It is not simply 6 strokes.