I believe the key to this topic is consistency. There are plenty of 20+ handicappers who can hit 300yd plus drives, I'm sure many long drivers have high handicaps simply because their focus is on the driver itself, not to say they're incapable of reducing their handicap with time and effort on other aspects of the game. However, many amateurs who have the capability and swing speed to produce a 300 yard drive do lack consistency in terms of keeping the ball in play whether it be lack of experience, practice, etc. But I have personally witnessed 20+ handicappers hit the ball to or past the 300 yard mark, but again taking a 20 ball average might present a different average versus what they know they are capable of.
I agree sometimes reality hurts especially when it comes to distance off the tee. There are so many variables to consider, to include temperature, wind, hole layout(doglegs) that can make it seem like a person hits it further than they do on a consistent basis. The biggest thing is that a 300 yard drive ob is not as effective as a 250 yard fairway finder.
Yeah, size has little to do with it. Swing technique, flexibility, width of arc, length of arc, centerdness of contact and ultimately hitting the ball with a square club-face relative to the swing path all effect how well any swing speed produces a quality shot.