I get what you're saying but it seems overly simplified. There must be some technique involved or you could swing superfast and miss it completely or present the hosel to the ball. Also, it seems to me that a better physique is going to help promote a faster swing than a "poor" physique, but then maybe good technique can overcome a poor physique. Do all golf balls fly the same distance given the same swing speed or are the differences too small to matter?
Personally, it sometimes "feels" as if I've given everything I had to the swing, struck the ball well, and still not think it's gone as far as it should have gone for the amount of effort I "feel" I put into the swing. Which leads me think (perhaps wrongly) that if my technique were better, I might get more out of the same swing.
On the same note, when I watched the fellow yesterday, his swing didn't "appear" any faster than mine but it certainly looked more fluid than mine "feels". It also seemed like he might be getting his peak speed at the ball (or very near to it). Whereas, I may be delivering my peak speed too early or too late and again, that seems like a technique thing more than simply a swing speed thing.