Three questions:
1) How many guys on the PGA tour are world class athletes in any other sport (e.g., make it to the Olympics), or could become professional athletes in another sport? Not many.
2) By contrast, how many NFL players are world class athletes in any other sport (e.g., make it to the Olympics), or could become professional athletes in another sport? Quite a few.
3) How good of a golfer would Bobby Jones have been if he had the mental outlook of, say, Babe Ruth?
Point: Sure the physical side is important than golf. But top golfers in general don't have the type of athletic ability that carries over to other sports. The mental side may be more important in golf than in other traditional sports like football, baseball, hockey, track and field, etc.