Drive for show, putt for dough!!
It is rare for a golfer to swing up near 100% off every tee. You swing will gradually get faster as the round progresses, so you shouldn't start off fast. Your "slower" swing speed should be a good start-of-round speed as you seem to be in control and hitting out of the middle of the face. There is nothing wrong with letting rip occasionally on a hole where you have a wider fairway.
Try not to think of it as an average driving distance over the round, way each shot individually. On the holes with more room go for it and then you will build your confidence with the long ball and feel more confident on tighter holes. Nobody hits the same shot off every tee.