You are not crazy and perhaps the "Fantastic Four Wood" should be resurrected! Good enough for Sarazen...
Truth is for the distance, there is not a great difference between a 3-wood (most are 15° +/-) & a 16°,16.5, 17° 4-wood, but for me accuracy is more important than distance and the 4-wood is perfect. My 3-wood, 5-wood days were too inconsistent. I just picked up the Ping G20: Nice ball flight, straight down the pipe and it looks to be a versatile weapon. Combine with my Tour Edge XCG5 22° 4-hybrid and I've got the "distance" thing covered. And if I don't hit the green, I'm mostly right in front & 3 Cleveland wedges 51°, 57°, 64° give me lots of options for getting up and down. My irons are Cleveland TA5 5-PW. Yep, that's 13 clubs. All I really need. I've been through the 2, 3, 4 iron thing, 3/5/7 wood thing, 16°/19°/22° driving irons, and finally settled on this configuration.
I like your setup. What driver, 4-wood, and hybrid are you using?