If the grip is too thick the club won't feel secure in your fingers so I can see how that can be a factor.
Wait, you have oversized grips but you grip down to where the grip is thinner? Why not just play standard grips and grip the club at the end?
The only time I choke that far down with a full swing motion is when I'm hitting a knockdown or if I'm hitting out of trouble and trying to hit a big sweeping draw and I need a little more control over the club.
Yea, it's mostly about comfort. I have meaty hands with short fingers, so I play standard grips. I'd probably be more comfortable playing an undersize grip than I would a midsize grip. But if you get too far out of the "proper" range for you, it's probably going to affect how you hold the club.