I think it's possible. I have some prescription sunglasses I'll wear while golfing sometimes. I will sometimes have trouble with ball contact while wearing them. I think it's because my brain thinks the ball is in a slightly different place because of the vision correction. So if your hat is somehow impeding your vision, then maybe.
Off topic but ...
Coincidentally (or not), I got a shingles vaccine Friday after putting it off or forgetting for a good while. It is nothing to mess with, good luck. I learned of this disease from my little brother getting it. It was not easy and it never fully goes away (to my understanding). It is everywhere, non-discriminating it seems.
Hang in there; it gets better.