I agree with you, at least in regards to myself and playing. I can't really speak much for others. I assume that some people who are playing golf very often can notice more drastic differences.
That being said, I do notice some differences here and there and sound and feel definitely have some say in which ball I end up liking. But that seems more psychological than physical.
I've only been playing golf regularly for a year now and I'll play almost any ball since I often lose balls. However, since I've been able to get out once a week for a while now, I am down to losing only about 2-3 balls from losing 10-14 per round. Which feels huge for me and so I've begun to narrow down the balls I choose to play with.
I've gone through a box of Vice Pro Plus, Vice Pro, Sugar, Titleist DT Trufeel and Kirklands. Bought a bucket of 350 used balls which had a majority of Srixon Soft Feels in there. I decided to give the Soft Feels a try and have really been enjoying them quite a bit. Only yesterday did I hit a Pro V1 for the first time and it felt quite nice. However, that all being said, I have to admit, I've played my best overall with Vice Pro Plus balls. I get the most distance from these balls and am able to hit them quite well. I am not sure on green side spin/grip, I think I'd prefer them to check up a bit more. But I believe one really big factor for me with these balls is that I usually club down when hitting them because I gain so much distance off of them and I also tend to bring my swing speed down which causes me to have a more accurate swing and ball strike. As opposed to trying to whack the snot out of the ball each time, which ends up in a slice 75% of the time.
I do love testing out new balls though, sound and feel is always fun to experience.