I've got an Audi A5 sportback, and really like it. I had an A3 as a loaner for a day (same size as S3/RS3), and while I was appreciative of having the loaner, I found that there were things in the A4/S4 and A5/S5 lines I like better. No volume knob other than on the steering wheel. The gear shifter is only a toggle switch. And felt a little too small for my taste, and not as "solid" as my A5. (Some say the A4/S4 sedans are more solid than the A5/S5 sportbacks.)
Perhaps more importantly, check the trunk/hatch space for your clubs. In my (previous) Infiniti sedan, I could fit two sets of clubs and two hand carts in the trunk. In my A5 sportback, there's not enough width in the back to fit my clubs sideways--only fits at an angle, and only one set and one handcart fit. (If I get a second player, I can drop the rear seat all the way and then there's plenty of room for a second set, etc.). I knew this going in, and before purchasing I drove to the dealer with my clubs to see how they fit. (Helped that he was a golfer too.) So check on the trunk width/size for your clubs on the S3/RS3 to make sure even one set fits in with cart, etc.
"Someday" maybe I'll move up to an S4 or S5. I didn't test-drive one, 'cause I knew I'd probably like it too much. And my A5 is just as fast as my Infiniti was, maybe a smidge faster.