I may be missing something but surely it's better to mark your balls on a green and then allow them to play onto the green and putt out. That way by the time you get to your balls on the green to mark them the single is reaching his approuch position. then you are only waiting on him finishing up the hole. When he goes to the next tee and gets ready to drive you are busy putting with your buddies. by the time you are ready to tee off the single is gone.
Otherwise you wait as the single pulls up to the tee, gets his driver un-covered and fishes out ball and Tee. Then you're watching his tee shot putting him under pressure. then you have to wait for him to walk past, play shot 2 and walk away. The chances of him finding some rough under pressure are far more than finding trouble with his 9-iron onto the green. I always get called through from the tee when the group infront is looking for a ball. Last week a 3-ball made me play behind them for the whole 18 holes and it took over 4 hours instead of under 3. I was finished each green before the 2nd guy tee'd off. there were 5+ holes clear infront. I don't understand the mentality that says they should not let me through.
I'm playing a fourball at the weekend and I fully expect to let the odd group through.
EDIT: I feel like you should be able to ask yourself what it would be like to be the guy behind that loses an hour of his day. Not being able to empathise is a classic sign of a madman.