I play golf at my speed. I don't doddle, I don't spend ridiculous amounts of time searching for lost balls...I generally finish 9 holes in 2 hours ish walking.
However, I do have the common decency to recognize that people are playing faster than me. If I notice on the 4th hole that someone is waiting in the tee box behind me, and there has been no one behind me all morning, I can safely assume that their rate of play is faster than mine. I generally don't wait until the tee box to waive them past...I'll just give em the waive and step behind a tree and let them play through.
However, if I am waiting to pass the group in front of me, I won't waive the group behind be until I have passed...seems fair to me.