I only play about once or twice a month. Used to get really annoyed by slower players and at one point thought about taking a yoga pose (sit down with legs behind head and hands in front) to subtly send them the message to speed up a bit.
The last few times I've played though I've brought a book or iPod to pass the time. Amazing how much you can get through while waiting. Helps to keep the aggravation level down as well.