Exactly this, here are the details I've read so far;
- the OP was walking, "yahoo's" were riding.
- OP was playing from longer tees, yahoos were playing from shorter tees
- Yahoos were waiting on and hitting into OP
- OP decides to say nothing nor let them play through. At the last hole he hits their balls off the green to a location where they won't find them.
Aside from hitting into the group and risking injury of someone, I'd say the OP and his group didn't handle the situation very well.