Without knowing the full story, it's probably best not to assume anything. We have enough of that going on already these days.
Dad and I went golfing every now and then after his heart attack and hip replacement. I knew how hard it was on him because I'd see him AFTER the round and notice how difficult it was for him to get out of bed for the next couple of days. I learned my lesson, and decided since he had too much pride to ask for the handicap flag, I'd ask for it and I would be his driver for the day, trying to do my part in making sure his round was as non-strenuous as possible.
If he hit it on the front part of the green, that's where I'd drop him off. He may have to walk to the back of the green after his putt, but I'd just pull the cart around to the back and pick him up there. It seemed to work just fine, until I started getting ugly looks because obviously I wasn't handicapped, and sometimes I'd be spotted alone in the cart as I went to pick him up wherever he was.
The story isn't mine, personally. It's from one of my golf buddies a while back.
The point is, if there is ONE story out there, then there's more.
You may witness PART of the story, for 10 minutes in this case, but that doesn't tell you the WHOLE story.
Without knowing the person, and without knowing his particular medical history, no fair judgements can be made.
Could they be jerks just taking advantage of a handicap flag? Oh yeah. Definitely. Just like the people who park in the spots when they have no business doing so. There are plenty of people out there who apparently don't feel any guilt when they take a handicapped spot away from someone who needs it. But it's probably best to do what you did in this case and NOT say anything or else you may find yourself listening to one of these stories and feeling like a jerk afterward.
Just my opinion.