I think for the most part the majority of golfers know what the basic rules are and will abide by them. I think there are very few people who actually know the rule book enough to say they follow every one of them to the letter. Probably 90% or more of the golfers out there simply don't care about all of them, and that is fine too. Golf is supposed to be fun. Go out and hit the little white ball around and try and get it in the hole. Do it however you want. If you are playing for stakes of some sort then everyone needs to be on the same page, but if not, who cares what someone does or what they say their score is? I certainly don't. I have been paired up with many random strangers and have a good chuckle everytime I see them doing something wrong, but that is all it is. I don't know them, I don't care what they score or how they got there.
To the OP, I understand why you did your bet that way. It is frustrating when you have people wanting to gamble, but don't have the game to do it. I had a few friends that were not a good as I was who wanted to bet, but I don't do strokes - at all. So I would play from the tips and play by the strictest of rules. They got to play from a shorter tee box and could play the ball up, and take a few other small liberties we agreed on. It evened out the match quite a bit and we had fun with it.
From the friend perspective however, the better way to handle it may have been to poke fun at them when you see a rule break rather than call them out on it, at least to start. Instead of calling a penalty on the first tee box, just wait until the guy is ready to hit then say "wait a sec, you know it is a penalty to hit in front of the tee markers, right?" Help educate them a little with humor rather than just calling them out. They are still your friends, you don't want to be the a-hole of the group. If a guy hits OB, just toss a ball at his feet and tell him to try again from that spot. After a few times, they'll get the idea, and they won't hate you for it. They might learn something too. Plus, once you get in their head about all the rules they are breaking, that will net you way more strokes then calling penalties ever would.