Just to clear up a misconception, your handicap is not based off your career rounds. While it does pick your 10 best from the recent times, it also gives a "Bonus for Excellence" which makes it a bit harder to reach the lower handicaps than the higher ones from a mathematical standpoint. Since they take 96% of your course differentials (adjusted scores after course rating and slope) to create your handicap, a high handicapper will get a larger benefit from that. This makes it so that the low handicapped player is slightly more likely to win thanks to not being given as much of a lower handicap than he would otherwise need.
The other part of the handicap equation is slope and course rating. Unless your course has a higher slope or course rating than average from the tees you were playing at, it's likely that a score of +4 would actually equal a +5 or +6 handicap since the course would be easier than the set average to play. Conversely, a +4 at a very difficult course and tees could give you a handicap of +2.