I hear ya! I sometimes feel this way, but I have a friend who is a member at the harder course I play and he also has the same problem when playing my course!
But ya must take into consideration 2 things!
1, when you play a course regularly you develop familiarity, and therefore the course DOES become easier to YOU,
2, a handicap is to represent what your CAPABLE of scoring, NOT what you should score all the time, if you play to your handicap then you have had a good...