It sounds like you are my long lost twin. My story is virtually identical to yours. I was a low single digit in HS, then for about 15 years played about 3 rounds a year. I would go out and expect to shoot par, which of course never happened, and I would get upset and not play again for months.
A couple years ago I finally just told myself to enjoy being out there. Don't worry about shooting par, just relax and enjoy it. almost immediately my game improved some, playing to the mid to high 80's. I still felt that my swing was WAY out of whack, so I decided to start taping myself and try that. when I watched myself I thought it looked ok, but after I posted it to a site online where people could critique it, I actually got some pretty good advise that seemed to be working. By the end of the year I was shooting low to mid 80's and a couple rounds back in the 70's.
I have a ton more I need to work on this year, but I feel like I am moving in the right direction, and am looking forward to playing this year for the first time in many.
Brew