I live in Cape Town, and it would really be a shame to miss out because you've heard exaggerated stories about crime. Just use your common sense - don't walk around alone at at night, don't go to remote areas unless you're in a small group, etc. It is a beautiful city and I have always felt very safe here.
In terms of golf, make sure you get to De Zalze and Pearl Valley, both just a few minutes drive from Stellenbosch. Royal Cape (the oldest course in South Africa) is also great. Arabella (about an hour's drive from Cape Town) is one of the country's top courses and has an amazing 5-star hotel if you are looking for a bit of luxury. Further afield (four to five hours from Cape Town), is Fancourt and Pinnacle Point - two more of the country's top courses.
And you must go wine tasting if you are in Stellenbosch, and as someone else said, visit Ernie Els' winery.
Good luck and enjoy.