These points will probably echo everyone else's, but here's my two cents.
As others have said, the cost is the biggest thing golf has going against it in terms of getting new people into it, or at least I think so. Even to just get into the sport I tried to find ways to keep it cost-effective, but it still ate up more funds than I would have preferred. After I first started playing, I got frustrated and gave up for a while not because the game is hard, but because it's often overly-expensive and I felt the sport, by and large, catered to the rich or upper-class, which I am thoroughly not. I also felt there was a strong inclination by golf in general to keep it that way. It wasn't until I got back into it and had more exposure that I found this wasn't totally the case, which leads me to my second point.
The idea that golf is a sport only for rich snobs needs to go, and society/the media/the entertainment industry must stop encouraging this stereotype. Embarrassing Story: the first time I went to the driving range I was terrified I would offend the golf gods or real golfers if I wasn't decked out as noble as possible. So I got a nice polo, some khakis, and a pair of golf shoes. I go to the range and find a solid percentage of people there are just wearing t-shirts and tennis/running shoes. I felt kind of silly afterwards for thinking I'd tick off everyone there if I dressed like that. Wholly average joes play golf. Everyone can. For some it will require more work to get in than others, but it's one of those things you CAN make happen, but it would be a lot easier if the cost would just simply go down.
Plus, other sports require far less commitment to get into, are just easier, and have much more exposure. Compare the requirements of getting into and good at golf to going down to a toy store or sports shop and buying a basketball for ten bucks. You can play with a basketball just about anywhere; playing golf anywhere, not so easily done. Once you have that basketball you don't have to pay a fee to go play with it. More youths are into basketball and thus will pick up and play a lot, so it's easier for them to get together doing that. Sports media pushes people like LeBron James or Kobe Bryant onto viewers and kids dream of being like them when they grow up--it's easier for them do that than to dream of being Mickelson or Hogan or even Tiger. They just don't connect with them, partly because of that "only for upper-class" mentality.
I think there are a few good things that golf has going for it, though. Take Rickie Fowler. He's bringing in new audiences to the game with his style and that certain appeal he has to younger viewers. And if he starts winning more often there'll be more exposure to that crowd. Plus, they're always coming out with golf video games that get better with each iteration. I mean, real golf is different from Mario hitting a five iron only to have his ball be eaten by a piranha plant, but it's still fun. Every little bit helps.