I'm late to the thread, so my apologies if this has been covered...
I voted "too difficult", but I really think it's a combination of too difficult, too expensive, and takes too long. If the game was easier, and beginners were quicker to hit those satisfying shots that make us addicts, then I could see them finding a way to justify the cost and the time away from family. But, golf is pretty frustrating to learn, and the newbie usually has to spend time really learning the game to get to the point where the financial and time costs are justified by the enjoyment returned...
For example, I work in a field (medicine, anesthesia to be precise) that traditionally has a large number of golfers. I can't think of another golfer under the age of 50 in my work circle of colleagues (an OR staff of about 50 anesthesiologists and around 100 surgeons, plus many nurses and techs that are potential golfers). Plenty of older guys golf, but most of the younger guys are into cycling, tennis, running, adult basketball or soccer leagues. These are all sports where they can quickly do their thing after work or on a weekend morning, then still be home to get the kids/wife to whatever activity it is that day. I can think of maybe 3 or 4 guys that have tried to pick up the game, only to quit after their first year. The reason? They had more fun doing other things that they could excel at with less practice. They had fun golfing, but not ENOUGH fun to justify the time, and they didn't see themselves having the time to practice and get to the point where every round was worth finding a way to make the free time.
Similarly, if I make the argument that joining a club would reduce the time obstacle (generally faster rounds vs. public courses, and free access to practice facilities), the money obstacle takes over. Not enough disposable income to justify anywhere from $6k to $15k/year (at least in our area), on a sport that they aren't fully committed to (yet).
Oddly, I know about 6 guys, all under 40, that grew up golfing (via parents at a country club), and at one point in their life were pretty good at it. They are now successful in their careers, have the disposable income, enjoy the game, but just don't have the time to play regularly. Wives, kids, career, etc...
I'm fortunate enough to have some financial flexibility and a supportive wife, but I see a day in the next few years, as the kids' activities increase, that my rounds per year will go WAY down. Then, if I'm only playing 5-10x per year, giving it up completely isn't that far off. Golf is incredibly fun when you're playing well. Hacking your way through a case of balls over 6 hours? No thanks...