MHO, it's all about time. Golf is a very time consuming activity, from the actual playing of a round to practice. With all of the demands a family and a job put on an individual, there is little time left for an activity like golf.
I was somewhat fortunate, my wife is a school teacher with summers off, so I was able to play golf in the summer time without putting much of a strain on family time. Plus I would find time when "other" activities were going on that I wasn't needed, like attending mass. I would get up early on Sunday mornings, have a 6:30am tee time, be home by 10:30am and they would just be getting home from church. But once school started, my golf was over, I would feel lucky to get in 2 rounds of golf after September 1st.
I also found alternate ways to practice, after I took my son to football practice I would practice sand shots in the school play ground area or while my daughter was at volleyball practice I would hit partial wedges in the school yard.. All that said, I still took time away from family and it was not easy making it work, so I can understand why people would not be drawn to the game.