If anything, golf is easier now than it was 25 years ago because of equipment advances. When I started it was with forged blades because that is all there was. Guys who feel that it's too hard with their SGI shovels ought to give blades a try for one round just to get a perspective. The first couple of popularity spurts in the game happened in spite of the difficulty of playing with blades, so I really don't see that as a major factor in the decline. Issues today are economics and time management. The economics factor is self explanatory - there is only so much money to go around, and golf can be expensive. Time management is just as key.
For whatever reason, everyone is in too much of a hurry to do everything possible in the least amount of time. That may in part be rooted in economic issues too, with people working longer hours to achieve their financial goals than they did 25 or 30 years ago, but those goals seem to be set higher than they once were, and are harder to reach. We want to do everything at once, so we race around popping 5 Hour Energy shots so that we only have to sleep 4 hours a night, and still can't find the time to do everything that we feel we should.
People who started in the game during the Tiger boom, or in some cases earlier, had to reassess priorities upon getting married and having a couple of kids. Golf usually loses out - just witness the number of people who join here and the first thing they say is that they played a lot when they were younger, but had to mostly give up the game at some point after college when work, wife and family took up too much time. Now the kids are gone, or nearly so, and they have the disposable income and time to try the game again.
The Sirens never stopped singing, and now you can remove the ear plugs and heed the call again.