I agree it's luck. I don't think that's a bad thing or something you can change with the size of the hole.
As an athlete (myself included) it's hard not to buy into getting "hot". As a statistician I've seen a number of studies essentially proving that in lots of cases it's literally false that players get hot, and a player having made a couple shots in a row, or having hit a higher percentage of the past 10 shots than normal or whatever (to take basketball for example), actually has no relationship at all to the chance of making the next shot. You just are, say, a 40% shooter and sometimes you get lucky and hit a couple in a row and sometimes you get unlucky and miss 8 in a row, but really it's the same as if you were just flipping a coin that landed "hit" 40% of the time and if you've made 4 of your last 6 you still have a 40% chance of making the next shot. People are wired to over-interpret random patterns as meaningful.
I've never seen a good study of this for golf (if anyone has please post a link!). But I'd bet that the argument I outlined above is true of putting for solid players. Even the pros have a dispersion distribution (circular or elliptical, errors left to right and far and short) that's bigger than the hole for all but the shortest short putts. I'd bet that on most days at the course you've got the same chances of hitting exactly the putt you lined up, or missing a couple inches short but on line, or missing left a few inches, etc., as you do on other days. It's just some days you're unlucky and hit a few more of those less likely, worse misses. But I'd bet that for most good players having missed a putt or two or four worse than average doesn't affect your odds of various makes or misses on the next green.
So yes, I agree and think it does have to do with "luck", but making the hole bigger wouldn't change that. You'd still have the same elliptical dispersion distribution, it's just that more of it would be covered by the hole, so you'd have to have a more extremely unlucky day to miss tons of putts. So what you're really saying is that you want to drop more putts, not that you want to take the luck out of it!