As a "very" high handicapper, I can weigh in on this. When I was starting out, I had a pretty bad grip and it showed in the form of slices, mishits and wildly inconsistent shots. After fixing my grip (it still needs some work, but is decent according to some low handicap friends), my game has dramatically improved (no slices, mishits etc).
Having said that, even with a decent grip, I continue to struggle with distance, accuracy and consistency due to my poor swing. An improved swing may have a bigger impact on the game but I think improving the grip is much easier for beginners.