It would mess up your swing. If you have a fitted club, don't stray from it. Your fit can affect quality of contact (subject to the quality of your swing) and that's not something on which to be compromised at any level. This includes length, lie, sole grind, weighting, and shaft type; loft is important but it won't affect your contact.
On the topic of shaft flex, you can control the clubs fairly well if the shaft is too soft, but not as well as the proper fit. For example, I hit X flex in everything but I have hit my brother's stiff 7 iron on the occasional whim and I can hit it straight albeit not as far. However, I hate the feel and I can't control my ballflight too well because it feels so much less responsive for me. I used the stock shafts in my woods at first as well, but eventually switched the shafts for better fitting models and I'm glad I did, but I was still able to use them and improve until then. My shots are better behaved with my set, but my score isn't more than 2 or 3 strokes different from using my brother's set over 18 holes, I'd bet. In fact, I'd say losing half the clubs in my set would be about the same degree of difficulty, but neither is as big a deal as one would think. I'd also like to point out that I practice a lot with my set so I'm very used to them. If you're not a good shotmaker yet with any set of clubs, or if you don't have a solid or repeatable swing, it won't make as much of a difference.
There's no reason to play the wrong equipment for you except money, and even then you can piece together a great set without paying several grand. If you're just starting out, I suggest you not worry about a full set and try to get a usable if not perfect fit, perhaps from used clubs.