A few thoughts, in no particular order:
In order to create power in your swing, you need to drive your body rotation and maintain your extension and stability. It's going to put a lot of torque on your lower back unless you either leak power somewhere or control your acceleration and deceleration smoothly. It makes sense that having any more weight involved in your moving parts (ie your entire upper body) will increase this stress, especially if it's far from your center of rotation. Seems like muscle-heavy arms are a questionable idea for speed, no matter how strong they are. Some of your muscles in the shoulder and wrists are useful for keeping the club supported and on plane, as well as controlling the face and path, but they're not the primary power source IMO. If you're going to have bulky arms, you'd better have a very strong and resilient core and back to prevent injury from the extra stresses in your swing. If you're built bigger naturally, your muscles are designed to handle the load.
Look at the iron byron; it has a totally rigid left arm and no right arm because it doesn't need the extra support to handle the club. The wrist is a bit overpowered, but the only other spot that moves is the shoulder plane. So as long as you can rotate your body and keep your left arm straight, your arms themselves don't contribute much outside of staying stable and guiding the club. The legs are also about stability and don't contribute much to rotation, but they do allow a weight shift or a thrust to widen the arc at the bottom. This facilitates clean ballstriking with the correct angles, but not necessarily extra rotational power. So the wrists, which you can't really bulk up because they're hard to isolate and easy to injure, and the core are the two obvious choices for getting more power. I'd say the wrists need work if yours are weaker than the norm, but otherwise stick to core and flexibility/stability exercises.
When I'm getting my best power, I feel like I'm turning almost "mechanically" around my spine rather than throwing with my hands or twisting my shoulders open, I just need to keep my arc wide and keep my spine and head centered and follow through with my shoulders. I think almost of Stricker and how weak his swing looks from afar. It's very efficient and smooth most of the time, and I have some extra speed if I desire. Just need to keep connected to ensure a good path and let my setup do the rest. I'd say my right asscheek and my back muscles around the level of my lowest ribs are the biggest source of conscious effort in my swing. My arms are more about setting my wrists and plane on the backswing, then just keeping the club and hands caught up to the body rotation.
I'm about 6 feet and 150 pounds soaking wet, my arms are relatively lean but I have strong hands and forearms for my size. They give me a firm grip and that gives me the ability to prevent the clubface from getting goofy at high speeds, but all I have to do is keep them extended. I also still have very strong legs from TKD in past years; my calf muscles are still way bigger than my dad's and my brother, who both have chicken legs despite my dad being 80lbs heavier than myself and much stronger in his upper body. That gives me some stability, in fact I have an unusually low center of gravity for my height, so I don't have to throw away my angles to keep in balance. Helped me kick people in the face in plenty of competitions without losing my balance and I could basically counterbalance my torso with one leg for a whole match because I had such uneven musculature.
I go into the swing with the mindset of a much stronger guy, that the club feels light and controllable in my grip and I don't need to hit it too hard or force it to fly a certain way; I just try to control it through my chosen path perfectly without needing to compensate. Trying to be a skinny kid who uses every bit of flexibility to get enough length to compete with stronger guys is exactly the wrong mindset. Trying to bulk up to let you compete with those guys is wrong too. Taking the mindset of being Superman trying to precisely control his distance and form is a better way to play golf. Superman is not curious about his limits, or trying to show off. He knows he has enough strength, he just wants to hit the ball on the green and not create a volcanic eruption with his divot. So he uses exactly the right amount of power, no more than necessary.
Oh, and clubhead speed is useless without efficiency. Being brutally honest, I can hit a 5 iron longer on average than I used to get with a 120mph driver because I hit it so bad. I'd deloft it, come in off plane, and the ball was way too close and too far back. This basically either thinned it or drove the ball really hard but with the loft pointing low and left, resulting in a smothered hook. Think like using a -2 degree driver with a 6 degree closed face with a normal swing. With proper alignments I can easily get a true launch and powerful high flight with any club. Though your driver might say 8 degrees I've learned there's no such thing as a set loft on a driver (plus many are dead wrong), dynamic loft matters a lot more, as does the delivered face angle and angle of attack, and face bulge and roll. Hitting a 12 degree off the lower face, where it comes off weak and spinning, will go a lot lower and shorter than an 8 degree off the high part of the sweet spot, where it might play more like 10 degrees but spin less. Plus you have the fact the lie angle can make the loft point crooked in certain positions, or mess with your eye depending whether you sole or hover the club.
Not sure what my chs is anymore, but I'd bet it's nearly as fast now even though it doesn't feel like it. I carry my 7 iron about 175 yards on average now. Most of my yardages seem well in line with the 116-120mph range.
By the way, remember that trying to outdrive someone you can't outdrive is the absolute stupidest thing you can do. Stay out of the pissing contest mentality if you're playing or practicing seriously. Even the pros try to do it all the time, and often suffer defeat in match play because they try to hit big aggressive shots that backfire. My brother was doing this to me, as he is older and played sports longer; on a par 3 course he'd try and hit the same clubs as me from the same tees. Trouble is, I'm about 30 strokes better than him and a hell of a lot longer because he's never been a serious player and golf isn't a forgiving sport. He had trouble believing I was consistently that much longer, always electing to hit his 9i 150 and other nonsense. After about 10 holes he wised up and moved up to the white tees, and started just hitting his 7 iron on every shot and actually did better. My friend Nick keeps trying to do this as well. Not only is he 6 inches shorter and has short arms, his swing isn't the best either. Every hole it goes the same:
Nick: OK what'd you hit?
Me: It's about 158, I hit a hard 9 to get to the front half.
Nick: OK I'll hit an 8 because you're a teensy bit longer.
(Hits shot, thin push fade, ends up rolling forward, 15 yards short of the green)
Nick: Well I sliced it a bit, stood up. How do you hit that high pitch shot again?
Me: (Shakes my head, trying to remember if there was ever a time I've seen him miss long.)
Good luck out there.