When I first started playing golf, having played a couple of other sports and was good at both, I was more concerned about how bad I played than who I was playing in front of or with. If someone asked me to join them, back then I would usually tell them what I normally shot and ask, "Are you sure you want me to join?".
I hated playing bad golf, still do. I worked very hard at golf to get to where I was not ashamed to play with better players. I actually enjoy playing golf with people who are way better than me. But, way back when, I could not afford lessons but loved to play. I learned a lot by watching better players. I would pick up things they did and then go to the range and practice them to see if it worked for me.
I have never been a slow player. Back when, I could shoot a 100 and still do it in 4 hours or less if not being held up. So, never was worried about holding people us playing in front of them or with them if I was shooting a monster number.