Relax! I used to have the same problem playing with people more advanced than me. Don't worry so much about them and play your own game. Golf is so much about confidence it's ridiculous!
My dad and I got stuck with a high handicapper last week (he said he usually shot in 120's) and he only shot a 105 playing with us cause "we helped him focus more". I don't know how that works but he wasn't worried about how my dad and I were playing. Guess it's just a mind set.