@Mtbdog and I talked about this not too long ago.
Its tough.... because when riding with faster riders and they have to wait for you, you do feel bad and start neglecting how you really feel.
I think most guys... as long as they understand from the getgo that you are a slower rider, don't mind waiting for you and if you need to take a 5, they will be accommodating. Sometimes they will push... to help you become a better rider but they wont drop you. At least I have never experienced that.
@liong71er loves riding with me and he waits a ton! But truth be told... I think it has more to do with my magnetic personality