Celibacy

Ok if a man says he wants to be celibate by choce, why is it always harder for the women to really respect a man for that. Like "There must be somthing wrong with him because he won't sleep with me!" Or here is a good one. "He must be gay!" Men who would like to practice celibacy must do so in private most of the time just to avoid unwant scrutiny.
Well, what do you think, am I right?