I think a good portion of society does hold religious figures to higher standards, and I can understand why they do so. Perhaps it is their role and function to a "world" larger than the majority of people, as well as the notion that they don't have a job; they have a social responsibility to uphold ethical standards because they represent something more prolific than what we may consider more mundane occupations or roles in society.
This being said, I don't consider myself to be a part of this group of people. When I conceptualize the role or function of a religious leader, it is no more substantial or crucial to the order of society than any other job. I think that holding one's self to social and ethical responsibility is something everyone should do. To trivialize or overly exaggerate one's responsibility to society is to create an imbalance of power as well as create systemic blame and scapegoating. Since I am a teacher, I see this far too often. Teachers are sometimes bestowed (at times unwillingly) with the responsibility of having to teach everything under the sun...and to teach it well. But if we all did our part to set good examples for kids - whether at home or in any other capacity - we would all have a social responsibility, and it would not fall on the shoulders of one solitary group.
So I guess your question could be answered with another question: if we do hold men (women, too?) of the cloth to a higher standard, what does that say about everyone else? Do they have less responsibility? And what are the implications of this dichotomy? What type of people come out of a society that conceives of moral responsibility in this way?