“I do not think that a man's rise to power is necessarily the climax of his life or that his loss of office should be equated with his fall.”