The time they ask directly, and the Black slave tells them directly: its not just because they're white, its not generally because they're masters, but because they specifically sold her children.
And in that conversation, he sees her not as Black, not as slave, not as woman, but as mom, as parent - as he himself is a father. He sees her as same to himself. And he can't deal, and has the freedom to leave.