“Even fervent abolitionists, viewing blacks as equal in rights but inferior socially and culturally, didn’t relish having freedmen come north to live beside them but wanted them to stay down south.”