“A bigger hurdle, once we drill down into it, is the basic assumption that underlies the pipeline theory: that we are fair in the first place.”