Evolution/Devolution discussion

I was watching a show on NOVA called 'Decoding the Neanderthal'. Up until recently, it was thought that Neanderthal had died out about 30,000 years ago, a hominid species separate from modern man. It has only been in the past few years that DNA studies have revealed that modern man has in fact inherited a small amount of Neanderthal DNA, about 1-5%. This means that modern man and Neaderthal interbred, contrary to earlier beliefs that modern man might have killed off Neanderthal and had no initmate contact. Now, I have no problem being part Neanderthal. Never did. Here's my point of disagreement with the experts, of which I am not one. The theory goes that Neanderthal ancestors left Africa about 800,000 years ago. Modern man followed about 40,000 years ago. Ten thousand years later, Neanderthal was gone, bred out of existence, their genetic structure, what was valuable, maintained down to this day. What this theory leaves out is where did modern man in Africa come from? Did they skip the Neanderthal phase? I doubt it. And why would modern man have evolved in one location, Africa, and not elsewhere, Asia, the Middle east, Europe etc? It seems more likely to me that human migration was going on all of the time, a constant back and forth, and that interbreeding was going on all the time, so that the slow disappearance of Neanderthal into modern man was going on from the outset, hundreds of thousands of years ago, rather than only in that time span of 40,000 to 30,000 years. What do you think?