Ahmed Mohamed’s Reviews > AI Engineering: Building Applications with Foundation Models > Status Update

Autoregressive models (GPT) win over Masked ones (BERT) because they learn like a storyteller, predicting the next word based only on the past. By not "cheating" with a view of the whole sentence, they actually learn how to generate text from scratch rather than just filling in blanks.