Xianshun Chen’s Reviews > Getting Started with Google BERT: Build and train state-of-the-art natural language processing models using BERT > Status Update

Completed chapter 9. Learnt about VideoBERT (how it was trained using cloze task and linguistic-visual-alignment) and BERT (transformer with encoder and decoder, where training is done by encoder takes corrupted text and decoder output the reconstructed text with reconstruction loss against original text), learn about ktrain and bert-as-a-service

Completed chapter 8, learnt about Sentence-BERT (pretrained BERT fine-tuned for computing sentence representation using NLI datasets) as well as XLM-SBERT (teacher model monolinugal pretrained SBERT; student model XLM-R; loss using MSE between source/target sentences of the student model and the source sentence of the teacher model using KD), as well as why max and mean pools, ClinicalBERT, BioBERT

Understand M-BERT (pretrained with WikiData on 104 langs via MLM; zero-shot via topological and language similarity instead of token overlap; work with code-switched text but not transliterated text), XLM (pretrained with MLM + TLM; TLM pretraining with parallel data set with pair language embedding and positional embedding), XLM-R (pretrained with CC and MLM with sentence-piece) as well as monolingual BERT

Done with chap 6, learnt about BERTSUM and how to change token and interval segment encoding for BERTSUM to generate sentence representation using [CLS] at the beginning of each sentence. Also learnt the classifier, transformer, and LSTM based summarization layer as well as the encoder-decoder transformer using pre-trained BERTSUM as encoder and ROUGE-N and ROUGH-L scores for automated evaluation of summarization

Done with chap 5, learnt about DistillBERT (distillation loss + student loss + cosine embedding loss; dynamic masking with larger batch size and MLM task on BERT-base pre-trained teacher) TinyBERT (knowledge transfer from embedding, transformer and prediction layers; use MSE with weight transformation for attention matrix layer and embedding as well as CE for prediction layer) and any (MSE with logit + student loss)

Learnt various BERT models:
* ALBERT (1. cross-layer parameter sharing; 2. factorized embedding parameterization; 3. replace NSP with SOP for pretraining)
* ROBERTa (1. remove NSP; 2. replace static with dynamic masking for MLM; 3. train with large batch size of 8000; 4. BBPE with vocab size of 500K)
* ELECTRA (1. generator train on MLM and used to generate tokens for masked tokens; 2. BERT discriminator )
* SpanBERT

learn fine tuning of BERT in various tasks (classification and NLI using BertForSequenceClassification, question and answering using BertForQuestionAnswering with dotproduct and softmax layer, NER with feedforward and softmax layers, BertModel for extracting [CLS] embedding, final layer token embeddings, token hidden states for all layers, TrainingArguments, Trainer, nlp.load_dataset for fine-tuning pre-trained BERT)

done with chapter one, on the plus side about the masked multi-head attention, on the minus side, not clear about whether the linear layer on top of the decoder block before the final softmax, is a flattened dense layer or not (my guess is yes)