# Pre-train an efficient transformer language model for natual language understanding

## Data

We use [wiki103](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip) data as example, which is publicly available. It contains three text files, `train.txt`, `valid.txt` and `text.txt`. We use `train.txt` to train the model and `valid.txt` to evalute the intermeidate checkpoints. We first need to run `prepara_data.py` to tokenize these text files. We concatenate all documents into a single text and split it into lines of tokens, while each line has at most 510 token (2 tokens are left to special tokens `[CLS]` and `[SEP]`). 

## Pre-training with Masked Language Modeling task

Run  `mlm.sh` to train a bert like model which uses MLM as the pre-training task. For example,

`mlm.sh bert-base` will train a bert base model which uses absolute position encoding

`mlm.sh deberta-base` will train a deberta base model which uses **Disentangled Attention**

## Pre-training with Replaced Token Detection task

Run `rtd.sh` to train a ELECTRA like model using RTD as the pre-training task. For example,

`rtd.sh deberta-v3-xsmall` will train a DeBERTaV3 XSmall model with 9M backbone parameters(12 layers, 256 hidden size) plus 32M embedding parameters(128k vocabulary size)

`rtd.sh deberta-v3-base` will train a DeBERTaV3 Base model with 81M backbone parameters(12 layers, 768 hidden size) plus 96M embedding parameters(128k vocabulary size)

`rtd.sh deberta-v3-large` will train a DeBERTaV3 Large model with 288M backbone parameters(24 layers, 1024 hidden size) plus 128M embedding parameters(128k vocabulary size)

## Continuously training with Replaced Token Detection

Run `rtd.sh deberta-v3-X-continue` to continuously train DeBERTaV3-X models with following checkpoints. 

Please check the script to specify initialization models of generator and discriminator.

|Model| Generator | Discriminator|
|-----|-----------|--------------|
|DeBERTa-V3-XSmall|[Hugging Face](https://huggingface.co/microsoft/deberta-v3-xsmall/resolve/main/pytorch_model.generator.bin)     |[Hugging Face](https://huggingface.co/microsoft/deberta-v3-xsmall/resolve/main/pytorch_model.bin) |
|DeBERTa-V3-Small|[Hugging Face](https://huggingface.co/microsoft/deberta-v3-small/resolve/main/pytorch_model.generator.bin)     |[Hugging Face](https://huggingface.co/microsoft/deberta-v3-small/resolve/main/pytorch_model.bin) |
|DeBERTa-V3-Large|[Hugging Face](https://huggingface.co/microsoft/deberta-v3-large/resolve/main/pytorch_model.generator.bin)     |[Hugging Face](https://huggingface.co/microsoft/deberta-v3-large/resolve/main/pytorch_model.bin) |

## Distributed training

To train with multiple node, you need to specify three environment variables,

`WORLD_SIZE` - Total nodes that are used for the training

`MASTER_ADDR` - The IP address or host name of the master node

`MASTER_PORT` - The port of the master node

`RANK` - The rank of current node

For example, to run train a model with 2 nodes,

- On **node0**, 
 ``` bash
 export WORLD_SIZE=2
 export MASTER_ADDR=node0
 export MASTER_PORT=7488
 export RANK=0
 ./rtd.sh deberta-v3-xsmall
 ```

- On **node1**, 
 ``` bash
 export WORLD_SIZE=2
 export MASTER_ADDR=node0
 export MASTER_PORT=7488
 export RANK=1
 ./rtd.sh deberta-v3-xsmall
 ```

## Model config options

- `embedding_sharing` The embedding sharing method
	- **GDES** Gradient Disentangled Embedding Sharing
	- **ES**  Embedding Sharing
	- **NES** No Embedding Sharing

- `relative_attention` Whether to used relative attention

- `pos_att_type` Relative position encoding type
	- P2C Postion to content attention
	- C2P Content to position attention