| ## A Lossless Syntax Tree Generator with Zero-shot Error Correction |
|
|
| - We follow [jam](https://huggingface.co/apcl/jam)'s pretraining procedure and use the same data to pretrain except we also use srcml to pretrain the models. |
| - In the finetuning stage, we finetune our models for 3 epochs. |
| - Our [GitHub repo](https://github.com/apcl-research/autorepair) contains the code for reproduction using the same [data](https://huggingface.co/datasets/apcl/autorepair). |
|
|
|
|
| ## Pretrained model parameters |
| | Hyperparameter | Description | Value | |
| | ----------- | ----------- |------------| |
| |e | embedding dimensions | 1024 | |
| |L | number of layers | 24 | |
| |h | attention heads | 16 | |
| |c | block size / context length | 256 | |
| |b | batch size | 4 | |
| |a | accumulation steps | 32 | |
| |r | learning rate | 3e-5 | |
| |y | weight decay | 1e-5 | |
| |iter | iterations | 570000 | |
|
|
| ## Model files |
|
|
| | Filename | Description | |
| | ------- | ------- | |
| |ckpt.pt|A model file for finetuning| |
| |ckpt_base.pt | A model file for generating syntax tree with the error correction in zero-shot setting| |
| |ckpt_finetune.pt | A model finetuned with the syntatic error dataset | |
|
|
| - Note that you can adjust the batch size and accumulation steps based on your GPU memory. But, the batch size * accumulation steps should be 128. |
| - If you finetune your models with multiple GPUs, you can turn down accumulation steps. For example, if you finetune with 2 GPUs, you will need to half the accumulation steps. |
|
|
|
|
