MLRS/BERTu_SentiMalti

BERTu (SentiMalti)

This model is a fine-tuned version of MLRS/BERTu on SentiMalti. It achieves the following results on the test set:

• Loss: 1.9082
• F1: 0.6907

Intended uses & limitations

The model is fine-tuned on a specific task and it should be used on the same or similar task. Any limitations present in the base model are inherited.

Training procedure

The model was fine-tuned using a customised script.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 32
• seed: 3
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: inverse_sqrt
• lr_scheduler_warmup_ratio: 0.005
• num_epochs: 200.0
• early_stopping_patience: 20

Training results

Training Loss	Epoch	Step	Validation Loss	F1
0.9723	1.0	165	0.7271	0.6184
0.6386	2.0	330	0.6306	0.6998
0.3595	3.0	495	0.7487	0.6893
0.172	4.0	660	1.0045	0.7018
0.0827	5.0	825	1.2351	0.6827
0.035	6.0	990	1.3912	0.7184
0.0195	7.0	1155	1.5390	0.7162
0.0078	8.0	1320	1.7268	0.7061
0.0038	9.0	1485	1.7315	0.7188
0.0027	10.0	1650	1.7508	0.7153
0.0036	11.0	1815	1.8214	0.7069
0.0004	12.0	1980	1.8759	0.7163
0.0004	13.0	2145	1.9154	0.7058
0.0004	14.0	2310	1.9490	0.7072
0.0002	15.0	2475	1.9955	0.7116
0.0002	16.0	2640	2.0237	0.7116
0.0003	17.0	2805	2.0499	0.6976
0.0002	18.0	2970	2.0582	0.7071
0.0013	19.0	3135	2.1837	0.6997
0.0003	20.0	3300	2.1333	0.7078
0.0002	21.0	3465	2.1226	0.7131
0.0021	22.0	3630	2.1613	0.7085
0.0001	23.0	3795	2.2053	0.7116
0.0001	24.0	3960	2.2182	0.7154
0.0013	25.0	4125	2.1939	0.7119
0.0001	26.0	4290	2.2181	0.7134
0.0001	27.0	4455	2.2612	0.7185
0.0018	28.0	4620	2.2793	0.7120
0.0001	29.0	4785	2.3041	0.7123

Framework versions

• Transformers 4.51.1
• Pytorch 2.8.0+cu128
• Datasets 3.2.0
• Tokenizers 0.21.4

License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Permissions beyond the scope of this license may be available at https://mlrs.research.um.edu.mt/.

Citation

This work was first presented in SentiMalti: A Maltese Sentiment Analysis Dataset and Models. Cite it as follows:

@inproceedings{SentiMalti,
    title = "{S}enti{M}alti: A {M}altese Sentiment Analysis Dataset and Models",
    author = "Caruana, Ian and
      Vella, Matthew and
      Zammit, Fabio and
      Micallef, Kurt  and
      Claudia Borg",
    booktitle = {Proceedings of the Fifteenth Biennial Language Resources and Evaluation Conference (LREC)},
    month = may,
    year = "2026",
    address = "Palma, Mallorca, Spain",
    publisher = "European Language Resources Association",
}