How to use ModernBERT with the AutoModelForQuestionAnswering class?
I wanted to fine tune the model on the SQUAD dataset. But currently AutoModelForQuestionAnswering does not work with ModernBERT. Is there a workaround for the moment where I can train ModernBERT for QA tasks given a context?
Hey! This is planned, but indeed missing at the moment.
cc @NohTow @bwarner @wgpubs @tomaarsen , we should ensure our January update includes the missing heads (QA, MultipleChoice)
Thank you very much for ModernBERT!
@bclavie It will be amazing to have TokenClassification also. : )
@bclavie @NohTow @bwarner @wgpubs @tomaarsen Can you check if this script implements the ModernBertForQuestionAnswering correctly, and if so, copy the script into your repo here and the large version? Then we would be able to use the models for that if we enable trust_remote_code 🙂