kingabzpro/diffusiongemma_pubmedqa

DiffusionGemma PubMedQA

This model is a LoRA fine-tuned version of DiffusionGemma 26B-A4B IT for biomedical question answering on PubMedQA.

The model was fine-tuned to answer biomedical research questions using PubMed abstract context and return one of three labels:

yes / no / maybe

This model is intended for experimentation, benchmarking, and educational use. It is not intended for clinical decision-making or medical advice.

Model Details

Model Description

This model adapts DiffusionGemma to the PubMedQA task. Given a biomedical research question and supporting PubMed abstract context, the model predicts whether the answer is yes, no, or maybe.

• Developed by: kingabzpro
• Shared by: kingabzpro
• Model type: Diffusion language model with LoRA adapter
• Language(s): English
• License: Apache 2.0
• Fine-tuned from: unsloth/diffusiongemma-26B-A4B-it
• Training framework: Unsloth + Transformers
• Task: Biomedical question answering / text generation
• Dataset: qiaojin/PubMedQA

Model Sources

• Repository: kingabzpro/diffusiongemma_pubmedqa
• Base model: unsloth/diffusiongemma-26B-A4B-it
• Dataset: qiaojin/PubMedQA

Uses

Direct Use

This model can be used to answer PubMedQA-style biomedical research questions where the input includes:

1. A biomedical research question
2. Relevant abstract/context text
3. A request to answer with yes, no, or maybe

Example task format:

Answer the biomedical research question using only the context.

Context:
[PubMed abstract context]

Question:
[Biomedical research question]

Answer with only one word: yes, no, or maybe.

Downstream Use

This model may be useful for:

• Biomedical QA experiments
• PubMedQA-style benchmark testing
• Fine-tuning tutorials
• LoRA adapter experiments with DiffusionGemma
• Educational demos for medical-domain model adaptation

Out-of-Scope Use

This model should not be used for:

• Medical diagnosis
• Treatment recommendations
• Emergency medical advice
• Replacing a doctor, pharmacist, or clinical expert
• Patient-specific medical decisions
• High-stakes biomedical or healthcare deployment without further validation

The model is trained on a narrow benchmark-style task and may produce incorrect answers.

Bias, Risks, and Limitations

This model has several important limitations:

• It was fine-tuned on PubMedQA-style examples, not general medical conversations.
• The model predicts only yes, no, or maybe, so it may oversimplify complex biomedical findings.
• The evaluation set used in this experiment was small: 50 examples.
• The model may be sensitive to prompt format.
• The model may answer incorrectly if the context is incomplete, misleading, or unrelated.
• The training target was short, so the loss dropped quickly and may not reflect deep medical reasoning.
• The model should not be treated as medically reliable.

Recommendations

Users should:

• Use this model only for research and educational experiments.
• Always verify outputs against trusted biomedical sources.
• Avoid using the model for real clinical or patient-facing decisions.
• Run larger evaluations before drawing strong conclusions.
• Consider training on explanations, not only one-word labels, for a more meaningful medical QA setup.

How to Get Started with the Model

Install dependencies

pip install unsloth
pip install transformers datasets peft accelerate sentencepiece protobuf

Load the model

If this repository contains the LoRA adapter, load the base model first and then attach the adapter:

import copy
import torch
from peft import PeftModel
from unsloth import FastModel

base_model_name = "unsloth/diffusiongemma-26B-A4B-it"
adapter_name = "kingabzpro/diffusiongemma_pubmedqa"

model, tokenizer = FastModel.from_pretrained(
    model_name=base_model_name,
    dtype=torch.bfloat16,
    load_in_4bit=False,
)

model = PeftModel.from_pretrained(model, adapter_name)

processor = tokenizer
tok = processor.tokenizer if hasattr(processor, "tokenizer") else processor

dev = next(
    (p.device for p in model.parameters() if p.device.type != "meta"),
    torch.device("cuda"),
)

canvas_len = model.config.canvas_length

Run inference

def answer_question(prompt, steps=16):
    input_ids = processor.apply_chat_template(
        [{"role": "user", "content": prompt}],
        tokenize=True,
        add_generation_prompt=True,
        return_tensors="pt",
    ).to(dev)

    gen_config = copy.deepcopy(model.generation_config)
    gen_config.max_denoising_steps = steps
    gen_config.max_new_tokens = canvas_len

    model.eval()

    with torch.no_grad():
        output = model.generate(
            input_ids=input_ids,
            generation_config=gen_config,
        )

    generated = output.sequences[0, input_ids.shape[1]:]
    text = tok.decode(generated.tolist(), skip_special_tokens=True)

    return text.strip().lower()


prompt = """Answer the biomedical research question using only the context.

Context:
[Paste PubMed abstract context here]

Question:
[Paste biomedical question here]

Answer with only one word: yes, no, or maybe."""

print(answer_question(prompt, steps=16))

Training Details

Training Data

The model was fine-tuned on qiaojin/PubMedQA.

The notebook used:

• Training split: pqa_artificial
• Evaluation split: pqa_labeled
• Training examples used: 3,000
• Evaluation examples prepared: 200
• Evaluation examples used for reported result: 50

Each training example was converted into a prompt-answer pair:

Input:
Biomedical question + PubMed abstract context

Target:
yes / no / maybe

Training Procedure

Preprocessing

For each PubMedQA row:

1. The abstract contexts were joined into one context block.
2. The context was truncated to 2,500 characters.
3. The question was inserted below the context.
4. The target answer was the final_decision field.
5. Only examples with yes, no, or maybe labels were used.

Prompt format:

Answer the biomedical research question using only the context.

Context:
{context}

Question:
{question}

Answer with only one word: yes, no, or maybe.

Target format:

{final_decision}

Training Hyperparameters

• Training regime: bf16
• LoRA rank: 64
• LoRA alpha: 128
• Trainable parameters: 149,630,976
• Total parameters: 25,973,409,840
• Trainable percentage: 0.5761%
• Training examples: 3,000
• Training steps: 60
• Gradient accumulation: 4
• Learning rate: 1e-4
• Optimizer: AdamW
• Scheduler: OneCycleLR
• Weight decay: 0.0
• Max context characters: 2,500
• Canvas length: 256
• Dataset: qiaojin/PubMedQA

Speeds, Sizes, Times

Training was run on a RunPod H100 notebook.

Training logs from the saved notebook:

step 20/60 | loss 0.0019 | 43s
step 40/60 | loss 0.0003 | 85s
step 60/60 | loss 0.0001 | 126s

Approximate training time:

126 seconds for 60 steps

Evaluation

Testing Data, Factors & Metrics

Testing Data

Evaluation used the pqa_labeled subset of qiaojin/PubMedQA.

The reported run used:

• Evaluation examples: 50
• Denoising steps: 16
• Metric: Accuracy

Factors

The evaluation was not disaggregated by biomedical topic, article type, answer class, or question type. Results should be treated as a small sanity-check evaluation, not a full benchmark.

Metrics

Accuracy was used because PubMedQA final decisions are discrete labels:

yes / no / maybe

A prediction was counted as correct if the cleaned model output matched the gold final_decision.

Results

Setting	Accuracy	Correct / Total
Before fine-tuning	0.60	30 / 50
After fine-tuning	0.80	40 / 50
Improvement	+0.20	+10 / 50

Summary

In the saved RunPod H100 notebook run, the model improved from 60% accuracy before fine-tuning to 80% accuracy after fine-tuning on a 50-example PubMedQA evaluation sample.

This is a +20 percentage point improvement.

The result shows that the model can quickly adapt to the PubMedQA answer format. However, this is a small evaluation and should not be interpreted as a clinically meaningful benchmark.

Model Examination

No detailed interpretability or model examination was performed.

Environmental Impact

Carbon emissions were not measured for this run.

• Hardware Type: NVIDIA H100 80GB HBM3
• Hours used: Approximately 0.04 hours for the 60-step training loop, excluding setup, model loading, and evaluation
• Cloud Provider: RunPod
• Compute Region: Not recorded
• Carbon Emitted: Not measured

Technical Specifications

Model Architecture and Objective

The base model is DiffusionGemma 26B-A4B IT, a diffusion-style language model. The fine-tuning used LoRA adapters.

The training objective followed a block-diffusion setup:

1. Encode the target answer into the model canvas.
2. Randomly corrupt answer tokens.
3. Train the model to reconstruct the clean answer.
4. Apply loss only over the target answer tokens.

Compute Infrastructure

Hardware

• NVIDIA H100 80GB HBM3
• Reported GPU memory: approximately 85 GB total

Software

• Python
• PyTorch 2.10.0+cu128
• Transformers
• Unsloth
• Unsloth Zoo
• PEFT
• Datasets
• RunPod Jupyter Notebook

Citation

If you use this model, please cite the original PubMedQA dataset and DiffusionGemma base model.

PubMedQA:

@inproceedings{jin2019pubmedqa,
  title={PubMedQA: A Dataset for Biomedical Research Question Answering},
  author={Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William W. and Lu, Xinghua},
  booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing},
  year={2019}
}

Glossary

• PubMedQA: A biomedical question-answering dataset based on PubMed abstracts.
• LoRA: Low-Rank Adaptation, a parameter-efficient fine-tuning method.
• DiffusionGemma: A diffusion-style language model.
• Denoising steps: Iterative generation steps used by diffusion models.
• Accuracy: Percentage of predictions matching the gold label.

More Information

This model was created as a simple fine-tuning experiment for adapting DiffusionGemma to a medical QA dataset.

The task is intentionally simple:

Biomedical context + question → yes / no / maybe

For a stronger medical QA model, future versions should train on both:

Decision: yes/no/maybe
Explanation: short evidence-based explanation

Model Card Authors

• kingabzpro

Model Card Contact

For questions, contact the model repository owner on Hugging Face.