kitaniai/OpenJudgement-4B-Preview

OpenJudgement 4B Preview

Experimental open-weights judgment model by Kitani

Dataset · Python example · HTTP API · Evaluation results

This is a preview, and we mean that

OpenJudgement is unfinished. We're releasing this checkpoint for people to experiment with, inspect, and build on. It still needs work on judgment quality, calibration, and inference efficiency.

It is not as good as Jev overall in our internal task comparisons. It does show a meaningful improvement over untouched Qwen on our recorded validation comparison: 75.4% versus 64.2% annotation agreement. That is a result on a particular evaluation set, not a claim that we beat the base model on every task.

There are questions it handles well and questions it confidently gets wrong. Please judge the preview by your own examples rather than assuming that clean JSON means a correct answer.

What is this?

OpenJudgement-4B-Preview is a Qwen3.5-4B derivative trained on custom typed-judgment datasets. Give it evidence, a question, and criteria; it returns a probability or a distribution over your answer options.

It supports three question types:

Type	What it returns	Example
Noul	A number from 0 to 1 for the true option	Does this ticket describe a damaged delivery?
Choice	A selected key and probabilities over named options	Which support team should handle it?
Score	An expected value over an ordered rubric, plus its distribution	How urgent is this request?

The API accepts multiple questions about the same state. The reference runtime currently evaluates those questions sequentially; it does not share document computation between them.

This is the checkpoint previously named OpenJudgment-4B-v2, released under the Preview name. The weights are unchanged. It does not contain our later experimental shared-document reader or encoder models.

How we modified Qwen

We kept Qwen's pretrained backbone, tokenizer, and vocabulary output head. We trained rank-16 LoRA adapters on the text backbone, including its attention, recurrent, and feed-forward projections, then merged the selected adapters into the released weights.

The important change is the training objective and inference path:

state + question + indexed answer criteria
                    ↓
       Qwen3.5 text backbone, one forward pass
                    ↓
    scores for answer-index tokens: 0, 1, 2, ...
                    ↓
     temperature scaling + probability normalization
                    ↓
          Python constructs the response JSON

Each candidate is assigned a numeric index. We score the corresponding next-token logits at the final prompt position, normalize over the available candidates, and train against the target distribution. We preserve soft human-label distributions where the dataset provides them. Thinking is disabled in the prompt.

The model does not generate the JSON token by token. It also does not generate an explanation or a hidden reasoning trace through this runtime. It remains an LLM-derived model with a causal backbone; this is not a newly invented encoder architecture or a model trained from scratch.

All options for one question are scored together. Seven questions still require seven backbone passes. This distinction matters when measuring latency.

Run it

The reference implementation uses Python 3.12, PyTorch, and Transformers. A CUDA GPU is recommended. The stored weights alone are approximately 9.1 GB; activations and runtime overhead require additional memory, especially for long inputs. CPU execution is available through device="cpu", but uses float32 weights and is much slower.

Download the repository, including its Python files and weights:

python -m venv .venv
source .venv/bin/activate
python -m pip install huggingface_hub
hf download kitaniai/OpenJudgement-4B-Preview --local-dir ./OpenJudgement-4B-Preview
cd OpenJudgement-4B-Preview

# Linux / NVIDIA CUDA 12.8 reference installation:
python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r requirements.txt
python examples/basic.py --device cuda

Use a PyTorch build appropriate for your platform if you are not using that CUDA environment. The merged checkpoint does not require PEFT for inference.

From a Python script in the downloaded repository:

import json
from openjudgement import OpenJudgement

model = OpenJudgement.from_pretrained(".", device="cuda")

result = model.system_one(
    state={"ticket": "My parcel arrived with a cracked screen. I need a replacement."},
    questions={
        "damaged_delivery": {
            "type": "noul",
            "instructions": "Does the customer report receiving a damaged item?",
        },
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this ticket?",
            "criteria": {
                "shipping_returns": "Damaged deliveries, returns, and replacements.",
                "billing": "Payment failures, duplicate charges, and invoices.",
                "account": "Passwords and account access.",
            },
        },
        "urgency": {
            "type": "score",
            "instructions": "Rate urgency using this ordered rubric.",
            "criteria": [
                "Routine information request; no reported problem.",
                "Minor inconvenience; the product is still usable.",
                "Damaged or unusable product requiring a timely resolution.",
                "Immediate danger or an explicitly reported safety emergency.",
            ],
        },
    },
)

print(json.dumps(result, indent=2))
print(result["answers"]["damaged_delivery"]["noul"])
print(result["answers"]["team"]["choice"])
print(result["answers"]["urgency"]["score"])

Once these Python files are available, OpenJudgement.from_pretrained("kitaniai/OpenJudgement-4B-Preview", device="cuda") can also download/cache the weights. Pass revision="<commit SHA>" to pin a Hub checkpoint.

Use the supplied wrapper, not a generic text-generation or text-classification pipeline. Simply calling .generate() does not reproduce this judgment API.

Serve a small HTTP API

From the downloaded repository:

uvicorn serve:app --host 127.0.0.1 --port 8000 --workers 1

The API loads one model instance. It exposes /health, /v1/judgments, and interactive API documentation at /docs.

curl http://127.0.0.1:8000/v1/judgments \
  -H 'Content-Type: application/json' \
  --data-binary @examples/request.json

Or run the included standard-library Python client:

python examples/client.py

OPENJUDGEMENT_MODEL, OPENJUDGEMENT_DEVICE, and OPENJUDGEMENT_MAX_LENGTH override the local model path, device, and context limit. Use one worker per model instance; increasing Uvicorn workers loads additional copies. This is a minimal local serving example, not a managed production service.

Understanding the response

• Noul: answers[key].noul is the model's normalized true-option probability. Optional criteria must contain both "true" and "false" descriptions. Absence of evidence is not universally the same as falsity; define your intended meaning in the question.
• Choice: choice is the highest-probability key; probabilities contains every supplied key. Keys remain in input order.
• Score: rubric positions are zero-based. score is the probability-weighted position, not necessarily the winning level. With four criteria, its range is 0–3. legend maps positions to their descriptions.
• Confidence: Choice uses normalized maximum probability, (K × max(p) − 1) / (K − 1); Score uses 1 − H(p) / log(K). These are distribution-concentration heuristics, not independently calibrated probabilities of correctness.

Per-type temperatures from calibration.json are applied before normalization. This improves fit to the recorded calibration data; it does not guarantee reliable uncertainty on a new domain. We do not claim these formulas reproduce Jev's internal calibration.

Training and data

Dataset available at kitaniai/OpenJudgment-4B-Preview.

The custom corpus adapts human-supervised preference, moderation, evidence-assessment, and intent datasets into the same typed decision format. Sources include HelpSteer3, Aegis 2.0, ClimateCheck, UAReviews, and CoVal. This is not a dataset of Jev API responses.

Item	Recorded value
Full corpus	111,049 decisions
Training split	87,621 decisions
Native training-split input tokens	98,928,078
Selected checkpoint	Step 700
Examples seen at selected checkpoint	44,741
Input tokens seen at selected checkpoint	49,046,168
LoRA rank	16
Stored model parameters	4,539,265,536

Corpus size, training-split size, and actual training exposures are different quantities. The selected checkpoint did not consume every training row. The repository retains the complete upstream backbone, including multimodal weights, but the judgment API is text-only.

Source revisions, annotations, adaptations, and source-specific licenses are documented in the dataset. Original training and evaluation records are retained in reports/; historical filenames and repository names in those records reflect the run at the time.

How good is it?

The saved comparison against untouched Qwen used the same 786 validation decisions:

Metric	Qwen3.5-4B base	Selected fine-tune
Annotation argmax agreement ↑	64.2%	75.4%
Macro-source negative log-likelihood ↓	0.918	0.661

The calibrated release recorded 76.8% annotation agreement on 816 test decisions. That test figure is not a base-versus-fine-tune comparison. Some targets encode subjective preferences or tied human votes, so agreement is not universal factual accuracy. Validation was used for selection and is not an untouched final test.

Full per-source and per-type results are in reports/final_metrics.json. These numbers are not a Jev benchmark and should not be read as a general model ranking. Our separate internal comparisons still put Jev ahead overall.

Known problems

It's unfinished. Date arithmetic, experience-duration questions, nuanced qualifiers, unfamiliar rubrics, and complicated evidence combinations can still go wrong. It can assign confident scores to an incorrect answer. Results vary by domain, and a better aggregate result can hide weak individual tasks.

The runtime currently supports 2–10 options per Choice or Score question, 1–128 questions per request, and a configured limit of 32,768 tokens per encoded question, including state, criteria, and prompt overhead. Oversized questions are rejected rather than silently truncated. Supporting a context length is not a guarantee of equal quality throughout that context.

Performance depends on hardware, input length, question count, and kernel support. Transformers can fall back to slower PyTorch implementations for Qwen's recurrent layers if optimized dependencies are absent. We are not advertising a universal latency number for this preview.

Why release it now?

The checkpoint is useful enough to experiment with, and the implementation is small enough to inspect. We would rather make the working weights, data, and inference path available than present an unfinished experiment as a finished Jev replacement.

The remaining work includes better evidence reasoning, stronger domain coverage, calibration, and reducing repeated work across questions. If you benchmark this checkpoint, please call it OpenJudgement-4B-Preview so the results stay tied to this release.

Attribution and license

Developed by Kitani, based on Qwen3.5-4B from the Qwen team / Alibaba. The upstream developers are not responsible for our fine-tuning, runtime, evaluations, or model behavior.

Model and release code: Apache 2.0; see LICENSE and NOTICE. Dataset contents retain their source-specific terms; the model license does not relicense the dataset.

OpenJudgement 4B Preview
experimental · open weights · unfinished

Kitani