MonkeyOCRv2: A Visual-Text Foundation Model for Document AI

arXiv MonkeyOCRv2 MonkeyDocv2 GitHub issues GitHub closed issues Demo

News

  • 2026.07.21 📦 We release MonkeyDoc v2, an open multilingual corpus for document-oriented pretraining. We hope it can serve as a shared data foundation for more transparent, reproducible, and fair comparisons in Document AI.
  • 2026.07.11 🚀 We release MonkeyOCRv2, including MonkeyOCRv2 vision encoder, MonkeyOCRv2-Parsing for multilingual document parsing, MonkeyOCRv2-Und for efficient document understanding.

Use MonkeyOCRv2 as a Vision Backbone

MonkeyOCRv2 is released as a standalone, document-native vision encoder. It can be integrated into different OCR and document AI systems as a visual backbone.

The current release has been evaluated on document parsing, document understanding, text recognition, formula recognition, text detection, document tampering detection, and overlapping-text segmentation.

Beyond these evaluated tasks, the encoder may also be useful for text-rich scenarios such as scientific papers, historical documents, medical reports, charts and tables, and remote-sensing maps or reports. We welcome community exploration of these directions.

from transformers import AutoModel

encoder = AutoModel.from_pretrained(
    "zenosai/MonkeyOCRv2-B",
    trust_remote_code=True,
    dtype="auto",
    device_map="auto",
)

See the Vision Encoder Quick Start for installation and feature-extraction examples. If you adapt MonkeyOCRv2 to a new task or domain, feel free to open an issue or pull request and share the results.

MonkeyDoc v2

MonkeyDoc v2 is currently the largest document image pre-training image-text pair dataset, comprising 113 million document images across 17 languages. The open-sourcing of MonkeyDoc v2 is still underway. So far, we have released all 52 million synthetic samples and 35 million real-world samples derived from FinePDF. Additional real-world data from other sources is currently being organized and will be released progressively. You can download the full datset as follows:

pip install modelscope
modelscope download --dataset zenosai/MonkeyDocv2 --local_dir ./MonkeyDocv2

Model Zoo

1. Vision Encoder

Model Backbone Params Pretraining
Resolution
Applicable Tasks Checkpoint Link
Monkey
OCRv2-S
ViT-S28M1280*28*28Recognition / Parsing / Understanding🤗HuggingFace
🤖ModelScope
Monkey
OCRv2-B
ViT-B113M1280*28*28Recognition / Parsing / Understanding🤗HuggingFace
🤖ModelScope
Monkey
OCRv2-AS
ViTAEv2-S21M1760*32*32Detection / Segmentation🤗HuggingFace
🤖ModelScope

2. Document Parsing Model

Model Link Total Params ViT LLM All Digit. Photo. Latin Avg. DE EN ES FR ID IT NL PT VI Non-Latin Avg. AR HI JP KO RU TH ZH ZH-T
MonkeyOCRv2-S-ParsingHuggingFace ModelScope 0.6B 0.03B 0.6B 82.5 87.9 80.7 83.2 87.3 83.6 76.8 73.6 85.4 87.2 85.5 87.4 81.9 81.7 91.2 87.1 69.9 88.7 78.0 79.8 84.4 74.7
MonkeyOCRv2-B-ParsingHuggingFace ModelScope 0.7B 0.1B 0.6B 83.3 88.1 81.7 84.2 87.7 84.5 75.2 78.4 86.5 88.6 86.1 87.9 83.2 82.1 90.7 87.2 71.9 87.6 80.1 80.8 83.6 75.3

3. Document Understanding Model

Model Link Total Params Overall DocVQA InfoVQA DF KLC WTQ ChartQA DT-VQA OCRBench
MonkeyOCRv2-S-UndHuggingFace ModelScope 1.7B 55.9 79.3 44.5 65.1 37.6 43.0 62.0 63.1 52.2
MonkeyOCRv2-B-UndHuggingFace ModelScope 1.8B 57.2 79.3 46.3 65.8 38.2 43.2 62.0 64.3 58.1

Quick Start

Vision Encoder

1. Install

Install transformers and flash attention:

conda create -n MonkeyOCRv2 python=3.10
conda activate MonkeyOCRv2
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu126
pip install transformers==4.57.6
pip install flash-attn==2.7.4.post1 --no-build-isolation
pip install accelerate
pip install qwen_vl_utils

2. Download Model Weights

Download our model from Huggingface.

python download_model.py -n MonkeyOCRv2-B # or MonkeyOCRv2-S / MonkeyOCRv2-AS

You can also download our model from ModelScope.

pip install modelscope
python download_model.py -t modelscope -n MonkeyOCRv2-B # or MonkeyOCRv2-S / MonkeyOCRv2-AS

3. Extract Image Feature

cd vision
# For MonkeyOCRv2-B and MonkeyOCRv2-S
python extract_feature.py
# For MonkeyOCRv2-AS
python extract_feature_vitae.py

Document Parsing

1. Install

Install vLLM following its official guide:

conda create -n MonkeyOCRv2Parsing python=3.10
conda activate MonkeyOCRv2Parsing
pip install uv
uv pip install vllm==0.11.2 --torch-backend=auto -i https://pypi.tuna.tsinghua.edu.cn/simple requests
pip install -r parsing/requirements.txt

2. Download Model Weights

Download our model from Huggingface.

python download_model.py -n MonkeyOCRv2-B-Parsing # or MonkeyOCRv2-S-Parsing

You can also download our model from ModelScope.

pip install modelscope
python download_model.py -t modelscope -n MonkeyOCRv2-B-Parsing # or MonkeyOCRv2-S-Parsing

3. vLLM Serving

You should start a vLLM service before parsing documents:

cd parsing
python serve.py -m ../model_weight/MonkeyOCRv2-B-Parsing -p 8888
# Show help messages
python serve.py -h

4. Inference

You can parse documents using CLI or serve with demo and FastAPI.

4.1 Parse using CLI

Parse a single document or a directory containing PDFs or images:

cd parsing
python parse.py \
    -i ../images_test \
    -o output/test \
    -s http://127.0.0.1:8888 \
    --draw-layout \
    --skip-processed
# Show help messages
python parse.py -h
4.2 Serve with Web Demo
cd parsing
python demo/gradio_demo.py -s http://127.0.0.1:8888 -p 8891
# Show help messages
python demo/gradio_demo.py -h

You can access the web demo at http://localhost:8891.

4.3 Serve with FastAPI
cd parsing
python fastapi/main.py -s http://127.0.0.1:8888 -p 8000
# Show help messages
python fastapi/main.py -h

You can access the API documentation at http://localhost:8000/docs to explore available endpoints.

Document Understanding

1. Install

See install part of MonkeyOCRv2.

2. Download Model Weights

Download our model from Huggingface.

python download_model.py -n MonkeyOCRv2-B-Und # or MonkeyOCRv2-S-Und

You can also download our model from ModelScope.

pip install modelscope
python download_model.py -t modelscope -n MonkeyOCRv2-B-Und # or MonkeyOCRv2-S-Und

3. Inference

cd understanding
python infer.py \
    -m ../model_weight/MonkeyOCRv2-B-Und \
    -i ../images_test/vqa.png \
    -q 'What is the serving size?'
# Show help messages
python infer.py -h

Visualization

Our model supports robust document parsing in real-world scenarios across 17 languages, including Simplified Chinese (ZH), Traditional Chinese (ZH-T), English (EN), Arabic (AR), German (DE), Spanish (ES), French (FR), Hindi (HI), Indonesian (ID), Italian (IT), Japanese (JP), Korean (KO), Dutch (NL), Portuguese (PT), Russian (RU), Thai (TH), and Vietnamese (VI).

Evaluation Results

1. Text recognition results on Common Benchmarks, Union14M-Benchmark, OST, and Chinese Benchmarks. We follow the training and evaluation protocols of OpenOCR.

Model Overall Union14M-Benchmark Chinese Benchmarks Occlusion SceneText
Avg Artistic Context less Curve General Multi Oriented Multi Words Saliency Avg Scene Web Document Hand writing
ABINet 73.7 75.771.774.780.479.869.076.877.6 70.366.663.298.253.1 75.0
MAERec 81.6 85.279.084.289.184.687.185.986.3 83.184.483.099.565.6 76.4
CPPD 80.4 81.976.582.986.283.578.781.983.5 81.782.782.499.462.3 79.6
IGTR-AR 81.0 84.977.082.490.484.491.284.084.7 81.782.081.799.563.8 76.3
SMTR 80.4 85.076.883.989.183.787.789.384.6 82.783.483.099.365.1 73.5
SVTRv2 83.1 86.179.386.190.685.189.086.786.2 83.383.583.399.567.0 80.0
 
CRNN (ResNet) 58.7 49.251.262.348.168.213.060.441.4 68.863.868.297.046.1 58.0
CRNN (MonkeyOCRv2-S) 67.3 65.263.773.071.174.528.672.173.4 74.273.074.996.951.8 62.4
PARSeq (ViT) 82.2 84.376.583.487.684.988.884.384.4 82.484.282.899.563.0 79.9
PARSeq (MonkeyOCRv2-S) 84.3 87.678.686.492.185.493.988.787.7 83.784.683.299.567.3 81.5

2. Formula recognition results on OmniDocBench 1.6, MathWriting, and UniMER-Test.

Model Params Overall OmniDocBench 1.6 MathWriting SPE CPE HWE SCE
CDM ExpRate CDM ExpRate CDM ExpRate CDM ExpRate CDM ExpRate CDM ExpRate CDM ExpRate
Pix2tex 25.5M 53.8 23.3 69.427.0 0.40.0 96.272.4 64.97.1 24.50.6 67.632.8
Texify 312M 67.3 40.4 76.546.4 26.62.0 98.591.0 70.428.2 52.723.6 79.351.3
UniMERNet-B 325M 89.5 64.5 90.459.5 63.812.3 99.193.3 96.080.5 94.064.3 93.777.0
UniMERNet-S 202M 89.8 64.0 90.159.1 65.912.7 99.193.4 95.977.7 93.763.9 94.176.9
 
UniMERNet-T (Swin) 107M 89.4 61.8 89.957.2 65.612.9 99.192.3 94.969.9 93.361.9 93.876.6
UniMERNet-T (MonkeyOCRv2-S) 110M 90.9 66.4 90.861.1 70.816.2 99.293.8 96.179.2 94.369.5 94.078.6

3. Text detection results on Total-Text, CTW1500, ICDAR2015 and ArT. We follow the training and evaluation protocols of MMOCR and DPText-DETR.

4. Document tampering detection results on DocTamper benchmark.

Method Params Overall DocTamper-Test DocTamper-FCD DocTamper-SCD
IoU F IoU P R F IoU P R F IoU P R F
PSCC-Net 5M 13.731.317.025.083.039.0 13.019.082.030.0 11.015.083.025.0
UperNet 67M 49.354.070.066.060.062.0 30.057.035.043.0 48.057.058.057.0
CAT-Net 114M 67.371.078.075.069.072.0 66.085.070.076.0 58.065.065.065.0
Swin-UPer 81M 66.771.779.075.072.073.0 64.080.070.075.0 57.066.068.067.0
SegFormer 85M 70.374.081.077.074.075.0 69.082.074.078.0 61.068.070.069.0
Mask2Former 69M 69.778.084.082.083.082.0 66.081.075.078.0 59.070.079.074.0
ConvNext 122M 69.775.384.081.078.079.0 62.076.071.074.0 63.071.074.073.0
ConvNextV2 121M 72.777.786.082.079.081.0 65.079.075.077.0 67.074.076.075.0
InternImage 128M 73.377.784.081.077.079.0 72.083.079.081.0 64.073.074.073.0
ASC-Former 80M 68.280.881.591.887.889.8 61.374.977.176.0 61.978.075.076.5
DTD 66M 77.079.784.081.077.079.0 79.088.082.085.0 68.075.076.075.0
 
FFDN* (ViTAEv2) 69M 70.782.7 69.476.288.782.0 79.092.584.488.3 63.679.176.577.8
FFDN (MonkeyOCRv2-AS) 71M 78.287.5 87.494.891.893.3 79.990.487.488.9 67.281.079.880.4

* denotes models trained with the ViTAEv2 pretrained by DeepSolo

5. Overlapping text segmentation results on MOT dataset.

Model mIoUText IoUOcc IoUOccd IoUOv
Unet 62.2 80.2 65.7 40.7
Deeplab v3 67.9 83.2 71.2 49.3
OCRNet 65.8 81.0 68.5 47.8
Segformer 69.0 83.6 74.1 49.3
MaskFormer 68.4 83.5 70.3 51.4
TexRNet 68.9 84.2 73.2 49.3
EAFormer 69.1 83.8 74.2 50.5
WASNet 70.8 84.8 74.4 53.1
 
Mask2Former (ResNet) 70.3 84.7 73.3 52.8
Mask2Former (MonkeyOCRv2-AS) 76.6 88.6 83.4 57.7
MOTS (ResNet) 72.6 85.2 77.5 54.9
MOTS (MonkeyOCRv2-AS) 76.9 88.6 82.6 59.4

6. Document parsing results on MDPBench, a comprehensive multilingual benchmark for real-world document parsing.

Model Total Params ViT LLM All Digit. Photo. Latin Avg. DE EN ES FR ID IT NL PT VI Non-Latin Avg. AR HI JP KO RU TH ZH ZH-T
Closed-source VLMs
ChatGPT-5.2-2025-12-11 - - - 68.6 85.6 63.0 75.2 70.8 79.4 71.4 60.0 77.7 78.5 71.6 85.0 82.1 61.1 64.9 63.4 55.8 65.4 60.7 63.8 56.3 58.7
Claude-Sonnet-4.6 - - - 73.1 85.0 69.3 79.2 79.8 80.6 72.8 66.5 82.3 83.3 76.7 88.0 83.1 66.2 67.8 71.7 63.4 64.3 70.8 65.2 61.3 65.1
Doubao-2.0-pro - - - 74.2 78.9 72.8 75.7 82.8 74.4 69.0 70.0 73.3 82.0 69.9 83.4 76.5 72.5 81.3 75.7 65.8 74.7 63.3 71.9 71.9 75.2
Gemini-3-pro - - - 86.4 90.4 85.1 88.4 91.2 90.6 83.4 82.7 91.5 91.6 87.7 91.4 85.9 84.1 89.4 90.4 74.8 85.5 84.9 80.6 85.1 82.1
Open-source VLMs
InternVL-3.5-8B8.3B0.3B8B42.759.737.053.439.864.247.542.753.860.652.263.257.030.68.29.045.630.326.110.855.359.3
MinerU-2.51.2B0.7B0.5B46.361.940.863.068.878.454.757.367.575.260.458.846.027.41.39.039.114.78.611.372.962.2
DeepSeek-OCR3.4B0.4B3B51.880.742.254.555.058.344.143.260.969.352.453.054.148.956.952.249.128.236.249.459.759.2
MonkeyOCR-pro-3B3.7B0.7B3B52.268.047.065.171.777.955.962.166.274.566.371.140.237.64.64.255.260.542.69.172.252.4
Nanonets-OCR-s4.7B0.7B4B63.778.858.771.375.178.561.262.570.381.069.675.967.555.059.561.855.951.243.539.567.461.5
Nanonets-OCR2-3B3.7B0.7B3B64.279.259.371.476.776.461.866.168.478.574.174.266.056.260.259.252.154.745.544.668.365.1
Qwen3.5-Instruct-9B9.7B0.7B9B65.774.862.772.572.872.072.064.466.277.674.579.174.058.253.456.255.760.354.756.760.867.5
GLM-OCR0.9B0.4B0.5B67.377.963.778.782.784.575.876.279.782.880.277.469.254.321.739.665.561.264.227.478.576.7
Qwen3-VL-Instruct-8B8.3B0.3B8B68.378.465.073.673.771.469.366.268.579.178.382.273.462.563.158.459.961.957.962.062.673.8
HunyuanOCR1B0.4B0.6B68.380.264.372.475.073.163.066.169.980.361.481.980.663.768.373.155.668.952.260.766.864.2
PaddleOCR-VL0.9B0.6B0.3B69.687.663.672.178.279.362.966.077.478.467.972.066.666.765.868.459.977.856.957.878.268.5
olmOCR27.7B0.7B7B70.479.967.276.775.777.372.568.970.681.072.088.084.063.359.060.859.470.665.859.268.663.4
MinerU-2.5-Pro1.2B0.7B0.5B71.086.266.174.678.379.563.467.478.079.772.178.674.267.056.672.259.177.662.661.876.569.7
PaddleOCR-VL-1.60.9B0.6B0.3B75.082.872.678.084.179.769.274.881.682.074.776.479.371.669.465.668.782.570.762.378.075.7
HunyuanOCR-1.51B0.4B0.6B76.886.273.679.779.680.474.270.081.584.578.486.482.473.571.871.665.575.767.477.780.877.2
Kimi-K2.51T0.4B1T77.585.075.081.685.986.272.771.080.686.677.487.686.272.975.874.572.570.961.867.081.778.6
PaddleOCR-VL-1.50.9B0.6B0.3B78.387.475.281.284.883.075.778.183.985.280.680.278.974.971.367.769.586.076.068.484.875.7
chandra-ocr-25.3B0.5B4.8B79.787.877.182.786.686.569.770.384.687.482.790.785.676.478.281.168.880.374.078.573.876.3
dots.mocr3B1.2B1.8B80.590.577.281.782.687.471.370.184.589.383.286.879.979.283.383.675.078.771.277.984.679.6
MonkeyOCRv2-S-Parsing🤗0.6B0.03B0.6B82.587.980.783.287.383.676.873.685.487.285.587.481.981.791.287.169.988.778.079.884.474.7
MonkeyOCRv2-B-Parsing🤗0.7B0.1B0.6B83.388.181.784.287.784.575.278.486.588.686.187.983.282.190.787.271.987.680.180.883.675.3

7. Document understanding performance comparison across different vision foundation models. The evaluation benchmarks are selected following TextMonkey and DT-VQA.

Model Params Overall DocVQA InfoVQA DF KLC WTQ ChartQA DT-VQA OCRBench
CLIP-B 86M 16.0 20.1 24.2 2.3 13.8 12.8 22.2 22.3 10.6
SigLIP2-B 93M 24.9 27.0 23.5 3.1 16.7 17.4 35.0 41.5 35.1
RADIOv2.5-B 98M 37.5 60.3 31.2 29.9 30.4 29.7 51.1 44.2 23.1
OpenVision-B 87M 44.0 63.3 30.7 19.8 33.1 31.1 58.3 62.6 52.9
DINOv3-B 86M 16.1 26.5 20.8 5.6 13.2 14.0 28.9 15.8 3.9
SAM-B 90M 25.2 37.8 22.2 4.7 17.5 17.6 46.5 33.3 21.9
SAM2-B 69M 22.3 32.5 21.9 2.7 15.8 16.6 40.2 30.3 18.4
oCLIP 24M 12.4 14.8 19.5 1.4 7.4 11.4 17.9 19.2 7.4
DiT 86M 8.9 11.3 20.9 0.9 5.2 9.9 12.0 9.2 1.9
MonkeyOCRv2-S*🤗Link 28M 55.9 79.3 44.5 65.1 37.6 43.0 62.0 63.1 52.2
MonkeyOCRv2-B*🤗Link 113M 57.2 79.3 46.3 65.8 38.2 43.2 62.0 64.3 58.1

MonkeyDoc v2

MonkeyDoc v2 is currently the largest document image pre-training image-text pair dataset, comprising 113 million document images across 17 languages. The open-sourcing of MonkeyDoc v2 is still underway. So far, we have released all 52 million synthetic samples and 41 million real-world samples derived from FinePDF. Additional real-world data from other sources is currently being organized and will be released progressively.

Expert Model Labeling Toolchain

We adopt a multi-expert labeling pipeline to obtain reliable annotations for documents. The pipeline includes the following steps:

  1. Structure Detection
    We use dots.mocr for document structure detection and reading-order prediction. The detected regions, including text blocks, tables, formulas, and other layout elements, are cropped from the original page image for subsequent recognition.
  2. Content Recognition
    Each cropped block is independently recognized by three expert models: dots.mocr, PaddleOCR-VL, and Qwen3-VL. These complementary models provide multiple annotations for the same block, reducing reliance on any single OCR system.
  3. Block-Level Agreement Filtering
    We compare the recognition results from the three expert models and filter out blocks with low agreement. For retained blocks, we select the prediction that has the highest average agreement with the other two predictions as the final block-level annotation.
  4. Page-Level Quality Control
    Pages containing any filtered block are discarded. In addition, we use Qwen3 to verify whether the predicted reading order is reasonable, and Qwen3-VL to check whether document regions are missed during structure detection. This multi-expert agreement strategy reduces model-specific annotation errors and improves the reliability of the generated annotations.

References

Citation

If you use any part of this release — the MonkeyOCRv2 encoders, MonkeyOCRv2-Parsing, MonkeyOCRv2-Und, the MDPBench benchmark, or the MonkeyDoc v2 dataset — please cite:

@article{liu2026monkeyocrv2,
  title   = {MonkeyOCRv2: A Visual-Text Foundation Model for Document AI},
  author  = {Liu, Yuliang and Li, Zhang and Zhang, Ziyang and Zhang, Shuo and
             Liu, Qiang and Song, Jiajun and Guo, Zidun and Wang, Xinhan and
             Zheng, Handong and Liu, Yang and Luo, Dongliang and Ma, Zhiyin and
             Zhang, Jiarui and Bai, Xiang},
  journal = {arXiv preprint arXiv:2607.11562},
  year    = {2026}
}

Copyright

We warmly welcome your feedback, suggestions, and contributions, which are essential to the continued development and improvement of our framework. The models are are released under the Apache License 2.0 and are free for both research and commercial use. For any questions, please contact us at xbai@hust.edu.cn or ylliu@hust.edu.cn.

Downloads last month
85
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train zenosai/MonkeyOCRv2-B

Collection including zenosai/MonkeyOCRv2-B

Papers for zenosai/MonkeyOCRv2-B