Consuloji: Deep Dives into Overseas AI日本語

Which Configuration Produced Xiaomi-OCR-0's "#1" Rank? Reading the Footnotes Before Enterprise Invoice Processing

Published: November 1, 2026

Listen or watch on YouTube

A forensic benchmark analysis of Xiaomi-OCR-0’s 95.24, 93.45, and 92.73 Real5 scores, evaluating two-stage pipelines, layout detector overhead, and invoice extraction realities.

Japanese Edition: Read the original Japanese article on Substack

Video Episode (Japanese Audio): Watch on YouTube


Executive Summary & Key Takeaways

  • Core Finding: Xiaomi-OCR-0—an open-source compact optical character recognition model released by Xiaomi—claims the “#1 rank” on the Real5 benchmark with a composite score of 95.24. However, a forensic inspection of primary documentation reveals three conflicting overall scores for the exact same model: 95.24 in the technical report and GitHub README, 93.45 in the evaluation pipeline documentation, and 92.73 on the benchmark maintainer’s official Hugging Face leaderboard.
  • The Operational Distinction: The headline score of 95.24 is not achieved by the model operating alone; it requires an unbundled two-stage pipeline preceded by an external layout detector (PP-DocLayoutV3 via PaddleX at a 0.25 threshold). The advertised “0.8B” parameter count (873,438,784 weights) excludes this external detector entirely. The 93.45 figure reflects an end-to-end full-page pipeline without a detector. On the maintainer’s official leaderboard, Xiaomi-OCR-0 ranks #2 at 92.73, trailing PaddleOCR-VL-1.6 (93.19).
  • Extraction Weaknesses in Practice: While Xiaomi-OCR-0 posts an average VQA score of 83.2 (beating Qwen3.5-2B at 80.9), it falls behind Qwen3.5-2B on pure OCR text recognition in OCRBench (84.6 vs. 85.9). Furthermore, in Xiaomi’s internal corporate information extraction benchmark, the model scored only 65.00% on invoices, trailing Qwen3.5-2B (77.00%) by 12.00 percentage points. Critical operational variables—including Japanese language support, per-page latency, and GPU VRAM requirements—are entirely absent from official documentation.
  • Audience & Purpose: Written for enterprise IT leaders, digital transformation managers, and engineering teams evaluating compact vision-language models for automated document processing, invoice ingestion, and financial workflow automation.

Slide 1: Which Score Represents Xiaomi-OCR-0’s Rank #1?


Methodological Framework: Documentation, Computation, and Interpretation

This audit separates three categories of statements:

  1. Documented Statements (Primary Sources): Claims, tables, and notes drawn directly from Xiaomi’s research report (arXiv:2609.36136), Hugging Face model card, GitHub repository, evaluation scripts, and maintainer leaderboards for Real5 and OmniDocBench.
  2. Author Calculations: Mathematical values computed by the author from published figures (e.g., subtracting baseline scores, calculating condition averages, and parameter memory footprints).
  3. Analytical Interpretation: Explicit engineering inferences regarding enterprise operational constraints and reproducibility limits.

All primary artifacts were collected on November 1, 2026, and re-verified on November 2, 2026. The author did not execute local model weights or evaluate proprietary hardware latency; findings are based strictly on primary documentation.

Slide 2: Three Conflicting Real5 Numbers


Five Diagnostic Questions for OCR Benchmark Tables

When evaluating published OCR accuracy tables, engineering teams should apply five diagnostic questions:

  1. What pipeline configuration was evaluated? Is it a two-stage pipeline with an external layout detector or an end-to-end full-page parser? Are detector weights counted in advertised model size?
  2. Who performed and verified the benchmark? Did figures originate from developer promotional materials, official maintainer leaderboards, or independent third-party audits?
  3. What is the mathematical composition of the aggregate metric? Does the composite score account for reading order, or does it measure character error rates and table tree distances?
  4. Were competing models evaluated under identical runtime conditions? Did baselines use the same detector, thresholds, and prompt formatting?
  5. What operational specifications are omitted from documentation? Does documentation address target language support, inference latency, and VRAM overhead?

Section 1: The 95.24 Score Requires an External Detector Omitted from the “0.8B” Size

Slide 3: 95.24 Requires a Two-Stage Pipeline with PP-DocLayoutV3

The evaluation pipeline documentation defines two operational modes:

  1. End-to-End (e2e) Full-Page Processing: Feeds the complete page image directly into the vision-language model, transcribing text and structural Markdown in a single forward pass.
  2. Two-Stage Pipeline Processing: An external layout analysis model segments the page into bounding boxes (text paragraphs, headers, tables, images). Each cropped region is transcribed individually by Xiaomi-OCR-0, and outputs are stitched into a full-page document.

In Xiaomi’s technical report, Section 4 specifies that all primary document parsing results reflect the two-stage pipeline utilizing PP-DocLayoutV3 (PaddleX implementation) at a detection threshold of 0.25.

Crucially, the report notes that the advertised “0.8B” parameter count reflects only the vision-language backbone. Hugging Face repository metadata confirms a parameter count of 873,438,784 weights. The parameter count and memory footprint of the required PP-DocLayoutV3 detector are omitted from published totals.

While the technical report’s abstract explicitly conditions its 95.24 claim—stating the model achieves top rank “under the condition where PP-DocLayoutV3 handles layout detection”—the Hugging Face model card omits this qualifying clause, claiming unqualified “#1” status across three benchmarks.

Developer Guidance on Pipeline Selection

Xiaomi’s documentation provides practical architectural guidance:

  • Two-Stage Pipeline: Recommended for standard printed, multi-column documents with predictable geometry.
  • End-to-End Pipeline: Recommended for unconstrained camera captures, handwriting, calligraphy, and irregular layouts where bounding-box errors segment text across semantic boundaries, corrupting context.

In production, two-stage pipelines suffer from cascading errors: if the front-end layout detector misclassifies a table cell or merges separate paragraphs, the transcription model cannot recover lost layout structure.


Section 2: Reconciling Real5’s Three Conflicting Scores

Slide 4: Who Measured Real5? Deconstructing the Leaderboard

Real5 evaluates document transcription robustness across 1,355 pages from OmniDocBench v1.5 photographed under five challenging environmental capture conditions (6,775 total images): Scan, Bend (page curvature), Screen (display moiré), Light (uneven illumination/glare), and Tilt (perspective skew). The overall score is the arithmetic mean across the five conditions.

Tracing Xiaomi-OCR-0’s performance across published sources reveals three distinct figures:

Source Document Pipeline Configuration Overall Real5 Score Scan Bend Screen Light Tilt
Technical Report & GitHub Two-Stage (PP-DocLayoutV3) 95.24 96.41 95.46 94.47 95.39 94.46
Evaluation Pipeline README End-to-End (Full-Page) 93.45 94.52 93.40 93.28 93.86 92.17
Maintainer Leaderboard Not Specified in Table 92.73 93.73 92.76 92.80 92.68 91.69

Mathematical Discrepancy Analysis

  1. Two-Stage vs. Leaderboard: The technical report’s two-stage score (95.24) exceeds the maintainer leaderboard score (92.73) by 2.51 points. Looking at individual conditions, the two-stage pipeline scores higher across all five categories: Scan (+2.68), Bend (+2.70), Screen (+1.67), Light (+2.71), and Tilt (+2.77).
  2. End-to-End vs. Leaderboard: The evaluation pipeline’s end-to-end score (93.45) also exceeds the maintainer leaderboard score (92.73) across all five conditions by 0.48 to 1.18 points.
  3. Rank Reversal: On the maintainer leaderboard, PaddleOCR-VL-1.6 holds rank #1 at 93.19, placing Xiaomi-OCR-0 in second place at 92.73 (-0.46 points). While the technical report claims a +2.05 point lead over PaddleOCR-VL-1.6, that comparison holds only when comparing Xiaomi’s two-stage pipeline against PaddleOCR’s published figure.

Primary sources do not explain whether the 92.73 leaderboard score reflects a different checkpoint, alternate pre-processing, or a third-party rerun.


Section 3: Comparative Analysis Across Published Leaderboards

Slide 5: Comparing Competing Models Across Data Sources

Comparing candidate models requires cross-referencing figures across reporting bodies:

Benchmark & Metric Source Document Xiaomi-OCR-0 PaddleOCR-VL-1.6 OvisOCR2 TeleOCR Documented Pipeline Context
Real5 (Composite) Xiaomi Report Table 95.24 93.19 92.29 — Xiaomi uses 2-stage; competitors unstated
Real5 (Composite) Maintainer Leaderboard 92.73 93.19 92.29 90.72 Pipeline configurations unstated
OmniDocBench v1.6 Xiaomi Report Table 96.83 96.33 96.58 96.87 Xiaomi uses 2-stage; TeleOCR leads by 0.04
OmniDocBench v1.6 Maintainer Leaderboard — 96.34 96.47 96.91 Xiaomi omitted; OXR-1.0 leads at 97.03

Benchmark Composition: What OmniDocBench Measures

OmniDocBench comprises 1,651 PDF pages spanning 10 document genres, 5 layout formats, and 5 languages. Its composite score is an unweighted average of three metrics:

  1. Text Edit Distance: Normalized character edit distance for prose text.
  2. TEDS (Tree-Edit-Distance-based Similarity): Structural and content accuracy for tables.
  3. CDM (Character-level Dice Metric): Mathematical equation transcription accuracy.

Crucially, reading order is excluded from the composite metric. A model can transcribe text paragraphs out of sequence across multi-column layouts while retaining a high composite score.

Furthermore, OmniDocBench genres center on academic papers, financial reports, newspapers, textbooks, and handwritten notes. Standard enterprise commercial forms—such as receipts, purchase orders, and structured invoices—are unrepresented.


Section 4: What Averages Conceal: OCRBench Deficits and Invoice Realities

Slide 6: Average VQA 83.2 vs. OCRBench and Invoice Deficits

Xiaomi reports an average Visual Question Answering (VQA) score of 83.2, outperforming Qwen3.5-2B (80.9) and approaching Qwen3.5-4B (84.9). However, decomposing the five underlying benchmarks reveals critical trade-offs:

VQA Benchmark Component Xiaomi-OCR-0 Qwen3.5-2B Qwen3.5-4B Primary Evaluation Focus
DocVQA 93.1 92.4 94.6 Document layout question answering
InfoVQA 83.9 80.5 86.8 Infographic comprehension
ChartQA 85.3 81.3 84.8 Statistical charts and graphs
TextVQA 69.3 64.5 71.8 Scene text in natural images
OCRBench 84.6 85.9 86.6 Broad optical character recognition
Average VQA Score 83.2 80.9 84.9 Unweighted aggregate across 5 tasks

On OCRBench—the benchmark evaluating pure character and word recognition across complex document layouts—Xiaomi-OCR-0 scores 84.6, falling 1.3 points behind Qwen3.5-2B (85.9) and 2.0 points behind Qwen3.5-4B (86.6).

Enterprise Information Extraction Performance

In Section 4.3, Xiaomi evaluates zero-shot key-value information extraction across 1,655 internal document images (identity credentials, bank cards, forms, and business invoices):

  • Aggregate Extraction Accuracy: Xiaomi-OCR-0 achieved 59.09%, surpassing Qwen3.5-2B (51.60%).
  • Invoice Extraction Accuracy: On business invoices, Xiaomi-OCR-0 scored 65.00%, trailing Qwen3.5-2B (77.00%) by 12.00 percentage points.

The authors explicitly acknowledge that address and invoice field extraction represent ongoing architectural challenges.

Slide 7: Unaddressed Operational Parameters: Japanese, Latency, and VRAM

Three Critical Gaps in Official Specifications

  1. Zero Documented Japanese Capability: Textual searches across the research paper, model card, and code repositories yield zero mentions of Japanese text evaluation.
  2. Missing Latency Benchmarks: Official documentation contains no processing latency figures per page. Claims of “lightweight edge execution” lack published throughput measurements.
  3. Omitted Memory Footprint: At 16-bit precision, 873M parameters consume roughly 1.75 GB of static storage. However, runtime GPU VRAM requirements—including vision encoder activations, KV-cache allocation, and the external PP-DocLayoutV3 detector—are unstated.

Practical Five-Step Evaluation Protocol for In-House Corporate Documents

Slide 8: Conclusion: Five Auditing Questions and Testing Steps

Enterprise teams evaluating compact OCR models should execute an internal proof-of-concept:

  1. Curate 30 Representative Documents: Assemble approximately 30 internal documents, including structured invoices, purchase orders, complex multi-page tables, and angled mobile captures.
  2. Test Both Pipeline Configurations: Run candidate documents through both the two-stage pipeline and end-to-end full-page parser. Measure whether detector failures degrade transcription quality.
  3. Audit Unrepresented Formatting: Include document edge cases: vertical text, mixed Japanese/English typography, company seals, and degraded print quality.
  4. Benchmark Structured JSON Extraction: Prompt models to extract key fields (invoice date, total tax-inclusive amount, vendor registration number) into strict JSON schemas. Evaluate field-level exact-match accuracy.
  5. Profile Device Economics: Measure actual execution time per page and peak GPU memory on target infrastructure.

Conclusion & Governance Guidelines

Published benchmark tables serve to identify candidate architectures, not determine enterprise procurement. Xiaomi-OCR-0 demonstrates strong document parsing capabilities for an 873M parameter model, but its advertised “#1” rank depends on an unbundled two-stage pipeline with an external layout detector.

When evaluating OCR models, engineering leaders should:

  • Require vendors to disclose whether benchmark scores reflect standalone models or multi-stage pipelines.
  • Verify whether advertised parameter counts include pre-processing detectors.
  • Disregard aggregate averages and evaluate accuracy on specific corporate document formats.

Limits of This Investigation

  • Author findings are based on published documentation as of November 1–2, 2026.
  • Model weights were not executed locally; performance on Japanese documents and local hardware latency remain unverified.
  • Maintainer leaderboards and model repositories may be updated following publication.

Final Verification Checklist

  • Verified whether benchmark scores reflect two-stage pipelines or end-to-end execution.
  • Confirmed whether advertised parameter counts include external layout detectors.
  • Reconciled conflicting scores across developer reports and maintainer leaderboards.
  • Audited task-specific sub-scores (OCRBench, invoices) beneath aggregate averages.
  • Identified unaddressed operational parameters (target languages, latency, VRAM).
  • Formulated an empirical testing protocol for corporate document repositories.

Back to articles