Consuloji: Deep Dives into Overseas AI日本語

Running a Smaller Model Inside LLM Weights: What Primary Documentation Confirms About QwenPU and What Remains Unproven

Published: November 2, 2026

Listen or watch on YouTube

An empirical breakdown of QwenPU compiled CPU and vector architecture, 3.18M simulated execution steps, published checkpoint test coverage, and benchmark comparisons.

Japanese Edition: Read the original Japanese article on Substack

Video Episode (Japanese Audio): Watch on YouTube

Executive Summary: Full Verification in Emulation vs. Partial Checkpoint Coverage

In late October 2026, an open-source technical project designated QwenPU reported an intriguing architectural milestone: embedding a compiled 32-bit RISC-V central processing unit and vector acceleration unit directly into the weight matrices of an open-weights large language model, subsequently executing an entire secondary neural network program within standard forward-pass matrix multiplications. According to documentation released by the developer, the virtualized processor executed Karpathy stories260K language model across 3,184,920 consecutive execution steps with zero divergence from a reference implementation.

However, rigorous technical inspection of project source repositories, architectural notes, and verification test scripts reveals an essential operational boundary:

  1. The 3,184,920-step complete match was achieved exclusively within an internal software circuit emulator and GPU execution runner.
  2. In the published open-weights model checkpoint uploaded to Hugging Face, automated test coverage validated the initial 30,000 execution steps.
  3. The benchmark reference against which execution was verified is not an independent third-party emulator, but a custom Python simulator authored by the developer.
  4. As of November 2026, no independent third-party replication of full-sequence checkpoint execution has been documented.

For enterprise software architects, technical directors, and technology evaluators considering neuro-symbolic systems or vendor claims of “100% deterministic precision” inside neural networks, QwenPU offers a masterclass in separating formal mathematical proofs from experimental prototypes. This analysis delivers three practical capabilities:

  • Distinguishing between whole-sequence verification in specialized emulators and partial validation in published model files.
  • Analyzing execution throughput metrics by auditing whether hardware runners and measurement baselines remain consistent.
  • Establishing an architectural framework for separating probabilistic language generation from deterministic computation within enterprise workflows.

Slide 1: Scope of QwenPU Verification Claims


Methodological Boundaries: Documented Records, Author Calculations, and Interpretation

To ensure analytical precision, this evaluation classifies all findings into three distinct categories:

  • Documented Records: Explicit statements, structural schemas, test logs, and benchmark parameters recorded in project README files, architectural documentation (docs/vx.md), test harnesses (vxcheck.py), and foundational research papers.
  • Author Calculations: Mathematical ratios, percentages, and conversions calculated directly from published figures.
  • Author Interpretation: Synthesized evaluations and architectural frameworks regarding enterprise utility, explicitly identified as analytical deductions.

All primary repository records, model cards, and academic citations were retrieved on November 2, 2026. This analysis evaluates the published codebase as static engineering literature; neither the 26.7-gigabyte model weights nor local test runners were executed locally.

Slide 2: Architectural Evolution: From Qwenputer to QwenPU


Architectural Lineage: From Qwenputer to Vector Acceleration

QwenPU represents the direct sequel to Qwenputer, an earlier architecture released by the same developer. In Qwenputer, the developer prepended 11 mathematically compiled layers to the base Qwen3.5-4B architecture, constructing a 32-bit RISC-V integer CPU within the weight matrices without performing gradient-based model training. The processor utilized internal hidden states within the Gated DeltaNet recurrent mechanism as dynamic register and memory storage.

In a prior technical analysis, our commentary stated that one CPU instruction corresponded to exactly three generated tokens. A review of developer documentation clarifies that an instruction decomposes into several sequential trace tokens: designating memory addresses, transferring data values, and signaling instruction retirement. The developer did not claim an invariant three-token length; that phrasing is corrected here.

In QwenPU, the developer expanded the prepended structure from 11 layers to 13 layers by integrating a dedicated Vector Processing Unit (VPU). Where the original CPU processed scalar values sequentially, the vector unit executes matrix-vector products and activation functions over multi-lane arrays.

Slide 3: Compiling 13 Transformer Layers Without Gradient Training


Section 1: Compiling a 13-Layer Computer: What Actually Executes

QwenPU constructs a hybrid neural architecture:

  • Base Architecture: Qwen3.5-4B retaining all 32 original transformer layers untouched.
  • Prepended Processor: 13 compiled layers placed immediately ahead of the base model.
  • Dimensionality: Hidden dimension expanded from base 2,560 to 5,120, reserving the upper 2,560 dimensions for virtualized computer state.
  • Checkpoint Footprint: Stored in 16-bit brain floating-point format (bf16), the published checkpoint contains 13,364,479,936 total parameters, consuming 26,728,959,872 bytes (approximately 26.7 gigabytes). The developer notes that the majority of compiled parameters represent sparse zero-padding.

Crucially, the computer layers were not trained on gradient descent; their weight values were computed analytically by an automated compiler converting digital logic gates into linear projection matrices.

What program runs inside this structure? Project documentation indicates that the system executes Andrej Karpathy stories260K model from the llama2.c repository. The stories260K model is a compact language model comprising 260,000 parameters, a hidden dimension of 64, and 5 transformer layers. The small model 4-bit integer (int4) weights are not hardcoded into QwenPU neural weights; instead, they are loaded into the vector unit virtual memory as raw program data spanning 4,152 memory rows.

The vector acceleration unit comprises:

  • 16 vector registers of 64 words each.
  • 384-lane int8 activation buffers read in 64-lane increments.
  • A static key-value (KV) attention cache accommodating 32 sequence positions.
  • Dedicated vector operations: VMAT (int8-int4 matrix row multiplications across 64 lanes), VQ (quantization into int8 activations), VRED (reduction sums), VMATR (attention score-value multiplications), and fixed-point table lookups for SiLU and softmax activation functions.

This design encompasses every mathematical primitive required to evaluate a decoder-only transformer. However, the story text is authored strictly by the virtualized stories260K C program. Qwen3.5-4B itself does not generate or guide narrative content. “Running an LLM inside an LLM” literally describes running an emulated C interpreter inside compiled weight layers that sequentially evaluates a 260,000-parameter neural network.

Slide 4: Trace Token Compression vs. Real-World Execution Latency


Section 2: Trace Token Reduction vs. Execution Throughput

To advance virtual processor state, QwenPU generates execution trace tokens. Each generated token logs a memory address, register write, or branch event. Because neural network inference generates tokens sequentially, minimizing trace tokens per computational operation is essential to reducing latency.

The developer published comparative telemetry tracking the trace tokens required to generate a single story token across three implementation builds:

Implementation Build Quantization & Architecture Format Trace Tokens Per Output Token Relative Token Compression
Baseline Unaccelerated C int8 weights, scalar RISC-V emulation 2,430,000 45.8x more tokens
Predecessor NPU Build int4 weights, custom dot-product ops 225,000 4.25x more tokens
QwenPU Vector Architecture int4 weights, multi-lane vector engine 53,000 Baseline (1.0x)

Author calculation confirms that QwenPU achieves a 45.8x token reduction versus unaccelerated C (2,430,000 / 53,000 = 45.85) and a 4.25x reduction versus the predecessor NPU (225,000 / 53,000 = 4.25).

However, execution latency did not scale proportionally with token reduction. Project documentation notes:

  • The QwenPU GPU runner operating on an NVIDIA L4 GPU processed approximately 3,300 trace tokens per second, yielding approximately 16.1 seconds per story output token (53,000 / 3,300 = 16.06 seconds).
  • The predecessor NPU runner processed approximately 5,500 trace tokens per second, yielding approximately 40.9 seconds per story output token (225,000 / 5,500 = 40.91 seconds).

While trace token count decreased by 4.25x, wall-clock generation latency decreased by only 2.54x (40.9 / 16.1 = 2.54). The performance gap narrowed because the more complex vector instruction emulation lowered raw token generation velocity from 5,500 to 3,300 tokens per second. Furthermore, project documentation does not specify whether the 53,000-token figure reflects sequence position 0 or steady-state generation, which varied significantly in earlier builds.

Slide 5: Verification Horizons: 3.18M Simulated Steps vs. 30K Checkpoint Steps


Section 3: Verification Audit: 3,184,920 Simulated Steps vs. 30,000 Checkpoint Steps

Project verification documentation outlines three verification tiers:

  1. Instruction-Level Unit Tests: Synthetic micro-benchmarks evaluated within the circuit simulation harness, confirming zero divergence against the reference emulator.
  2. End-to-End Story Execution: Full evaluation of Karpathy story generation program within internal circuit intermediate representation (110,000 steps) and GPU runner across 3,184,920 steps in free-running mode, reporting zero mismatched steps.
  3. Weight Checkpoint Validation: Direct evaluation of published model layers (vxcheck.py) across the initial 30,000 execution steps under teacher forcing.

In free-running execution, the virtual processor feeds its own outputs into subsequent inputs; a single arithmetic mistake cascades into divergent behavior. Under teacher forcing, ground-truth inputs are supplied at each step, preventing error accumulation.

Author calculation highlights the critical verification disparity: evaluating 30,000 steps out of 3,184,920 total steps represents exactly 0.94% coverage (30,000 / 3,184,920 = 0.009419). Approximately 99.06% of the reported end-to-end execution was verified inside the developer circuit simulator, rather than through the distributed 26.7-gigabyte model checkpoint.

Additionally, a minor textual discrepancy exists between project files: the repository README cites 3,184,920 total execution steps, whereas architectural design specification docs/vx.md records 3,184,921 steps.

Furthermore, the verification reference (ref.Machine, defined in chipformer/rv/vx.py) is a custom Python emulator authored by the same developer. Matching this reference proves internal consistency between the compiler output and the developer specification; it does not constitute independent validation against standard RISC-V hardware test suites.

Finally, test scripts isolate and execute only the 13 prepended computer layers. Evaluating the 13 computer layers requires approximately 13 gigabytes of VRAM, fitting within a single 24-gigabyte consumer GPU. Loading the complete 45-layer model requires over 40 gigabytes of VRAM.

Slide 6: Comparative Taxonomy: Looped Transformers, Tracr, and QwenPU


Section 4: Academic Precedents and Architectural Comparisons

Synthesizing compiled computation within transformer weight matrices builds upon prior academic literature:

Research Artifact Publication Venue Architectural Paradigm Scale & Parameters Verification Telemetry
Looped Transformers as Programmable Computers arXiv:2301.13196 (Jan 2023) Hand-crafted looped transformer weights emulating a programmable instruction set 13 recurrent layers, open weight matrices Analytical correctness proofs on synthetic calculators
Tracr (Google DeepMind) NeurIPS 2023 (arXiv:2301.05062) Automated compiler converting RASP programs into standard transformer weights 4-layer demonstration models Formal unit-test guarantees for mechanistic interpretability
Qwenputer (Predecessor) Open Source (mid-2026) 11 compiled layers prepended to Qwen3.5-4B forming integer RISC-V CPU 11 prepended layers, 32 base layers 97.5% on multi-digit arithmetic vs. 55.0% for raw Qwen
QwenPU (Evaluated System) Open Source (Oct 2026) 13 compiled layers prepended to Qwen3.5-4B adding multi-lane vector engine 13 prepended layers, 26.7 GB total footprint 53,000 trace tokens/token, 30,000 steps verified on checkpoint

Giannou et al. (2023) proved that looped transformers can act as universal computers by passing hidden representations through recurrent steps, hypothesizing that hand-crafted modules could eventually fuse with large pre-trained models. Google DeepMind Tracr compiled human-readable programs into weights to establish ground-truth models for interpretability research, explicitly acknowledging that compiled weights exhibit severe computational inefficiency.

QwenPU novelty lies in prepending a compiled vector computing architecture directly to a production open-weights foundation model without modifying base weights, enabling an emulated neural network to execute within forward-pass tensor operations.

Slide 7: Unwritten Parameters in Public Documentation


Section 5: Unwritten Gaps in Public Documentation

Rigorous evaluation requires documenting what public repositories omit:

  1. Third-Party Replication: No external research group or engineering team has published independent confirmation of QwenPU execution.
  2. Full-Sequence Checkpoint Evaluation: Published verification on model files covers 30,000 steps (~0.94%), omitting the remaining 99.06% of the 3.18-million-step story run.
  3. Comparative Practical Speed: Generating 1 story token in 16 seconds corresponds to ~0.062 tokens per second. Running stories260K natively in C on host hardware yields thousands of tokens per second.
  4. General Linguistic Benchmarks: QwenPU repositories provide zero benchmark evaluations (MMLU, GSM8K, or KL divergence) measuring whether prepending 13 layers degrades Qwen3.5-4B native language capabilities.
  5. Architectural Footprint: Loading the hybrid model requires over 40 gigabytes of VRAM to evaluate a 260,000-parameter toy network.
  6. Discrepant Metric Reporting: A 1-step variance exists between README (3,184,920) and design documentation (3,184,921).

Slide 8: Synthesis: Proven Determinism vs. Enterprise Realism


Executive Implementation Framework: Probabilistic Reasoning vs. Deterministic Execution

QwenPU demonstrates that transformer linear projections can emulate deterministic digital logic gates with high internal precision. However, for enterprise IT architects, embedding virtualized processors inside neural network weights introduces immense latency and memory overhead.

The Hybrid Division of Labor

In mission-critical enterprise systems, organizations should enforce a strict architectural boundary:

  • Probabilistic Domain: Utilize foundation models for natural language parsing, unstructured document extraction, summarization, and semantic classification.
  • Deterministic Domain: Delegate mathematical calculations, tax rules, invoice ledger balancing, and transactional validations to external native code or verified database engines.

Three Questions for Vendor “Exact Match” Claims

When evaluating systems claiming 100% precision or deterministic execution within neural weights, technology leaders should demand three verifiable answers:

  1. Verification Boundary: Does reported precision apply to published model weights or internal circuit simulations?
  2. Reference Independence: Was execution matched against an established third-party standard or a proprietary simulator?
  3. Operational Overhead: What is the ratio of auxiliary compute cycles and VRAM required relative to executing the logic natively?

Primary Reference Documentation

Back to articles