Why AI Is Abandoning Tokenizers for Raw Bytes and Communicating via Latent Telepathy: Inside Cartesia H-Net and MIT Swarm Telepathy
By replacing fixed 128k BPE vocabularies with learned byte chunking and eliminating natural language relays in multi-agent swarms through direct KV-cache transfer, frontier research is dismantling two foundational bottlenecks of modern deep learning.
Published: October 30, 2026 | Read time: ~10 min | Watch Video (Japanese)
Executive Summary: Dismantling the Discrete Language Illusion
For nearly a decade, the architecture of Large Language Models (LLMs) has rested on two unquestioned assumptions: first, that models cannot ingest raw data directly and must rely on human-engineered Byte-Pair Encoding (BPE) tokenizers; second, that when autonomous AI agents collaborate, they must communicate by generating natural language text that downstream models read from scratch.
In late October 2026, two independent research breakthroughs simultaneously dismantled both assumptions.

First, a team led by Albert Gu (Carnegie Mellon University Assistant Professor and co-founder of Cartesia, creator of the Mamba state-space architecture) published H-Net (Hierarchical Network: From Tokens to Bytes Part I). By stripping away fixed 128,000-token BPE vocabularies and learning dynamic byte-level chunking end-to-end directly from 256 raw UTF-8 byte values, H-Net delivered:
- 1.28x compute efficiency in in-distribution validation bits-per-byte (bpb) over an iso-FLOP BPE Transformer baseline.
- 1.83x efficiency on Paloma and 2.06x efficiency on OLMES downstream language and reasoning benchmarks.
- 1.49x to 2.40x efficiency across programming languages, expanding to 3.10x median and 5.37x maximum efficiency (Prolog) on out-of-distribution code.
- 5.95x to 7.06x efficiency on raw DNA sequences (OpenGenome 274B base pairs).
- A 3.67B parameter 2-stage H-Net reached the average downstream evaluation score of SmolLM3-3B (trained on ~11 trillion tokens) using only ~1 trillion tokens—a greater than 10x data efficiency gain.

Second, researchers from MIT CSAIL and the MIT-IBM Watson AI Lab (Abdulrahman Alabdulkareem, Dan Gutfreund, et al.) introduced Swarm Telepathy (SwaT). Rather than forcing coordinating AI agents to summarize complex internal states into verbose text strings, SwaT completely freezes sender and receiver models, employing a lightweight cross-model adapter trained on fewer than 50 million tokens in under one GPU-day to translate and transfer Key-Value (KV) caches directly across disparate model families. Evaluated across 6 model families, 12 models, and 17 benchmarks, SwaT achieved:
- 40% higher average accuracy over natural language communication.
- 4x faster wall-clock execution by eliminating autoregressive generation and downstream prefill.
- Direct transfer of visual feature memory from Vision-Language Models (VLMs) into text-only LLMs without vision encoders, allowing pure language models to accurately reason over images.
| Architectural Dimension | Traditional Baseline (BPE & Text Relays) | Frontier Paradigm (Cartesia H-Net & MIT SwaT) | Quantitative Benchmark Impact |
|---|---|---|---|
| Input Tokenization | 128k fixed BPE dictionary (static subwords) | Raw UTF-8 byte input (256 values) + learned dynamic chunking | 1.28x bpb, 1.83x Paloma, 2.06x OLMES efficiency |
| Code Representation | BPE rigid space/newline chunking | Learned syntax-aware chunking | 1.49x (Markdown) to 2.40x (Go); 5.37x on Prolog |
| Pre-Training Efficiency | SmolLM3-3B (~11T tokens to baseline) | 2-stage H-Net (3.67B parameters) | Matches 11T-token SmolLM3 benchmark at ~1T tokens (>10x) |
| Genomic Modeling | Autoregressive base-by-base Transformer | H-Net biological chunking | 5.95x to 7.06x efficiency on OpenGenome |
| Agent Communication | Text generation + downstream prefill | Prefill-free KV-cache direct transfer | 40% higher accuracy, 4x wall-clock speedup |
| Cross-Modal Telepathy | Text descriptions of visual scenes | Direct VLM-to-text-only KV projection | Text-only LLMs solve image tasks without vision backbones |
1. The Death of Fixed BPE: Inside Cartesia H-Net
In his foundational 2019 essay The Bitter Lesson, Richard Sutton observed that heuristic domain knowledge engineered by humans consistently loses to general methods that leverage compute to learn directly from data.
Modern Large Language Models have violated this principle in one glaring component: the BPE tokenizer.


1-1. The Three Structural Flaws of Fixed Tokenizers
Standard LLMs do not process text; they process subword IDs generated by static frequency heuristics determined before model training begins. In Cartesia’s baseline distribution (70% English, 20% code, 5% math, 5% multilingual), a 128k vocabulary yields an average of 4.57 bytes per token (4.95 in English, 5.43 in Russian, 4.79 in C#, 4.48 in Swift).
This static partitioning introduces three crippling defects:
- Premature Boundary Freezing: Token boundaries are locked based on character co-occurrence frequencies without semantic understanding, preventing the network from adapting chunk sizes to context.
- Fragmentation in Non-English and Code: In unsegmented languages (Japanese, Hindi), specialized syntax, or noisy text with typographical errors, BPE fragments inputs into single-byte tokens, inflating sequence lengths and destroying attention efficiency.
- Incompatibility with Unstructured Modalities: Continuous biological sequences (DNA/RNA) and raw audio waveforms lack whitespace delimiters, rendering BPE frequency splitting arbitrary and inefficient.
1-2. H-Net: Five-Module Hierarchical Processing
Cartesia’s H-Net eliminates the tokenizer entirely, ingesting 256 raw byte values through a five-stage hierarchical pipeline:
- Outer Encoder: Ingests raw byte embeddings and mixes local contextual representations.
- Learned Dynamic Chunking Layer: Evaluates semantic boundary probabilities end-to-end, condensing raw byte streams into discrete variable-length abstract chunks.
- Inner Main Backbone: A deep Transformer network operating over compressed chunk representations (analogous to BPE token density) executing global attention.
- Dechunking Layer: Unpacks compressed representations smoothly back to the original byte resolution.
- Outer Decoder: Predicts the next raw UTF-8 byte token.
To establish rigorous parity, Cartesia evaluated an iso-FLOP 1-stage H-Net against a modern BPE Transformer ladder (Muon optimizer, WSD scheduling, 128-dim attention heads, $10^{18}$ to $10^{21}$ FLOPs) with matching parameters, FLOP allocations, and an enforced average compression ratio of 4.57 bytes/chunk.
Across every scale, 1-stage H-Net outperformed the BPE Transformer:
- Validation Loss (bpb): Reached identical loss with 22% less compute (1.28x compute efficiency).
- Downstream Reasoning: Delivered 1.83x efficiency on Paloma and 2.06x efficiency on OLMES.
- In-Distribution Code: Surpassed BPE across all 15 evaluated programming languages, ranging from 1.49x in Markdown to 2.40x in Go.
- Out-of-Distribution Code (The Stack v2): Evaluated on 10 programming languages excluded from pre-training (OCaml, Prolog, Lean, Fortran), H-Net achieved a median efficiency gain of 3.10x, peaking at 5.37x on Prolog.
1-3. 2-Stage H-Net (3.67B) and In-Context Tokenization
Cartesia extended the architecture into a 2-stage H-Net (3.67B parameters, 4.44 GFLOPs/byte), aligned within 2% compute per byte to the open-source SmolLM3-3B (3.08B parameters, 36 layers, 4.36 GFLOPs/byte):
- Stage 0 (Byte Level): 4 layers of Mamba-2 in encoder and decoder (~4x compression).
- Stage 1 (Intermediate Level): Sliding-window attention plus 4 layers of Mamba-2 (~2x additional compression; total ~7.6x to 8x compression).
- Main Stage: 40-layer Global Attention Transformer (3.13B parameters, 57% of total FLOPs).
When trained on identical token mixtures, 2-stage H-Net matched SmolLM3’s 11-trillion-token benchmark performance using only ~1 trillion tokens—delivering a greater than 10x data efficiency advantage.
Analysis of boundary activations revealed the emergence of In-Context Tokenization:
- Semantic Sensitivity: In English, the character
"a"is bound to adjacent nouns ("a foot"), whereas in Italian it is parsed as an isolated preposition chunk. - Dynamic Variable Absorption: When novel identifiers, neologisms, or repeated code symbols (such as
LURDorWHYV) recur within a context window, H-Net dynamically coalesces the multi-character string into a single chunk. - Emergent Biological Discovery: In OpenGenome (274 billion base pairs), H-Net autonomously chunked protein-coding genes into 3-base codons, segmented tRNA into individual bases, and compressed non-coding repetitive DNA into large blocks—achieving 5.95x to 7.06x efficiency gains over character-level models without prior biological training.
2. MIT Swarm Telepathy: Cross-Family KV-Cache Transfer
While H-Net eliminated tokens at the input layer, MIT CSAIL and MIT-IBM Watson AI Lab resolved the communication bottleneck between coordinating models.


2-1. The Friction of Natural Language Communication
Multi-agent coordination traditionally relies on text generation: Agent A digests context, autoregressively outputs a natural language summary, and Agent B reads this summary through full contextual prefill. This introduces severe degradation:
- Quantization Loss: High-dimensional continuous representations and probability distributions in Agent A’s KV cache are collapsed into discrete text tokens, discarding nuanced uncertainty.
- Double Latency Penalty: Agent A incurs token-by-token decoding latency, followed immediately by Agent B’s quadratic prefill latency.
2-2. Prefill-Free Cross-Family KV Transfer
Swarm Telepathy (SwaT) enables disparate models to bypass text generation entirely, transforming and injecting intermediate KV caches directly across different architectures:
- Frozen Base Models & Sub-GPU-Day Training: Both sender and receiver LLMs remain 100% frozen. SwaT trains only a lightweight cross-attention projection adapter using fewer than 50 million tokens in under one GPU-day.
- Cross-Family Universality: Evaluated across 6 diverse model families (Llama, Qwen, Gemma, Granite, Ministral, SmolLM) spanning 12 concrete models on 17 benchmarks, SwaT delivered 40% higher average accuracy while operating 4x faster in wall-clock time.
- Cross-Modal Telepathy: SwaT transferred visual representations from a Vision-Language Model directly into a text-only LLM lacking any native visual encoder. The text-only model solved visual reasoning tasks without processing raw pixels, operating entirely on projected latent KV caches.
3. Critical Counterpoints, Reproducibility, and Governance Risks
Deploying byte-level models and latent telepathy in production introduces major architectural and governance trade-offs.

3-1. Experimental Boundaries and Weight Verification
Technical leaders must distinguish empirical facts from research claims:
- Proprietary Weights vs. Open Baselines: The headline results for H-Net (2.06x OLMES, 10x data efficiency vs. SmolLM3) reflect internal Cartesia evaluations. The open-source weights currently available on GitHub (
goombalab/hnet) and Hugging Face represent the 2025 LH-Net baseline, with updated weights pending release. - In-Distribution vs. Out-of-Distribution Gains: While social media highlighted “3x to 5x gains,” in-distribution pre-training loss improvements stand at 1.28x. The extreme 3.10x to 5.37x multipliers emerge specifically on out-of-distribution code (Prolog, Fortran) where BPE catastrophic fragmentation artificially penalizes the baseline.
3-2. The Black-Box Governance Crisis of Agent Telepathy
Natural language agent communication, while inefficient, provides a transparent audit trail. Security filters, regex monitors, and human supervisors can inspect inter-agent text transcripts for prompt injections, confidential data exfiltration, or hallucinations.
When agents communicate via high-dimensional latent tensors:
- Auditability Disappears: Inter-agent signals become uninterpretable floating-point vectors, blinding traditional Data Loss Prevention (DLP) and prompt injection gateways.
- Adversarial Contamination: If an upstream vision model processes an adversarial perturbation, the corrupted latent state transfers directly into downstream executive agents without triggering textual safety filters. Enterprise deployments require dedicated latent representation monitors.
4. Structural Implications for Next-Generation AI Architecture


The convergence of H-Net and Swarm Telepathy signals three long-term shifts in AI engineering:
[Paradigm Shift 1: Input Layer]
From: Human-engineered static BPE dictionaries (128k tokens, biased toward English)
To: Raw UTF-8 byte streams + end-to-end learned dynamic chunking (universal, modality-agnostic)
[Paradigm Shift 2: Agent Interaction]
From: Lossy natural language text summaries + repeated quadratic prefill
To: Prefill-free direct KV-cache projection across frozen heterogeneous model families
[Paradigm Shift 3: System Security]
From: Inspecting plaintext conversational logs and regex filtering
To: Auditing latent vector norms and establishing explicit text-serialization trust boundaries
- Eliminating the Multilingual Tax: By moving from static subwords to adaptive byte chunking, non-English languages and proprietary programming DSLs achieve representation parity, ending the artificial token bloat imposed by English-centric vocabularies.
- The Emergence of Modular Cross-Model Adapter Ecosystems: Rather than training monolithic multi-modal models from scratch, enterprises can pair specialized frozen models (a vision specialist, a mathematical specialist, a coding specialist) via lightweight 1-GPU-day KV adapters.
- The Necessity of Latent Guardrails: Security architectures must evolve to validate vector spaces directly, establishing cryptographic boundaries where latent communication is restricted to verified clusters.
5. Practical Implementation Checklist for AI Engineering Teams

Engineering organizations can apply these insights across three practical initiatives:
| Implementation Area | Concrete Action Item | Engineering Standard |
|---|---|---|
| 1. BPE Inefficiency Audit | Profile internal domain data against current tokenizers (Llama-3, GPT-4o). | Calculate average bytes per token for internal logs, code, and non-English corpora; evaluate open H-Net (goombalab/hnet) for byte-level chunking viability. |
| 2. Multi-Agent Pipeline Partitioning | Audit inter-agent workflows to separate high-trust compute paths from external audit points. | Maintain explicit natural language serialization at compliance boundaries; implement shared KV-cache transfer across internal reasoning stages. |
| 3. Modular Adapter Prototyping | Evaluate cross-model adapter integration over full-model fine-tuning. | Train lightweight KV projection modules (<50M tokens) to connect heterogeneous frozen models, reducing infrastructure compute costs. |
Frequently Asked Questions
Why does BPE tokenization degrade performance on programming languages and DNA?
BPE builds vocabularies based on character co-occurrences in general text. Code and biological sequences contain structural hierarchies (syntax blocks, codons) that do not follow natural language spacing. BPE breaks unfamiliar syntax into tiny single-byte fragments, artificially inflating sequence lengths and diluting attention mechanisms.
How can a model understand images if it only possesses text weights?
Transformers process continuous numeric vectors. A vision model’s KV cache contains contextualized geometric and semantic representations of an image. If a lightweight adapter correctly projects these vectors into the latent dimensionality and key-value space of a text-only LLM, the receiving model processes the activations as if they were derived from prefilled text.
Can Swarm Telepathy transfer memory between completely different model families?
Yes. SwaT demonstrates transfer across 6 distinct model families (including Llama, Qwen, and Gemma). Because intermediate attention states across modern Transformers share geometric properties, a lightweight projection network trained on under 50 million tokens can align representations across architectures.
Primary Sources and References
- Cartesia Research: From Tokens to Bytes I: Scaling Hierarchical Networks — Albert Gu, Sukjun Hwang, Brandon Wang, et al. (October 28, 2026)
- Cartesia & Goomba Lab: H-Net Open Source Implementation (GitHub)
- Abdulrahman Alabdulkareem, Irene Tenison, Yang Zhang, Yada Zhu, Boris Katz, Dan Gutfreund (MIT CSAIL & MIT-IBM Watson AI Lab): Swarm Telepathy: Prefill-Free KV-Cache Transfer Across Families (arXiv:2610.15095, October 2026)
- Richard Sutton: The Bitter Lesson (Incomplete Ideas, March 13, 2019)