Why a 20-Billion-Parameter AI Runs on Just 2GB of RAM: Inside Apple AFM 3 Core Advanced Flash-Driven Sparse Inference
By pairing Instruction-Following Pruning with NAND flash streaming techniques from LLM in a Flash, Apple executes a 20B multi-modal model by dynamically patching 1B to 4B parameters onto DRAM shared experts.
Published: October 30, 2026 | Read time: ~10 min | Watch Video (Japanese)
Executive Summary: Breaking the Active Memory Barrier in Edge AI
Deploying large language models (LLMs) on edge consumer devices—such as smartphones and laptops—has historically hit an immovable hardware limit: the capacity of active system memory (DRAM).
Traditional model serving requires that all parameters reside permanently in RAM prior to inference. A 20-billion-parameter (20B) model demands approximately 40GB in 16-bit precision and over 10GB even under aggressive 4-bit quantization. In consumer devices where memory is shared with the operating system and applications, dedicating 10GB exclusively to an AI model is impractical.

In late 2026, Apple Machine Learning Research published technical reports detailing the third-generation Apple Foundation Models (AFM 3) family alongside the underlying framework LLM in a Flash (ACL 2024).
Central to this release is AFM 3 Core Advanced, an on-device 20B sparse foundation model. Rather than holding 20 billion weights in DRAM, the model stores its full parameter matrix on internal NAND flash storage. Utilizing Instruction-Following Pruning (IFP), the system evaluates requests at the prompt level, loading only 1 billion to 4 billion (1B to 4B) parameters into memory to dynamically synthesize an ephemeral compact model over a base of always-active shared experts.

Supported by foundational techniques from LLM in a Flash—specifically Windowing (reusing active neurons across a sliding window) and Row-Column Bundling (storing feed-forward weights contiguously for sequential flash throughput)—this architecture allows devices to execute models up to 2x larger than available DRAM, accelerating inference by 4x to 5x on CPU and 20x to 25x on GPU relative to naive offloading.
| Model Tier | Deployment Target | Architecture & Scale | Target Hardware | Primary Role & Benchmarks |
|---|---|---|---|---|
| AFM 3 Core | On-Device (Standard) | 3B Dense | Apple Silicon | High-efficiency 3B baseline; 45.6% vs 23.3% preference over 2025 on-device model. |
| AFM 3 Core Advanced | On-Device (Flagship) | 20B Sparse (1B–4B active) | Capable Apple Silicon | IFP sparse model storing weights on NAND; powers expressive speech (MOS 4.24) and dictation. |
| AFM 3 Cloud | Private Cloud Compute | Parallel-Track MoE | Apple Silicon Servers | Primary cloud model; 64.7% vs 8.7% preference over 2025 server model on complex tasks. |
| ADM 3 Cloud (Image) | Private Cloud Compute | Diffusion Backbone | Apple Silicon Servers | Drives Image Playground and Photos spatial reframing via modular task adapters. |
| AFM 3 Cloud Pro | Extended PCC Fleet | Advanced Reasoning | Google Cloud NVIDIA GPUs | High-tier reasoning extending PCC privacy to NVIDIA clusters; +14% math accuracy over Cloud. |

1. Why Standard MoE Fails on Flash Storage: The Mechanics of IFP
To understand Apple’s architecture, one must examine why conventional Mixture-of-Experts (MoE) cannot operate directly from flash memory.

1-1. The Routing Granularity Mismatch
Standard MoE models (such as Mixtral or DeepSeek) route computation dynamically at the token level. For each newly generated token, gate routers dispatch computation to different combinations of experts (e.g., Experts 3 and 12 for Token 1; Experts 7 and 19 for Token 2).
While computationally efficient, this fine-grained routing fails when weights reside on NAND flash:
- Bandwidth Disparity: Internal NAND read speeds (3 to 7 GB/s over mobile NVMe lanes) are two orders of magnitude slower than unified DRAM bandwidth (100 to 800 GB/s on Apple Silicon).
- The Swapping Trap: Generating 30 tokens per second would require swapping gigabytes of expert weights from flash to memory thirty times a second. Because physical transfer latency vastly exceeds compute time, token-level MoE collapses into permanent I/O wait states.
1-2. Instruction-Following Pruning (IFP): Synthesizing Ephemeral Dense Models
Apple solved this structural bottleneck through Instruction-Following Pruning (IFP), shifting expert selection from token-by-token gating to prompt-level synthesis:
- NAND Storage Residency: The 20B parameter matrix of
AFM 3 Core Advanced(primarily stacked feed-forward expert weights) resides on NAND flash rather than DRAM. - Per-Prompt Routing: When a user submits an instruction, an initial lightweight dense block analyzes the request and selects a dedicated subset of routed experts upfront.
- Patching onto Shared Experts: The model maintains a permanent core of always-active shared experts in DRAM. Selected routed experts are read from flash in a single sequential burst and patched onto the shared backbone, forming an ephemeral dense model (1B to 4B parameters) in memory.
- Periodic Reselection & Elasticity: During generation, the model decodes at full unified memory speed without swapping weights per token. Experts are updated only at periodic macro-intervals if context shifts. Furthermore, the system implements Inference-Time Elasticity, dynamically sizing active parameters from 1B for audio to 4B for complex text extraction.
2. Foundations in Hardware: LLM in a Flash and AFM 3 Empirical Benchmarks

2-1. Algorithmic Innovations in Flash Streaming
The operational viability of AFM 3 Core Advanced relies on two algorithms formalized in Apple’s ACL 2024 paper LLM in a Flash:
- Windowing (Data Volume Compression): Neural activations exhibit strong temporal locality; neurons active during recent tokens are likely to remain active in subsequent tokens. Windowing maintains a sliding window of recent neuron activations in DRAM, fetching only the incremental delta of newly activated neurons from flash. This drastically reduces the total volume of bytes transferred per step.
- Row-Column Bundling (Sequential I/O Optimization): Flash storage achieves peak read throughput only when reading large, contiguous data blocks; small, random 4KB reads suffer severe degradation. In Transformer FFN layers, the $i$-th row of the Up-projection matrix and the $i$-th column of the Down-projection matrix correspond to the same intermediate neuron and are always activated together. Row-Column Bundling physically co-locates these corresponding row and column slices as contiguous blocks on NAND, doubling chunk size and maximizing flash read throughput.
Combined with Quantization-Aware Training (QAT), these techniques enable devices to run models double the size of available DRAM while achieving 4x to 5x faster CPU inference and 20x to 25x faster GPU inference compared to naive offloading.
2-2. Comprehensive Empirical Evaluation of the AFM 3 Family


Apple’s evaluation suite provides detailed metrics across three key modalities:
1. Blind Side-by-Side Human Preference (General Text)
Professional evaluators conducted blind side-by-side evaluations measuring instruction following, factual truthfulness, and presentation quality against 2025 baseline models:
| Evaluation Setting | Evaluated Model Comparison | AFM 3 Win Rate | Tie Rate | 2025 Baseline Win Rate |
|---|---|---|---|---|
| Overall Prompts | AFM 3 Core vs 2025 On-Device (3B) | 45.6% | — | 23.3% |
| English Locales | AFM 3 Core vs 2025 On-Device | 38.0% | 39.0% | 23.0% |
| PFIGSCJK (incl. Japanese) | AFM 3 Core vs 2025 On-Device | 43.0% | 31.0% | 26.0% |
| DDNSTV (Nordic/SE Asian) | AFM 3 Core vs 2025 On-Device | 42.4% | 28.3% | 29.3% |
| AFIHHMPRTU (Global 10) | AFM 3 Core vs 2025 On-Device | 55.5% | 26.2% | 18.3% |
| Overall Prompts | AFM 3 Cloud vs 2025 Server | 64.7% | — | 8.7% |
| English Locales | AFM 3 Cloud vs 2025 Server | 56.0% | 33.0% | 11.0% |
| PFIGSCJK (incl. Japanese) | AFM 3 Cloud vs 2025 Server | 68.3% | 24.8% | 6.9% |
| DDNSTV (Nordic/SE Asian) | AFM 3 Cloud vs 2025 Server | 68.6% | 22.8% | 8.6% |
| AFIHHMPRTU (Global 10) | AFM 3 Cloud vs 2025 Server | 66.6% | 23.1% | 10.3% |
In single-sided evaluations, AFM 3 Cloud achieved a 36% relative gain in overall response satisfaction and a 21% gain in instruction following over 2025 server baselines. The premier AFM 3 Cloud Pro model extended this further, delivering a 10% satisfaction gain, 14% image understanding gain, and 14% mathematical reasoning gain over AFM 3 Cloud.
2. Image Understanding Human Evaluation
- AFM 3 Core vs 2025 On-Device: 26.6% win rate vs 16.8% baseline (56.6% tie rate). When non-tie outcomes occurred, AFM 3 Core won over 61% of matchups.
- AFM 3 Cloud vs 2025 Server: 37.8% win rate vs 9.6% baseline (52.6% tie rate).
3. AFM 3 Core Advanced Native Audio & Dictation (1B Active Setting)
Operating at its minimum 1B active parameter configuration, AFM 3 Core Advanced demonstrated exceptional quality:
- Expressive Speech Synthesis (TTS 5-Point MOS):
- General Voice: Improved from production baseline 3.87 to 4.15 (+0.28 MOS points).
- Conversational Voice: Improved from baseline 3.82 to 4.24 (+0.42 MOS points). A 0.1 MOS shift represents a clearly noticeable qualitative difference in human listening tests.
- Dictation Transcription Accuracy (vs. Legacy System):
- Overall Quality: 44.7% preference vs 17.6% legacy (37.7% tie).
- Punctuation Placement: 50.2% preference vs 12.4% legacy.
- Meaning Retention: 23.0% preference vs 13.9% legacy (63.0% tie).
- Disfluency Handling: 22.3% preference vs 10.6% legacy (67.2% tie).
3. Industry Debate and Operational Blind Spots
The release of AFM 3 crystallizes two competing visions for AI infrastructure.

3-1. Megacluster Centralization vs. Flash-Driven Edge
- Cloud Centralization: Centralizing inference on datacenter HBM GPU clusters provides massive scale. Apple embraced this for extreme reasoning by partnering with Google and NVIDIA to deploy
AFM 3 Cloud Proon Google Cloud NVIDIA clusters within the Private Cloud Compute (PCC) boundary. - Flash-Driven Edge: Conversely, running real-time dictation, responsive voice synthesis, and personal data summarization entirely in the cloud incurs latency, infrastructure cost, and privacy exposure.
3-2. Three Technical Trade-Offs in Practice
- Mid-Prompt Domain Shift Latency: IFP fixes the active expert set per prompt. Abrupt domain shifts (e.g., casual speech to legal analysis) force mid-generation expert swapping, introducing brief latency pauses.
- Pre-Processing and QAT Requirements: Weights require offline conversion to Row-Column Bundled layouts and Quantization-Aware Training (QAT) to preserve numerical accuracy.
- Hardware Boundaries:
AFM 3 Core Advancedis optimized for modern Apple Silicon architectures equipped with high unified memory bandwidth and fast NVMe storage controllers.
4. Strategic Implications: NAND Flash as Second-Tier Unified Memory

Apple’s architecture signals three fundamental transitions in systems design:
- NAND Flash Becomes Active Memory: Rather than serving as passive storage, NAND flash operates as an active execution tier streaming weight patches directly into DRAM compute paths.
- Edge-Cloud Memory Symmetry: Just as DeepSeek streams KV caches from 100 PiB SSD clusters in the cloud, Apple streams model weights from NAND flash at the edge. Both break hardware cost barriers using algorithmic sparsity and contiguous flash layouts.
- The End of Model Ensembles: Maintaining multiple on-device models clutters storage. Single elastic foundation models dynamically synthesized from a 20B knowledge base provide superior quality while economizing silicon footprint.
5. Practical Implementation Checklist for Edge AI Engineering

Engineering teams developing edge and hybrid AI applications can apply these lessons through a structured audit framework:
| Engineering Focus | Concrete Inspection Item | Production Standard |
|---|---|---|
| 1. Memory Budget Estimation | Do system requirements assume DRAM must equal total model parameter footprint? | Calculate memory using active parameters (1B to 4B) plus baseline shared weights and Windowing buffers rather than full parameter volume. |
| 2. Storage Layout Optimization | Are model weights loaded from storage using fragmented, random 4KB reads? | Reorganize FFN layers into contiguous Row-Column Bundles to maximize sequential flash read throughput during dynamic loading. |
| 3. Prompt Boundary Design | Are disparate multi-domain tasks combined into single, monolithic prompts? | Partition workflows into single-domain prompts to prevent mid-generation expert reselection and avoid flash swapping delays. |
| 4. Two-Tier Routing Topology | Does the application push all requests to cloud APIs or restrict everything to on-device? | Deploy hybrid routing: handle sensitive, low-latency tasks (speech, UI, summarization) on-device; escalate complex reasoning to verifiable private cloud infrastructure. |
Frequently Asked Questions
Does streaming weights from NAND flash degrade SSD lifespan through drive wear?
No. Inference operations are read-only. NAND flash degradation occurs during program/erase (write) cycles. Reading weights from flash to patch DRAM memory does not consume flash write endurance (TBW), allowing safe long-term operation.
Why is standard 4-bit quantization insufficient for running 20B models on mobile devices?
Quantizing a 20B model to 4 bits reduces storage to approximately 10GB to 12GB. While significantly smaller than 40GB FP16, a 10GB footprint still exceeds the active RAM budget available to background processes on mobile operating systems. Only sparse activation (reducing active weights to 1B to 4B) brings RAM consumption down to 2GB.
What is the advantage of activating 1B parameters from a 20B model over using a native 1B model?
A native 1B model possesses the representational capacity of only 1 billion weights. A 20B sparse model operating with 1B active parameters draws from a 20-billion-parameter pre-trained knowledge base. The specialized routed experts provide significantly higher nuance, resulting in measurable quality advantages such as MOS 4.24 conversational voice quality.
Primary Sources and References
- Apple Machine Learning Research: Introducing the Third Generation of Apple’s Foundation Models (June 8, 2026 / Updated September 9, 2026)
- Apple Machine Learning Research (ACL 2024): LLM in a Flash: Efficient Large Language Model Inference with Limited Memory — Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, S. Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, Mehrdad Farajtabar
- Apple Machine Learning Research: Private Cloud Compute: A new frontier for AI privacy in the cloud