Consuloji: Deep Dives into Overseas AI日本語

Why Complete Isolation Crashed OpenAI's Database: Inside Habitat's 500PB Multi-Tenancy Architecture and the 12x Rust Rewrite

Published: October 30, 2026

Listen or watch on YouTube

When isolating workloads into dedicated Azure Cosmos DB containers eliminated shared headroom and caused catastrophic 10,000% capacity overruns on launch day, OpenAI returned to a hardened multi-tenant architecture paired with an AI-assisted two-person Rust rewrite.

Published: October 30, 2026 | Read time: ~10 min | Watch Video (Japanese)


Executive Summary: The Multi-Tenancy Paradox at Frontier Scale

In distributed systems design, conventional wisdom states that when noisy neighbors and cascading failures threaten a multi-tenant database, the safest remedy is complete isolation: provisioning dedicated containers or database instances per service.

This doctrine was overturned by Habitat, OpenAI’s distributed online storage engine serving over 1 billion weekly active users, 500 petabytes (PB) of state, and 70 million requests per second across ~40 geographic cloud regions.

Slide 1: Why Complete Isolation Crashed the Database - OpenAI 500PB Habitat Architecture

In late 2026 technical reports, OpenAI’s storage infrastructure group detailed an evolutionary journey through three distinct operating phases:

  1. Phase I (Primitive Shared Containers): Suffered logical partition key collisions, noisy neighbors, and “Weak Partitions”—where accumulating data splits physical partitions, halving throughput from 10,000 to 5,000 RU/s and triggering unexpected HTTP 429 errors without traffic increases.
  2. Phase II (Dedicated Container Isolation): Siloed services into dedicated containers. This broke statistical multiplexing: when new features launched, spiky traffic surged from 1,000 to 1,000,000 RU/s, overwhelming container buffers by 10,000% and crashing production.
  3. Phase III (Hardened Multi-Tenancy): Co-developed with Microsoft Azure Cosmos DB engineers, adding per-partition hot-key throttling, Quality of Service (QoS) tiers, and RU-based serverless auto-splitting, restoring a unified shared pool with zero capacity outages.

Slide 2: Overall Architecture of Habitat Supporting Over 10x Annual Growth

At the application proxy tier, OpenAI overcame Python asyncio bottlenecks—including unjittered feature-flag JSON parsing freezes and aiohttp LIFO connection pool metastable failures—before executing a complete rewrite into Rust during Q2 2026. Built by just two engineers using Codex and GPT-5.5, the Rust rewrite delivered 6x CPU efficiency and 15x memory efficiency, while slashing global p50 read latency from 120 ms to 10 ms (12x speedup) alongside a Valkey cache updated via Change Data Capture (CDC).

System Dimension Production Metric Architectural Context
Weekly Active Users Over 1 Billion Covers ChatGPT, APIs, Codex, and internal tools
Request Throughput Over 70M Requests/sec Across ~40 geographic cloud regions (90%+ reads)
Managed Data Volume Over 500 Petabytes (PB) Largest Cosmos DB account handles tens of PBs and 20M+ QPS
Proxy Implementation Python to Rust service 95% of traffic handled by Rust rewrite by 2 engineers in Q2 2026
Hardware Efficiency 6x CPU, 15x memory efficiency Replaced multi-process Python workers with native async Rust
Global Read p50 Latency ~120 ms to ~10 ms (12x speedup) Regional replicas, Valkey cache, CDC, and Rust proxy
CDC Pipeline Scale Millions of events/sec Sub-second invalidation of distributed Valkey caches

1. The Multi-Tenant Cycle: Key Collisions, Weak Partitions, and Isolation Failure

OpenAI traversed a complete cycle from shared containers to isolated silos and back to hardened shared infrastructure.

Slide 3: Cosmos DB Multi-Tenancy Evolution - Three Architectural Defects and Isolation Failure

1-1. Phase I: Three Structural Defects in Basic Shared Containers

Azure Cosmos DB provisions throughput in Request Units per second (RU/s). Data is partitioned logically by key (capped at 10,000 RU/s), mapped onto underlying physical partitions. In Phase I, three defects emerged:

Figure 1: Evolution of OpenAI Habitat Multi-Tenancy Across Three Phases

  1. Logical Key Collisions: Early APIs lacked namespaces. Disparate features—conversations, auth, and memory—chose raw keys like user_id (PK: user-28104), competing for the same 10,000 RU/s limit.
  2. Physical Partition Noisy Neighbors: Cosmos DB enforces limits at the physical partition level. When a heavy tenant saturated physical capacity, colocated neighbors suffered throttled requests, converting heavy queries into login failures.
  3. The “Weak Partition” Trap: Cosmos DB divides total container provisioned RU/s equally among physical partitions. When historical storage crosses thresholds, Cosmos DB automatically executes a partition split, dividing data across two child partitions. Because container RU/s remained constant, throughput per physical partition dropped from 10,000 RU/s to 5,000 RU/s. Workloads at 6,000 RU/s suddenly encountered persistent HTTP 429 rate-limiting without any traffic spike.

1-2. Phase II: Dedicated Containers and the 10,000% Overrun Failure

To escape noisy neighbors and splits, OpenAI moved to dedicated containers per service. While isolating failure domains, this destroyed statistical multiplexing and aggregate headroom.

Operational Metric Phase I: Shared Pool Phase II: Dedicated Container
Total Container Capacity 100,000,000 RU/s (100M RU/s) 10,000 RU/s (10K RU/s) baseline
Available Buffer (Headroom) 20,000,000 RU/s (20M RU/s) 10,000 RU/s (10K RU/s)
Pre-Launch Load (1K RU/s) Consumes 0.005% of buffer Consumes 10% of buffer
Launch Surge (1M RU/s) Consumes 5% of buffer (absorbed) Demands 10,000% of buffer (100x overrun, outage)

In a 100M RU/s shared container, a 20M RU/s operational buffer absorbs a 1M RU/s launch surge using only 5% of headroom. But in a dedicated 10K RU/s container, that same 1M RU/s surge represents a 10,000% capacity overrun, crashing the service instantly. Dedicated isolation shifted forecasting onto product teams, causing recurring launch outages.

1-3. Phase III: Co-Engineering Hardened Multi-Tenancy

OpenAI partnered with Azure Cosmos DB engineers to build three native capabilities:

  1. Per-Partition Hot-Key Throttling: Rather than throttling an entire physical partition, Cosmos DB isolates and throttles only the specific hot key, protecting colocated neighbors.
  2. Quality of Service (QoS) Tiering: Habitat assigns priority headers. Interactive user traffic receives high priority; batch ingestion receives low priority, which is throttled first under load.
  3. Enhanced Serverless Mode with RU-Based Splitting: Upgraded serverless partitions support 10,000 RU/s per partition and split automatically on sustained RU consumption rather than storage size alone.

Returning to shared multi-tenant containers under Phase III eliminated capacity-related incidents entirely.


2. Application Tier: From Python Concurrency Failures to the Two-Person Rust Rewrite

While storage stabilized, the application proxy tier underwent an equally radical evolution.

Slide 4: Battling 20M Req/s in Python - Statsig Event Loop Freezes and LIFO Metastable Failures

2-1. Strategic Technical Debt: Why Habitat Began in Python

Habitat began at DevDay 2023 as an embedded Python library. By mid-2025, coordinating library updates across dozens of downstream microservices became untenable. After a client rollback triggered a site incident, OpenAI extracted Habitat into a centralized proxy service (habitat-service).

Leadership intentionally retained Python despite GIL overhead. Prioritizing rapid API stabilization over premature optimization, leadership reasoned that when scale demanded a systems language, internal models (Codex and GPT) would automate the rewrite.

2-2. Concurrency Failures in Python at 20 Million QPS

Scaling the Python proxy to 20M+ QPS revealed three critical concurrency failures:

  1. Event Loop Starvation from Unjittered Polling: In Python asyncio, CPU-bound work blocks the event loop. Profiling showed the feature-flag SDK (Statsig) polled and parsed a massive configuration JSON once per minute without jitter across all workers simultaneously. Workers froze synchronously at the minute boundary. Adding jitter, lengthening intervals, and pruning payloads eliminated spikes.
  2. The aiohttp LIFO Metastable Failure: Under peak load, backend pods entered metastable failure loops, handling 5x to 10x more concurrent requests than healthy peers. The cause was the default Last-In, First-Out (LIFO) connection reuse policy in Python’s aiohttp TCPConnector. When backend Pod C slowed down, requests took longer and returned to the pool last. Under LIFO, clients repeatedly selected the most recently returned connection, funneling traffic into overloaded Pod C. Switching to First-In, First-Out (FIFO) reuse broke the loop.
  3. Connection Explosion and NAT Saturation: Increasing Python workers to curb loop lag created connection storms on deployments. OpenAI deployed Envoy sidecars to translate up to 18 incoming HTTP/1 worker connections into a single multiplexed HTTP/2 stream to Cosmos DB.

Slide 5: Two-Person Rust Rewrite (6x CPU, 15x Memory) and 12x Read Acceleration

2-3. The AI-Assisted Two-Person Rust Rewrite

In Q2 2026, with Habitat ranking as OpenAI’s second-largest service by CPU usage, the technical debt was repaid.

Figure 2: Global Read Latency Compression (12x) via Rust Proxy, Valkey Lookaside, and CDC

Just two software engineers completed the end-to-end rewrite of Habitat into Rust using Codex and GPT-5.5 for code conversion, test generation, and invariant checking. Today, this Rust proxy processes 95% of production storage traffic, delivering 6x greater CPU efficiency, 15x lower memory utilization, and deterministic sub-millisecond proxy transit without garbage collection pauses.

2-4. Compressing Read Latency by 12x (120 ms to 10 ms)

To accelerate user response times, Habitat paired the Rust proxy with a tiered caching strategy:

  1. Selective Cross-Region Replication: Critical small data (auth tokens, workspace metadata) is replicated across all 40 regions.
  2. Valkey Lookaside with Sub-Second CDC Invalidation: Fixed TTL caching forces a choice between stale data and low cache hit rates. Habitat deployed Valkey caches backed by Cosmos DB Change Data Capture, streaming invalidation events across Kafka within seconds globally.
  3. Zero-Overhead Rust Runtime: Combining CDC invalidation with native async Rust compressed global p50 read latency from ~120 ms to ~10 ms (a 12x improvement).

3. Architectural Boundaries, Managed Service Limits, and Data Center Megawatts

Habitat’s scale illuminates the practical boundaries of modern cloud infrastructure.

Slide 6: Conflicting Perspectives and Blind Spots - Banning SQL and Datacenter Power Constraints

3-1. Why Habitat Banned SQL

Early use of PostgreSQL showed that an unindexed query or nested join in an active code path could trigger full table scans and down the database. Habitat strictly bans arbitrary SQL, exposing only point lookups (Object by ID) and direct edge traversals (Edge by source and target). For complex analytic scans, Habitat streams mutations via CDC to isolated Rockset clusters scaled independently by product teams.

3-2. When Managed Services Break: The 2024 Failover Outages

At tens of petabytes and 20M QPS, cloud failovers can fail unexpectedly. In late 2024, an Azure regional power outage forced a failover to a secondary region. Because neither OpenAI nor Azure had tested failover at this volume, the process took several hours. One month later, failback to the restored region caused a secondary outage: clients flooded connections as physical partitions came online, overwhelming Cosmos DB gateways. OpenAI and Azure established direct engineering paging, non-production cluster testing, and client connection backoff controls.

3-3. Megawatts vs. Terabytes

At 500 PB of online state, database storage competes directly with compute for data center power and cooling. Every megawatt powering local NVMe flash storage for low latency is a megawatt unavailable for GPU/TPU accelerators. To overcome this constraint, OpenAI is developing a proprietary Exabyte-scale storage engine combining RocksDB Cloud with custom Raft physical replication, decoupling compute from storage to reduce required flash footprint.


4. Strategic Engineering Lessons and System Design Principles

Slide 7: Strategic Deferral of Technical Debt and the AI-Assisted Systems Transition

Habitat provides three core principles for engineering leaders:

  1. AI Rewrites the Economics of Technical Debt: Building in Python to rapidly iterate while requirements fluctuate, then executing an AI-assisted rewrite into Rust once patterns stabilize, is a viable architectural strategy.
  2. Resilience Demands Statistical Multiplexing: Isolating workloads into dedicated containers removes the buffer needed to absorb non-linear traffic spikes. Shared pools protected by hot-key throttling and QoS provide superior stability.
  3. The Data Center Bottleneck Has Shifted to State: As models evolve into persistent agentic systems, distributed storage matches training clusters in operational criticality and power draw.

5. Practical Implementation Checklist for High-Throughput OLTP Systems

Slide 8: Three Practical Action Items for Backend and Infrastructure Teams

Teams operating high-scale OLTP systems can apply Habitat’s principles directly:

Architecture Domain Concrete Inspection Item Production Remediation Standard
Partition Key Namespacing Do multiple services share raw entity keys (user_id) as their partition key? Enforce feature prefixes (<service>:<entity_id>) to eliminate cross-feature partition saturation.
Weak Partition Auditing In managed NoSQL, does storage accumulation automatically split physical partitions? Monitor provisioned throughput divided by partition count; enforce RU-based auto-splitting or dynamic scaling buffers.
Event Loop Lag Monitoring Do SDKs or handlers execute synchronous CPU work in async runtimes? Measure event loop lag continuously; apply randomized jitter and background thread offloading to JSON parsing.
Connection Pool Semantics Does the HTTP/RPC client connection pool use LIFO connection reuse under burst traffic? Reconfigure connection pools to FIFO reuse; deploy Envoy sidecars to multiplex HTTP/1 connections into HTTP/2 sessions.
OLTP Query Boundary Can developers run unbounded SQL queries or multi-table JOINs on transaction databases? Restrict primary OLTP paths to point lookups; route analytical queries through CDC to secondary stores.

Frequently Asked Questions

Why does Azure Cosmos DB return HTTP 429 errors when traffic has not increased?

Cosmos DB divides total container provisioned throughput (RU/s) equally among physical partitions. When historical storage growth splits a partition into two child partitions without an increase in total container throughput, the effective RU/s ceiling per child partition drops from 10,000 to 5,000 RU/s. Workloads generating 6,000 RU/s suddenly exceed the new limit, triggering storage-driven rate limiting.

Why did dedicated single-tenant containers fail during product launches?

Dedicated containers lack the statistical multiplexing of large shared pools. In a 100M RU/s shared container, a 20M RU/s buffer easily absorbs a 1M RU/s launch surge (consuming 5% of headroom). In a dedicated 10K RU/s container, that same 1M RU/s surge represents a 10,000% capacity overrun, causing instant launch-day failure.

How does LIFO connection pooling trigger metastable failure loops?

Under high load, a degraded server takes longer to respond. Its connections finish late and return to the pool last. In a LIFO pool, the most recently returned connection is immediately dispatched to the next request. Clients repeatedly route new requests to the struggling server, cementing a permanent overload loop.


Primary Sources and References

  1. OpenAI Engineering (Part I, September 11, 2026): Rapidly scaling online storage to serve over 1 billion ChatGPT users — Jon Lee, Chaomin Yu, Ben Ries
  2. OpenAI Engineering (Part II, October 23, 2026): Rapidly scaling online storage to serve over 1 billion ChatGPT users (Optimizing Azure Cosmos DB) — Jon Lee, Hrachya Hakobyan, Chaomin Yu
  3. Azure Cosmos DB Documentation: Partitioning and Horizontal Scaling in Azure Cosmos DB
  4. Meta Engineering: TAO — The power of the graph in Facebook’s data infrastructure

Back to articles