Consuloji: Deep Dives into Overseas AI日本語

Inside OpenAI's Pando: How Git Serves 1 Petabyte per Day Without Monorepo Lock Contention

Published: November 3, 2026

Listen or watch on YouTube

A forensic engineering teardown of OpenAI’s Pando backend, trading global cross-branch consistency for write scalability, and how to audit whether your infrastructure faces the same bottleneck.

Japanese Edition: Read the original Japanese article on Substack

Video Episode (Japanese Audio): Watch on YouTube


Executive Summary & Core Engineering Thesis

On November 2, 2026, OpenAI published an infrastructure teardown titled “Scaling Git at OpenAI: Building agentic-scale source control while investing in the Git ecosystem.” Authored by engineering staff Aaron Friel, Taylor Blau, and Ted Nyman, the post introduces Pando, a distributed storage backend designed to sustain massive code generation throughput driven by automated AI coding agents.

The core conclusions of the technical disclosure center on three architectural realities:

  1. The Bottleneck Was Distribution Architecture, Not the Git Protocol: OpenAI’s engineering bottleneck resided in server-side monorepo coordination rather than inherent limitations of the Git protocol or data structures. By preserving full Git client and wire-protocol compatibility, OpenAI avoided forcing developers and automated tooling to abandon standard Git workflows.
  2. Trading Global Cross-Branch Ordering for Horizontal Scalability: To eliminate lock contention across concurrent pushes, Pando discarded the assumption of instantaneous global repository-wide consistency. Pando enforces strict causal consistency within individual branches while permitting concurrent pushes to unrelated branches to commit independently without global synchronization.
  3. Optimizing Reads via Immutable Blob Caching: Rather than synthesizing customized packfiles for each client request, Pando serves static, reusable packfiles via regional caching proxies (Express), enabling Git read operations to distribute nearly one petabyte of data per day with minimal CPU utilization.

For engineering leaders managing enterprise monorepos, developer tooling, and automated CI pipelines, Pando provides a case study in systems scalability. However, adopting similar designs requires evaluating whether internal workloads genuinely suffer from write-side lock contention or whether bottlenecks stem from unoptimized read caches.

Slide 1: What the Pando Disclosure Stated and Omitted


Primary Sources & Investigation Scope

This analysis cross-examines two primary technical disclosures retrieved on November 3, 2026:

  • S1. OpenAI Engineering Post (November 2, 2026): “Scaling Git at OpenAI: Building agentic-scale source control while investing in the Git ecosystem” by Aaron Friel, Taylor Blau, and Ted Nyman (OpenAI Index).
  • S2. Anthropic Engineering Post (October 13, 2026): “Redesigning source control at Anthropic” by Anthropic Engineering (Anthropic Blog).

Slide 2: Three Inquiries for Fact-Checking the Pando Architecture

To maintain rigorous analytical separation:

  • Disclosed Production Metrics: Numbers and architectural specifications explicitly published by the authors.
  • Self-Reported Limitations: Acknowledged boundaries, synthetic simulation models, and omitted operational data.
  • Comparative Analysis: Contrasting OpenAI’s protocol-compatible approach against alternative custom version control engines.

1. The Bottleneck: Agentic Acceleration and Monorepo Degradation

Slide 3: Root Cause of the Monorepo Scaling Bottleneck

The Catalyst: Codex and High-Velocity AI Concurrency

OpenAI attributes the breakdown of its legacy source control infrastructure directly to the widespread adoption of AI coding tools such as Codex. In traditional software organizations, developer push frequency is bounded by human typing and context-switching speeds. In an agentic development paradigm:

  • A single human engineer directs multiple autonomous coding agents executing in parallel.
  • Each agent generates discrete branches, submits rapid automated commits, and triggers independent CI verification cycles.
  • A single upstream commit fans out into hundreds of concurrent automated pull requests and test jobs, creating severe read-side amplification.

Infrastructure Breakdown Under +1,378% Growth

Beginning in late 2025, OpenAI’s legacy monorepo storage layer began degrading under exponential load. Production metrics charted in the post establish the trajectory:

  • Baseline commit volume was indexed to the first quarter of 2024.
  • By the third quarter of 2026, cumulative commits landing on the master branch increased by +1,378% (with internal models projecting +2,442% by the fourth quarter of 2026).
  • As load peaked, fetch and push latencies escalated dramatically, merge jobs frequently timed out, and operational reliability periodically dropped below 90%.

The authors formulated the central design question:

“Had we reached a limit of Git itself, or of the way we served our monorepo?”

Crucially, OpenAI determined that Git’s underlying Merkle tree structure remained sound. The failure stemmed from centralized server architectures attempting to serialize all concurrent operations.


2. Architectural Design: Independent Branch Writes and Packfile Reuse

Slide 4: Pando Architecture — Independent Writes and Packfile Reuse

OpenAI established two operational objectives for Pando:

  1. Horizontally Scalable Writes: Enabling thousands of concurrent pushes to separate branches without blocking on centralized mutex locks.
  2. Maximized Read Throughput: Enabling clients and CI runners to saturate 10 Gbps network interfaces without server CPU exhaustion.

Overcoming Legacy Coordination Bottlenecks

The post contrasts Pando against two conventional distributed storage paradigms:

  • Three-Phase Commit (3PC): Distributes references across replicas via multi-round consensus protocols. Updating individual references incurs high inter-node communication latency, causing write queues to stall under heavy concurrency.
  • Write-Ahead Logging (WAL): Serializes all reference updates into a single monolithic log. Pushes to completely unrelated feature branches queue sequentially behind one another, creating an unavoidable single-threaded choke point.

The Two Foundational Assumptions Pando Discarded

To scale beyond these limits, Pando abandoned two assumptions embedded in standard Git server designs:

  1. Global Repository-Wide Consistency: Traditional Git servers present a linear, unified snapshot of all repository references at any given nanosecond. Achieving this requires global synchronization. Pando discarded global cross-branch ordering while strictly preserving causal consistency within each individual branch. As the authors observe:

    “users and agents are agnostic to the global ordering of ref updates.”

  2. Per-Client Dynamic Packfile Generation: Standard Git servers compute customized packfile deltas on the fly based on what the client claims to have. Under thousands of concurrent CI fetches, dynamic packfile creation consumes vast CPU and disk I/O. Pando replaces dynamic synthesis with static, pre-packaged immutable blobs.

The Pando Runtime Mechanism

  • Ingestion: When an agent pushes changes, new Git objects are assembled into an immutable packfile and stored immediately in a horizontally scalable blob storage system.
  • Reference Metadata: Reference updates are written as versioned records to a distributed metadata store. Because writes to distinct branches do not share state, write capacity scales linearly with storage nodes.
  • Read Distribution via Express: During fetch operations, clients receive direct reference URLs (packfile-uri) pointing to pre-computed packfiles. These requests are fulfilled by Express, a regionally distributed caching layer.
  • Trunk-Only Advertisement: By advertising only the primary trunk branch during routine pushes, packfile sizes are dramatically compressed: median push sizes remain under 20 KiB, with the 99th percentile measuring under 3 MiB.

3. Dissecting Production Benchmarks: Fact vs. Interpretation

Slide 5: Dissecting Production Benchmark Metrics

The performance figures cited in the announcement must be parsed across their precise operational boundaries:

Production Benchmark Breakdown

Disclosed Benchmark Exact Description in Text Operational Boundary & Technical Context
~5x Latency Reduction Reduced median push latency by nearly 5x Median write-side only. P99 push latency is omitted from production text
~80% Latency Drop Reduced median push latency by nearly 80% Mathematically consistent with 5x speedup (1/5 duration = 80% reduction)
~1 PB / Day Distributed Serves nearly 1 petabyte of Git data daily Read-side distribution volume, sustained with minimal compute cores
>57 TB / Hour Peak Distributed over 57 TB of packfiles in 1 hour Single monorepo peak throughput; >99% served from local node disk caches
>99% Cache Hit Rate High local disk cache hit rates on Express Validates that CI workloads repeatedly fetch identical commit packfiles
10 Gbps Saturation Clone and fetch operations saturate 10 Gbps Demonstrates network-bound read throughput on standard cloud VMs

Critical Analytical Caveats

  1. Median vs. Tail Latency: The ~5x speedup applies strictly to median (p50) push operations. Tail latencies (p99 or p99.9), which dictate developer frustration during contention spikes, are not disclosed for production workloads.
  2. Read Dominance: The headline figure of “nearly 1 PB per day” measures read distribution to automated CI workers, not incoming push volume. Pando functions effectively as a specialized Content Delivery Network (CDN) for Git blobs.
  3. Synthetic Teaching Models: The post includes simulation charts comparing Pando against 3PC and WAL models under 1,000 concurrent client pushes. Crucially, the authors explicitly note:

    “Synthetic teaching models, not production benchmarks.”

  4. Self-Reported Data: All metrics represent internal telemetry published by OpenAI without independent third-party verification.

4. Omissions and Unanswered Architectural Inquiries

Slide 6: Crucial Operational Dimensions Omitted from the Post

A systematic inventory of undisclosed operational details highlights key boundaries:

  • No Open Source Release or Commercial Timeline: The authors state: “We’re currently exploring ways to share the technology that makes Pando possible with our users.” There is no repository, no open-source license, and no committed release date. Upstream Git improvements submitted by OpenAI engineers (such as Taylor Blau) remain distinct from Pando itself.
  • Infrastructure Financial Costs: The post discloses no data regarding the compute, network egress, or storage costs required to operate Pando and its distributed blob storage layer.
  • Failure Modes & Conflict Resolution: While general merge reliability improved, the post provides no quantitative metrics detailing merge conflict resolution or recovery protocols when metadata stores experience transient partitions.
  • Debunking Third-Party Rumors: External summaries frequently conflate Pando with distributed databases or claim custom SSH protocols. The original text mentions no SSH protocols, no custom database products, and makes no mention of rival laboratories or external models.

5. Strategic Comparison: OpenAI (Pando) vs. Anthropic (Scion)

Slide 7: Strategic Comparison — Protocol Compatibility vs. Interaction Redesign

The divergence between OpenAI and Anthropic illustrates two contrasting engineering philosophies when confronting agentic-scale source control:

Comparative Engineering Philosophies

Structural Dimension OpenAI Pando (S1) Anthropic Scion (S2)
Core Architecture Distributed Git-compatible storage backend Custom proprietary version control system
Workflow Interaction Preserves standard git push, git fetch, git clone Eliminates client-side “push”; commits created server-side
Protocol Compatibility 100% wire-protocol and client compatible Incompatible with standard Git clients
Scale Driver +1,378% master commits; ~1 PB/day read distribution Daily merges increased from ~2,500 (May 2026) to >30,000 (Sept 2026)
Implementation Scope Rebuilt backend storage, metadata, and proxy caching Migrated entire company off Git in 5 months
Target Applicability Exploring external sharing mechanisms “By no means a suitable replacement for git for general use”

The Architectural Trade-Off

OpenAI prioritized ecosystem preservation. By retaining standard Git protocols, OpenAI ensured that existing developer muscle memory, IDE integrations, and third-party developer tools remained completely functional. The cost was engineering a complex distributed storage and cache-invalidation layer to bypass Git’s legacy synchronization bottlenecks.

Conversely, Anthropic concluded that the client-side push model was fundamentally mismatched with autonomous agents, choosing to invent a server-centric paradigm (Scion). Both labs demonstrate that at hyperscale, standard out-of-the-box Git servers fail—but the choice between protocol compatibility and interaction redesign depends on organizational priorities.


6. Actionable Audit Protocol for Enterprise Monorepo Teams

Slide 8: Three Mandatory Inquiries for Monorepo Infrastructure Leads

Before assuming your organization requires a bespoke distributed backend like Pando, execute three diagnostic audits:

1. Deconstruct Read Bottlenecks vs. Write Contention

  • Measure your telemetry independently: track median and 99th percentile push durations separately from fetch/clone times.
  • If 90% of latency stems from CI build runners pulling identical commits, deploying localized caching proxies or Git mirrors will resolve the issue at a fraction of the complexity of a distributed backend.

2. Audit Cross-Branch Snapshot Dependencies

  • Inventory internal deployment pipelines, security scanners, and release tools.
  • Verify whether any mission-critical systems rely on an instantaneous, globally consistent snapshot of all repository branches simultaneously. If your tools require global ordering, adopting a causally consistent architecture like Pando will break downstream assumptions.

3. Exhaust Upstream Git Optimizations First

  • Benchmark recent upstream Git enhancements—many contributed by the Pando team itself—such as commit-graph generation, multi-pack indexes, sparse checkouts, and shallow cloning, before considering custom storage replacements.

Primary References & Source Directory

  • S1. OpenAI Engineering Post: “Scaling Git at OpenAI: Building agentic-scale source control while investing in the Git ecosystem” by Aaron Friel, Taylor Blau, and Ted Nyman (November 2, 2026) (Link).
  • S2. Anthropic Engineering Post: “Redesigning source control at Anthropic” by Anthropic Engineering (October 13, 2026) (Link).

Back to articles