How Google Eliminates Cloud Idle Costs and Secret Leaks as AI Agent Workloads Surge 24x to 120 Quadrillion Tokens
A deep architectural analysis of Goldman Sachs token forecasts, Google TPU Sync achieving 223 GB/s, Agent Substrate sub-500ms sleep-resume cycles, and Agent Compute Engine zero-secret security.
Japanese Edition: Read the original Japanese article on Substack
Video Episode (Japanese Audio): Watch on YouTube
Executive Summary: The Structural Shift from Chat to Autonomous Agents
When enterprise IT organizations and systems integrators transition from single-turn chat to autonomous multi-step agents, they face three major infrastructure bottlenecks:
- Idle server waste during asynchronous waiting: While agents wait seconds or hours for external APIs, database queries, or human managerial sign-offs, standard Kubernetes containers hold CPU and memory allocations, burning cloud budgets on zero active compute.
- KV-cache transfer congestion: Loading tens of thousands of prompt tokens repeatedly creates massive Time-to-First-Token (TTFT) latency. Decoupling prompt processing (Prefill) from token generation (Decode) across separate accelerators bottlenecks PCIe and network buses during multi-gigabyte state transfers.
- Sandbox credential theft: Autonomous agents require code execution sandboxes. If an agent processes untrusted external data (such as emails or web pages), indirect prompt injection attacks can hijack the runtime. If long-lived database passwords or API keys reside in local environment variables, attackers can exfiltrate credentials.

Six primary industry disclosures illuminate this architectural turning point. In May 2026, Goldman Sachs Research projected global token consumption expanding 24x to 120 quadrillion tokens per month by 2030. Concurrently, Google Cloud unveiled three core infrastructure technologies:
- TPU Sync (Apache 2.0 open source, November 2026): A native C++ runtime hitting 223 GB/s over PCIe Gen5 (87.1% of the 256 GB/s ceiling), cutting KV-cache transfer delays by over 90% to ~230 ms.
- Agent Substrate 1.0 (CNCF Sandbox PR #523, November 2026): An orchestration platform achieving sub-500ms state snapshot/recovery, sustaining 500 activations/sec across 200,000 agents with 10x container packing density.
- Agent Compute Engine (ACE) (October 2026): A secure execution fabric with Titan silicon, MicroVM isolation without direct network cards, zero persistent sandbox secrets, a 4-tier identity model, and in-flight token swapping at off-node Agent Gateways.

1. Goldman Sachs Macro Forecast: The 24x Expansion to 120 Quadrillion Monthly Tokens
On May 20, 2026, Jim Schneider, Senior US Technology Equity Analyst at Goldman Sachs Research, published AI agents are forecast to boost tech cash flow as usage soars, modeling enterprise cash flows against cloud infrastructure spending.

The Mechanics of 24x Token Growth
Goldman Sachs projects global model token consumption will expand 24x to 120 quadrillion tokens per month (1.2 x 10^17 tokens) by 2030:
- Consumer Utilization (12x Growth): Conversational search and multimodal inputs grow 12x by 2030.
- Enterprise Query Growth (40% CAGR): Daily model queries compound at 40% CAGR, reaching 11 billion daily queries by 2030 as linear prompts give way to multi-turn agent execution loops.
- Knowledge Worker Penetration: Autonomous agent adoption among US and European knowledge workers is modeled at 12% in 2030, reaching 37% by 2040. Even with one in eight workers using agents by 2030, background tool loops create massive token demand.
| Metric | Goldman Sachs Research Projection | Mechanism & Strategic Implication |
|---|---|---|
| Global Monthly Tokens (2030) | 120 quadrillion / month (24x expansion) | 1.2 x 10^17 tokens across public and private foundation models |
| Consumer Usage Expansion | 12x growth by 2030 | Multimodal inputs, conversational search overhauls |
| Daily LLM Queries (2030) | 11 billion / day (40% CAGR) | Transition from single chat prompts to multi-turn agent tool loops |
| Enterprise Agent Penetration | 12% in 2030 -> 37% by 2040 | Phased adoption curve across US/EU knowledge workers |
| Inference Cost Deflation | 60% to 70% annual decline per token | Rapid silicon upgrades, custom ASICs, and precision quantization |
| Hyperscaler Gross Margin Inflection | Within 3 to 12 months | Compute cost deflation outpaces price cuts, expanding gross margins |
Annual 60% to 70% Unit Cost Deflation
Will a 24x token surge exhaust enterprise cloud budgets? Goldman Sachs demonstrates that underlying hardware cost per token drops 60% to 70% annually. Hardware generations (TPU v5 to v7x, Blackwell to Vera Rubin) and model optimizations lower compute costs faster than public API prices decline. This triggers a Gross Margin Inflection within 3 to 12 months, turning hyperscaler unit economics accretive. However, capturing this efficiency requires removing memory and networking bottlenecks.
2. Google TPU Sync: Delivering 223 GB/s and 90% KV-Cache Latency Cuts
On November 9, 2026, Google Cloud released TPU Sync (Apache 2.0 at https://github.com/google/tpu-sync), a native C++ runtime engineered with the Inferact, RadixArk, and llm-d communities to eliminate inference memory transfer bottlenecks.

The Prefill/Decode Disaggregation Dilemma
Inference separates into two phases:
- Prefill (Compute-Bound): Processing context prompts and generating Key-Value (KV) attention caches.
- Decode (Memory-Bandwidth-Bound): Generating tokens sequentially one by one from cached states.
To maximize throughput, hyperscale architectures split clusters into dedicated Prefill and Decode nodes (P/D Disaggregation) and employ Tiered Prefix Caching. In legacy Python-based frameworks, moving KV-cache tensors between accelerator High Bandwidth Memory (HBM) and host dynamic RAM (DRAM) over PCIe causes serialization stalls and memory copies, creating multi-second delays.
Direct DMA Across Four Storage Tiers
TPU Sync bypasses Python runtimes and intermediate memory allocations by executing asynchronous Direct Memory Access (DMA) directly on XLA/PJRT raw buffers across four storage tiers:
- Tier 1 (HBM): TPU on-chip memory.
- Tier 2 (Host Memory): Pinned host CPU DRAM.
- Tier 3 (Storage): Local NVMe SSDs and Google Cloud Managed Lustre file systems.
- Multi-Host DCN: Zero-copy TCP and gRPC streaming across cluster networks.
In July 2026 production benchmarks on TPU v7x (Ironwood) clusters (2x2x1 topology, tpu-inference v0.24.0):
- Host DRAM to HBM (PCIe Gen5): Reached 223 GB/s of the 256 GB/s ceiling (87.1% bus efficiency).
- Multi-Host Streaming: Achieved ~192 Gbps on single 200G NICs and ~350 Gbps on dual 200G NICs, operating near line-rate.
| Benchmark Scenario & Target Model | Legacy Framework Latency | TPU Sync Result | Measured Improvement |
|---|---|---|---|
| P/D KV Transfer Delay (Qwen3 235B) | Several seconds (Python serialization) | ~230 milliseconds | Over 90% reduction |
| P/D TTFT (Qwen3 235B) | Multi-second prompt stalls | Compressed time to initial token | 60% TTFT cut; 30% to 70% throughput gain |
| P/D TTFT (Qwen3 Coder 480B) | Inter-node buffer stalls | Raw C++ DMA streaming | Up to 90% TTFT cut |
| 64K Context Prefix Cache (Host DRAM) | Prompt recomputation or HBM overflow | 223 GB/s reload from host DRAM | 90% TTFT cut; >50% throughput gain; 30% P99 ITL cut |
| RL Weight Sync (OpenRL) | Rollout worker idle waiting | High-speed DCN parameter distribution | Near-zero rollout idle stall time |
For enterprise systems processing 64,000-token enterprise manuals or legal archives, offloading prompt caches to host DRAM preserves HBM while slashing TTFT by 90%.
3. CNCF Agent Substrate 1.0: Sub-500ms Sleep/Resume and 10x Density
While TPU Sync optimizes active inference memory, Agent Substrate 1.0 eliminates compute billing during agent idle periods.

Released on November 5, 2026, as generally available on Google Kubernetes Engine (GKE) and contributed to CNCF Sandbox (PR #523), Agent Substrate 1.0 decouples agent orchestration into three primitives:
- Actors: Stateful entities tracking conversational memory, scratchpads, and execution history.
- Gateways: Policy and routing checkpoints handling tool callbacks and user responses.
- Workers: Pre-warmed pools of stateless Kubernetes execution containers.
Eliminating Long-Tail Idle Costs
Unlike stateless microservices, autonomous agents spend over 90% of their operational lifespan waiting:
- Active Planning: Generating code or parsing inputs (2 to 5 seconds).
- Tool Waiting: Awaiting external APIs or database query returns (3 to 30 seconds).
- Human Approval: Waiting for corporate sign-off or multi-factor confirmations (minutes to hours).
Dedicated containers bill continuous compute throughout these delays. Conversely, full teardown creates 10-to-30-second cold starts.
Agent Substrate 1.0 snapshots an Actor state to storage the millisecond it enters an idle wait, immediately recycling the Worker container. When an event arrives, the Gateway injects the state into a warm Worker in under 500 milliseconds.
In production stress tests, Agent Substrate sustained 500 activations/sec across 200,000 agents, delivering a 10x increase in container packing density. Aligned with gVisor container sandboxing (CNCF PR #521), OpenRL (PR #518), and KubeRay (PR #525), it establishes an open foundation for multi-cloud agent clusters.
4. Agent Compute Engine (ACE): Zero-Secret Sandboxes and 4-Tier Identity
Enterprise deployments require strict security controls to prevent prompt injection attacks from compromising internal systems.

Google designed Agent Compute Engine (ACE) (detailed in technical releases on October 22, 2026) to power commercial Gemini infrastructure without storing credentials in execution environments.
Hardware-Rooted Isolation and No Direct NICs
ACE provisions isolated lightweight MicroVMs with robust defenses:
- Hardware Root of Trust: Titan chips and Titanium IPUs enforce cryptographic boot checks and a read-only host OS via dm-verity.
- DAX and Copy-on-Write: Shared read-only base images minimize RAM usage and enable near-instantaneous startup.
- No Direct Network Interface: MicroVMs have no direct network cards to access the intranet or public web; all traffic routes through off-node Agent Gateways.
- Zero Persistent Secrets: Neither sandbox memory nor virtual disks contain permanent credentials.
The 4-Tier Identity Hierarchy and In-Flight Token Swapping
ACE separates identity into four discrete scopes:
| Identity Tier | Specification | Scope, Storage, and Security Role |
|---|---|---|
| Tier 1: User Identity | Human End-User Credential | Validates the human employee or client delegating the task. |
| Tier 2: Delegated Agent Identity | On-Behalf-Of (OBO) Context | Scopes specific permissions granted to the agent for the task. |
| Tier 3: Agent Workload Identity (ID-3) | Ephemeral SPIFFE Certificate | Sole credential inside the sandbox. Proves process origin with zero inherent data access. |
| Tier 4: Agent Workforce Identity | Autonomous Agent ID | Corporate IAM identity for background autonomous agents, maintaining audit accountability. |
When an agent calls an external API, it transmits only its ephemeral ID-3 token. The off-node Agent Gateway validates the token, confirms the Tier 2 delegation scope, and executes an in-flight token swap—inserting the genuine enterprise API bearer token into the outbound stream. If an attacker breaches the sandbox via prompt injection, no durable secrets exist to exfiltrate.
5. Silicon Scalability: NVIDIA Vera Rubin NVL72 and Architecture Cohesion
Complementing custom TPUs, Google Cloud announced general availability for NVIDIA Vera Rubin NVL72 GPU instances (A5X) at GTC Berlin on November 6, 2026.
- Rack Architecture: 72 Rubin GPUs and 36 Vera CPUs coupled via NVLink in liquid-cooled racks.
- Virgo Network Scaling: Connects up to 80,000 Rubin GPUs in a single data center and scales up to 960,000 Rubin GPUs across multi-site federations.
- Unified Software Stack: GKE, Agent Substrate 1.0, KubeRay, and Managed Lustre operate uniformly across both TPU and GPU clusters.
This establishes an integrated multi-tier stack:
- Application Layer: Autonomous enterprise agents and multi-turn workflows.
- Orchestration Layer: Agent Substrate 1.0 (sub-500ms sleep/resume) and ACE (zero-secret MicroVMs and in-flight token swapping).
- State Acceleration Layer: TPU Sync (223 GB/s DMA over PCIe Gen5).
- Physical Silicon Fabric: Google TPU v7x and NVIDIA Vera Rubin NVL72 A5X instances.
(Note: Commercial pricing for standalone ACE MicroVM instances and hourly rates for A5X Vera Rubin instances were not disclosed in launch documentation and remain unverified at this publication date.)

6. Enterprise Implementation Playbook: 3 Strategic Actions
For IT leaders, systems integrators, and software architects preparing for exponential token expansion, three practical actions should be prioritized:

Action 1: Implement Tiered Prefix Caching to Host DRAM
Stop pinning static prompt KV-caches in scarce accelerator HBM.
- Execution: Deploy tiered prefix caching via open-source TPU Sync or equivalent connectors in vLLM, SGLang, or llm-d.
- Impact: Offloading prompt caches to host DRAM slashes TTFT by up to 90% and lifts throughput by over 50%.
Action 2: Decouple Agent State from Compute Workers
Eliminate compute costs incurred while agents wait for external callbacks.
- Execution: Adopt CNCF Agent Substrate 1.0. Store state in logical Actors and run tasks on warm, ephemeral Worker pools.
- Impact: Sub-500ms state recovery increases container packing density by 10x, curbing unnecessary cloud spend.
Action 3: Purge Static Credentials from Code Sandboxes
Harden agent execution environments against indirect prompt injection exploits.
- Execution: Remove static database passwords and service keys from container environments. Issue short-lived SPIFFE workload tokens and route outbound traffic through off-node gateways that perform in-flight token swapping.
- Impact: Sandboxes contain zero durable credentials, structurally eliminating data exfiltration risks.
7. Primary Source Documentation
- Goldman Sachs Research (May 20, 2026): AI agents are forecast to boost tech cash flow as usage soars, Jim Schneider: Goldman Sachs Insights Article
- Google Cloud Blog (November 9, 2026): Introducing TPU Sync: Fast KV cache and weight transfer on TPU: Google Cloud Infrastructure Blog
- Google Cloud Blog (November 5, 2026): Open infrastructure for autonomous agents comes to CNCF: Google Cloud Containers Blog
- Google Cloud Blog (October 22, 2026): Introducing Agent Compute Engine and expanded agent governance: Google Cloud AI Infrastructure Blog
- Google Cloud Blog (October 22, 2026): Inside Agent Compute Engine: Building the compute foundation for the agentic era: Google Cloud Deep Dive Blog
- Google Cloud Blog (November 6, 2026): Google Cloud infrastructure at NVIDIA GTC Berlin 2026: Google Cloud Hardware Announcements