Consuloji: Deep Dives into Overseas AI日本語

How Google Eliminates Cloud Idle Costs and Secret Leaks as AI Agent Workloads Surge 24x to 120 Quadrillion Tokens

Published: November 12, 2026

Listen or watch on YouTube

A deep architectural analysis of Goldman Sachs token forecasts, Google TPU Sync achieving 223 GB/s, Agent Substrate sub-500ms sleep-resume cycles, and Agent Compute Engine zero-secret security.

Japanese Edition: Read the original Japanese article on Substack

Video Episode (Japanese Audio): Watch on YouTube


Executive Summary: The Structural Shift from Chat to Autonomous Agents

When enterprise IT organizations and systems integrators transition from single-turn chat to autonomous multi-step agents, they face three major infrastructure bottlenecks:

  1. Idle server waste during asynchronous waiting: While agents wait seconds or hours for external APIs, database queries, or human managerial sign-offs, standard Kubernetes containers hold CPU and memory allocations, burning cloud budgets on zero active compute.
  2. KV-cache transfer congestion: Loading tens of thousands of prompt tokens repeatedly creates massive Time-to-First-Token (TTFT) latency. Decoupling prompt processing (Prefill) from token generation (Decode) across separate accelerators bottlenecks PCIe and network buses during multi-gigabyte state transfers.
  3. Sandbox credential theft: Autonomous agents require code execution sandboxes. If an agent processes untrusted external data (such as emails or web pages), indirect prompt injection attacks can hijack the runtime. If long-lived database passwords or API keys reside in local environment variables, attackers can exfiltrate credentials.

Slide 1: Cloud Idle Costs and the Autonomous Agent Bottleneck

Six primary industry disclosures illuminate this architectural turning point. In May 2026, Goldman Sachs Research projected global token consumption expanding 24x to 120 quadrillion tokens per month by 2030. Concurrently, Google Cloud unveiled three core infrastructure technologies:

  • TPU Sync (Apache 2.0 open source, November 2026): A native C++ runtime hitting 223 GB/s over PCIe Gen5 (87.1% of the 256 GB/s ceiling), cutting KV-cache transfer delays by over 90% to ~230 ms.
  • Agent Substrate 1.0 (CNCF Sandbox PR #523, November 2026): An orchestration platform achieving sub-500ms state snapshot/recovery, sustaining 500 activations/sec across 200,000 agents with 10x container packing density.
  • Agent Compute Engine (ACE) (October 2026): A secure execution fabric with Titan silicon, MicroVM isolation without direct network cards, zero persistent sandbox secrets, a 4-tier identity model, and in-flight token swapping at off-node Agent Gateways.

Slide 2: Sub-500ms Sleep and Fast State Relocation


1. Goldman Sachs Macro Forecast: The 24x Expansion to 120 Quadrillion Monthly Tokens

On May 20, 2026, Jim Schneider, Senior US Technology Equity Analyst at Goldman Sachs Research, published AI agents are forecast to boost tech cash flow as usage soars, modeling enterprise cash flows against cloud infrastructure spending.

Slide 3: Why Profit Margins Improve Despite 24x Usage Growth by 2030

The Mechanics of 24x Token Growth

Goldman Sachs projects global model token consumption will expand 24x to 120 quadrillion tokens per month (1.2 x 10^17 tokens) by 2030:

  • Consumer Utilization (12x Growth): Conversational search and multimodal inputs grow 12x by 2030.
  • Enterprise Query Growth (40% CAGR): Daily model queries compound at 40% CAGR, reaching 11 billion daily queries by 2030 as linear prompts give way to multi-turn agent execution loops.
  • Knowledge Worker Penetration: Autonomous agent adoption among US and European knowledge workers is modeled at 12% in 2030, reaching 37% by 2040. Even with one in eight workers using agents by 2030, background tool loops create massive token demand.
Metric Goldman Sachs Research Projection Mechanism & Strategic Implication
Global Monthly Tokens (2030) 120 quadrillion / month (24x expansion) 1.2 x 10^17 tokens across public and private foundation models
Consumer Usage Expansion 12x growth by 2030 Multimodal inputs, conversational search overhauls
Daily LLM Queries (2030) 11 billion / day (40% CAGR) Transition from single chat prompts to multi-turn agent tool loops
Enterprise Agent Penetration 12% in 2030 -> 37% by 2040 Phased adoption curve across US/EU knowledge workers
Inference Cost Deflation 60% to 70% annual decline per token Rapid silicon upgrades, custom ASICs, and precision quantization
Hyperscaler Gross Margin Inflection Within 3 to 12 months Compute cost deflation outpaces price cuts, expanding gross margins

Annual 60% to 70% Unit Cost Deflation

Will a 24x token surge exhaust enterprise cloud budgets? Goldman Sachs demonstrates that underlying hardware cost per token drops 60% to 70% annually. Hardware generations (TPU v5 to v7x, Blackwell to Vera Rubin) and model optimizations lower compute costs faster than public API prices decline. This triggers a Gross Margin Inflection within 3 to 12 months, turning hyperscaler unit economics accretive. However, capturing this efficiency requires removing memory and networking bottlenecks.


2. Google TPU Sync: Delivering 223 GB/s and 90% KV-Cache Latency Cuts

On November 9, 2026, Google Cloud released TPU Sync (Apache 2.0 at https://github.com/google/tpu-sync), a native C++ runtime engineered with the Inferact, RadixArk, and llm-d communities to eliminate inference memory transfer bottlenecks.

Slide 4: TPU Sync: 223 GB/s Direct Transfer Cutting Memory Latency by Over 90%

The Prefill/Decode Disaggregation Dilemma

Inference separates into two phases:

  1. Prefill (Compute-Bound): Processing context prompts and generating Key-Value (KV) attention caches.
  2. Decode (Memory-Bandwidth-Bound): Generating tokens sequentially one by one from cached states.

To maximize throughput, hyperscale architectures split clusters into dedicated Prefill and Decode nodes (P/D Disaggregation) and employ Tiered Prefix Caching. In legacy Python-based frameworks, moving KV-cache tensors between accelerator High Bandwidth Memory (HBM) and host dynamic RAM (DRAM) over PCIe causes serialization stalls and memory copies, creating multi-second delays.

Direct DMA Across Four Storage Tiers

TPU Sync bypasses Python runtimes and intermediate memory allocations by executing asynchronous Direct Memory Access (DMA) directly on XLA/PJRT raw buffers across four storage tiers:

  • Tier 1 (HBM): TPU on-chip memory.
  • Tier 2 (Host Memory): Pinned host CPU DRAM.
  • Tier 3 (Storage): Local NVMe SSDs and Google Cloud Managed Lustre file systems.
  • Multi-Host DCN: Zero-copy TCP and gRPC streaming across cluster networks.

In July 2026 production benchmarks on TPU v7x (Ironwood) clusters (2x2x1 topology, tpu-inference v0.24.0):

  • Host DRAM to HBM (PCIe Gen5): Reached 223 GB/s of the 256 GB/s ceiling (87.1% bus efficiency).
  • Multi-Host Streaming: Achieved ~192 Gbps on single 200G NICs and ~350 Gbps on dual 200G NICs, operating near line-rate.
Benchmark Scenario & Target Model Legacy Framework Latency TPU Sync Result Measured Improvement
P/D KV Transfer Delay (Qwen3 235B) Several seconds (Python serialization) ~230 milliseconds Over 90% reduction
P/D TTFT (Qwen3 235B) Multi-second prompt stalls Compressed time to initial token 60% TTFT cut; 30% to 70% throughput gain
P/D TTFT (Qwen3 Coder 480B) Inter-node buffer stalls Raw C++ DMA streaming Up to 90% TTFT cut
64K Context Prefix Cache (Host DRAM) Prompt recomputation or HBM overflow 223 GB/s reload from host DRAM 90% TTFT cut; >50% throughput gain; 30% P99 ITL cut
RL Weight Sync (OpenRL) Rollout worker idle waiting High-speed DCN parameter distribution Near-zero rollout idle stall time

For enterprise systems processing 64,000-token enterprise manuals or legal archives, offloading prompt caches to host DRAM preserves HBM while slashing TTFT by 90%.


3. CNCF Agent Substrate 1.0: Sub-500ms Sleep/Resume and 10x Density

While TPU Sync optimizes active inference memory, Agent Substrate 1.0 eliminates compute billing during agent idle periods.

Slide 5: Agent Substrate 1.0: Sub-500ms Recovery and 10x Density

Released on November 5, 2026, as generally available on Google Kubernetes Engine (GKE) and contributed to CNCF Sandbox (PR #523), Agent Substrate 1.0 decouples agent orchestration into three primitives:

  1. Actors: Stateful entities tracking conversational memory, scratchpads, and execution history.
  2. Gateways: Policy and routing checkpoints handling tool callbacks and user responses.
  3. Workers: Pre-warmed pools of stateless Kubernetes execution containers.

Eliminating Long-Tail Idle Costs

Unlike stateless microservices, autonomous agents spend over 90% of their operational lifespan waiting:

  • Active Planning: Generating code or parsing inputs (2 to 5 seconds).
  • Tool Waiting: Awaiting external APIs or database query returns (3 to 30 seconds).
  • Human Approval: Waiting for corporate sign-off or multi-factor confirmations (minutes to hours).

Dedicated containers bill continuous compute throughout these delays. Conversely, full teardown creates 10-to-30-second cold starts.

Agent Substrate 1.0 snapshots an Actor state to storage the millisecond it enters an idle wait, immediately recycling the Worker container. When an event arrives, the Gateway injects the state into a warm Worker in under 500 milliseconds.

In production stress tests, Agent Substrate sustained 500 activations/sec across 200,000 agents, delivering a 10x increase in container packing density. Aligned with gVisor container sandboxing (CNCF PR #521), OpenRL (PR #518), and KubeRay (PR #525), it establishes an open foundation for multi-cloud agent clusters.


4. Agent Compute Engine (ACE): Zero-Secret Sandboxes and 4-Tier Identity

Enterprise deployments require strict security controls to prevent prompt injection attacks from compromising internal systems.

Slide 6: Agent Compute Engine: Zero Sandbox Secrets and 4-Tier Identity

Google designed Agent Compute Engine (ACE) (detailed in technical releases on October 22, 2026) to power commercial Gemini infrastructure without storing credentials in execution environments.

Hardware-Rooted Isolation and No Direct NICs

ACE provisions isolated lightweight MicroVMs with robust defenses:

  • Hardware Root of Trust: Titan chips and Titanium IPUs enforce cryptographic boot checks and a read-only host OS via dm-verity.
  • DAX and Copy-on-Write: Shared read-only base images minimize RAM usage and enable near-instantaneous startup.
  • No Direct Network Interface: MicroVMs have no direct network cards to access the intranet or public web; all traffic routes through off-node Agent Gateways.
  • Zero Persistent Secrets: Neither sandbox memory nor virtual disks contain permanent credentials.

The 4-Tier Identity Hierarchy and In-Flight Token Swapping

ACE separates identity into four discrete scopes:

Identity Tier Specification Scope, Storage, and Security Role
Tier 1: User Identity Human End-User Credential Validates the human employee or client delegating the task.
Tier 2: Delegated Agent Identity On-Behalf-Of (OBO) Context Scopes specific permissions granted to the agent for the task.
Tier 3: Agent Workload Identity (ID-3) Ephemeral SPIFFE Certificate Sole credential inside the sandbox. Proves process origin with zero inherent data access.
Tier 4: Agent Workforce Identity Autonomous Agent ID Corporate IAM identity for background autonomous agents, maintaining audit accountability.

When an agent calls an external API, it transmits only its ephemeral ID-3 token. The off-node Agent Gateway validates the token, confirms the Tier 2 delegation scope, and executes an in-flight token swap—inserting the genuine enterprise API bearer token into the outbound stream. If an attacker breaches the sandbox via prompt injection, no durable secrets exist to exfiltrate.


5. Silicon Scalability: NVIDIA Vera Rubin NVL72 and Architecture Cohesion

Complementing custom TPUs, Google Cloud announced general availability for NVIDIA Vera Rubin NVL72 GPU instances (A5X) at GTC Berlin on November 6, 2026.

  • Rack Architecture: 72 Rubin GPUs and 36 Vera CPUs coupled via NVLink in liquid-cooled racks.
  • Virgo Network Scaling: Connects up to 80,000 Rubin GPUs in a single data center and scales up to 960,000 Rubin GPUs across multi-site federations.
  • Unified Software Stack: GKE, Agent Substrate 1.0, KubeRay, and Managed Lustre operate uniformly across both TPU and GPU clusters.

This establishes an integrated multi-tier stack:

  1. Application Layer: Autonomous enterprise agents and multi-turn workflows.
  2. Orchestration Layer: Agent Substrate 1.0 (sub-500ms sleep/resume) and ACE (zero-secret MicroVMs and in-flight token swapping).
  3. State Acceleration Layer: TPU Sync (223 GB/s DMA over PCIe Gen5).
  4. Physical Silicon Fabric: Google TPU v7x and NVIDIA Vera Rubin NVL72 A5X instances.

(Note: Commercial pricing for standalone ACE MicroVM instances and hourly rates for A5X Vera Rubin instances were not disclosed in launch documentation and remain unverified at this publication date.)

Slide 7: Eliminating Idle Stalls Determines the Strategic Winner


6. Enterprise Implementation Playbook: 3 Strategic Actions

For IT leaders, systems integrators, and software architects preparing for exponential token expansion, three practical actions should be prioritized:

Slide 8: Three Strategic Countermeasures for Enterprise AI Systems

Action 1: Implement Tiered Prefix Caching to Host DRAM

Stop pinning static prompt KV-caches in scarce accelerator HBM.

  • Execution: Deploy tiered prefix caching via open-source TPU Sync or equivalent connectors in vLLM, SGLang, or llm-d.
  • Impact: Offloading prompt caches to host DRAM slashes TTFT by up to 90% and lifts throughput by over 50%.

Action 2: Decouple Agent State from Compute Workers

Eliminate compute costs incurred while agents wait for external callbacks.

  • Execution: Adopt CNCF Agent Substrate 1.0. Store state in logical Actors and run tasks on warm, ephemeral Worker pools.
  • Impact: Sub-500ms state recovery increases container packing density by 10x, curbing unnecessary cloud spend.

Action 3: Purge Static Credentials from Code Sandboxes

Harden agent execution environments against indirect prompt injection exploits.

  • Execution: Remove static database passwords and service keys from container environments. Issue short-lived SPIFFE workload tokens and route outbound traffic through off-node gateways that perform in-flight token swapping.
  • Impact: Sandboxes contain zero durable credentials, structurally eliminating data exfiltration risks.

7. Primary Source Documentation

  1. Goldman Sachs Research (May 20, 2026): AI agents are forecast to boost tech cash flow as usage soars, Jim Schneider: Goldman Sachs Insights Article
  2. Google Cloud Blog (November 9, 2026): Introducing TPU Sync: Fast KV cache and weight transfer on TPU: Google Cloud Infrastructure Blog
  3. Google Cloud Blog (November 5, 2026): Open infrastructure for autonomous agents comes to CNCF: Google Cloud Containers Blog
  4. Google Cloud Blog (October 22, 2026): Introducing Agent Compute Engine and expanded agent governance: Google Cloud AI Infrastructure Blog
  5. Google Cloud Blog (October 22, 2026): Inside Agent Compute Engine: Building the compute foundation for the agentic era: Google Cloud Deep Dive Blog
  6. Google Cloud Blog (November 6, 2026): Google Cloud infrastructure at NVIDIA GTC Berlin 2026: Google Cloud Hardware Announcements

Back to articles