Consuloji: Deep Dives into Overseas AI日本語

What OpenAI Custom Inference Silicon Changes, What Remains Unaltered, and How to Audit Efficiency Ratios

Published: November 2, 2026

Listen or watch on YouTube

A technical analysis of the OpenAI and Broadcom Jalapeño inference processor, 1.5x to 1.9x throughput-per-watt benchmark conditions, TSMC N3P silicon limits, and enterprise AI accelerator evaluation.

Japanese Edition: Read the original Japanese article on Substack

Video Episode (Japanese Audio): Watch on YouTube

Executive Summary: Targeted Latency and Efficiency vs. Shared Silicon Supply Realities

On June 24, 2026, OpenAI and Broadcom Inc. announced Jalapeño, a custom application-specific integrated circuit (ASIC) engineered exclusively for large language model inference. According to official joint announcements, the processor aims to unify the high raw throughput of mainstream accelerators with the ultra-low interactive latency of dedicated inference engines. Initial system deployments within production clusters were scheduled to commence by the end of 2026. However, final production performance metrics remain under active characterization, and the inaugural June announcement contained zero absolute performance numbers.

On August 25, 2026, OpenAI published preliminary benchmark results evaluated against NVIDIA Blackwell-generation hardware (specifically the GB200 and GB300). Technology publications reported that Jalapeño delivered:

  1. Between 1.5x and 1.9x higher maximum token throughput per watt.
  2. Between 1.7x and 3.6x lower end-to-end request latency.

Crucially, these efficiency multipliers are governed by strict measurement conditions:

  • Mathematical Denominator: Ratios were calculated using manufacturer rated thermal design power (TDP)—700W for Jalapeño, 1,200W for GB200, and 1,400W for GB300—rather than real-world dynamic power draw measured at the utility meter.
  • Workload Boundary: Evaluations comprised isolated single-turn prompts with 8,000 input tokens and 1,000 output tokens.
  • Algorithmic Constraints: Tests operated without speculative decoding.
  • Data Provenance: All telemetry originated from OpenAI internal benchmarks utilizing the SemiAnalysis InferenceX test suite; while analysts witnessed runs in OpenAI laboratories, the research firm did not independently execute the benchmark suite.

Equally significant is what custom silicon does not alter. According to technical reporting from SemiAnalysis, Jalapeño compute logic is fabricated on TSMC N3P process node—the identical fabrication node utilized for NVIDIA next-generation Rubin GPUs. Furthermore, Jalapeño incorporates fourth-generation High Bandwidth Memory (HBM4), identical to Rubin. Consequently, custom ASIC development does not bypass global semiconductor bottlenecks; OpenAI must queue for the exact same foundry allocation and advanced packaging lines as commercial merchant silicon vendors.

For corporate infrastructure architects, chief technology officers, and procurement leads evaluating enterprise AI infrastructure, this analysis provides three critical analytical frameworks:

  • Auditing the denominator of claimed efficiency ratios (rated TDP vs. empirical dynamic power vs. facility-level utility draw).
  • Deconstructing benchmark configurations (prompt lengths, token ratios, generation algorithms, and hardware generational parity).
  • Verifying benchmark provenance (vendor self-reporting vs. independent third-party replication).

Slide 1: What Custom Inference Silicon Changes


Methodological Boundaries: Documented Records, Author Calculations, and Interpretation

To maintain analytical rigor, this evaluation segments information into three categories:

  • Documented Records: Explicit statements, engineering specifications, and benchmark disclosures published in official corporate press releases, engineering whitepapers, SEC filings, and accredited industry research.
  • Author Calculations: Mathematical ratios, percentages, and conversions calculated directly from published figures.
  • Author Interpretation: Synthesized evaluations and frameworks regarding enterprise procurement, explicitly designated as analytical deductions.

Primary source documentation comprises the OpenAI-Broadcom joint announcement (June 24, 2026), OpenAI preliminary results disclosure (August 25, 2026), SemiAnalysis architectural teardowns, accredited technology reporting (WinBuzzer and Igor Lab), and official datasheets from NVIDIA, Google Cloud, and Amazon Web Services, retrieved and cross-verified on November 2, 2026.

Slide 2: Three Core Analytical Questions


Technical Foundations: The Mechanics of Transformer Inference

Evaluating Jalapeño requires examining the dual-phase nature of autoregressive model execution:

  • Prefill Phase: The model processes the input context prompt in parallel. This phase is heavily compute-bound, saturating matrix multiplication tensor cores.
  • Decode Phase: The model generates output tokens sequentially. Each token generation step requires streaming parameter weights and historical key-value (KV) attention cache states from off-chip memory into on-chip registers. This phase is heavily memory-bandwidth-bound.
  • Speculative Decoding: An optimization technique where a smaller draft model predicts multiple candidate tokens, which the larger target model verifies in a single forward pass, shifting memory-bound decoding toward compute saturation.
  • Thermal Design Power (TDP): The maximum theoretical power dissipation specified by hardware vendors, distinct from real-world dynamic consumption.
  • High Bandwidth Memory (HBM): Vertically stacked DRAM dies connected to logic through silicon interposers, progressing from HBM3E to HBM4.

Slide 3: Officially Confirmed Architecture and Timeline


Section 1: Officially Confirmed Parameters: Architecture, Roles, and Delivery Horizons

Official corporate announcements published on June 24, 2026, confirm several key operational parameters:

  • Designation and Positioning: Jalapeño is OpenAI first custom “Intelligence Processor,” architected specifically for generative model inference. The companies framed the chip as the inaugural silicon release in a multi-generational computing partnership.
  • Architectural Philosophy: The design was created from a blank slate tailored specifically to OpenAI proprietary model weights, software kernels, and serving systems, rejecting the general-purpose compromises of traditional GPUs.
  • Corporate Division of Labor:
    • OpenAI: Architecture definition, logic design, and software integration.
    • Broadcom: Physical implementation, co-packaged optics, PCIe switching, and Tomahawk network interconnects.
    • Celestica: System-level hardware engineering, PCB fabrication, and server rack assembly.
  • Silicon Status: OpenAI stated that laboratory prototypes were operational at production clock frequencies and thermal targets, successfully executing production models including GPT-5.3-Codex-Spark.
  • Development Velocity: The companies claimed a 9-month timeline from architecture freeze to tapeout. SemiAnalysis noted that initial team hiring and engineering commenced in mid-2024, representing approximately 16 months from initial staffing to tapeout.
  • Deployment Trajectory: Initial cluster deployment was slated for late 2026, with Broadcom leadership noting multi-gigawatt infrastructure collaboration with Microsoft. SemiAnalysis projected that production volume would ramp progressively throughout 2027, with the bulk of deployments landing in late 2027.
  • Omitted Public Data: The official announcement omitted fabrication process names, memory configurations, and numerical throughput data.

Slide 4: Memory Bandwidth as the Fundamental Decode Bottleneck


Section 2: Architectural Focus: Why Inference Demands Memory Bandwidth

In standard GPU clusters, memory bandwidth constraints during autoregressive decoding leave compute engines idling while waiting for weights and KV cache transfers.

Jalapeño addresses this through targeted dataflow optimization. OpenAI joint statement notes that the architecture minimizes data movement, balancing compute, memory, and networking to operate near theoretical hardware limits.

SemiAnalysis architectural reporting describes a partitioned layout:

  • The monolithic die is partitioned into localized compute and memory zones, allowing execution units to access adjacent HBM dies with minimal interconnect latency.
  • The system rejects disaggregated prefill-decode clustering. In disaggregated architectures, separate server pools handle input processing and token generation. SemiAnalysis explains that because input-to-output token ratios fluctuate continuously across real-world enterprise traffic, static pooling causes chronic underutilization and queue stalls. Jalapeño unifies prefill and decode across identical execution engines.

Industry peers are pursuing complementary optimizations. NVIDIA Vera Rubin NVL72 product disclosures combine Rubin GPUs with Groq LPX racks to target trillion-parameter models across million-token contexts.

Slide 5: Deconstructing the 1.5x to 1.9x Efficiency Ratios


Section 3: Deconstructing Reported Benchmarks: Conditions Behind the Multipliers

On August 25, 2026, OpenAI released preliminary benchmarks evaluated on the SemiAnalysis InferenceX test harness across three open-weights models: GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. All tests evaluated single-turn interactions with 8,000 input tokens and 1,000 output tokens.

Reported performance differentials against NVIDIA Blackwell hardware were published as follows:

Evaluated Model Comparison Baseline Reported Throughput Per Watt Gain Reported Latency Reduction Absolute Throughput (Tokens/s/kW) Absolute End-to-End Latency
GPT-OSS 120B NVIDIA GB200 NVL72 ~1.9x higher ~1.7x lower latency 85,448 (Jalapeño) vs. 44,960 (GB200) 1.03s (Jalapeño) vs. 1.80s (GB200)
DeepSeek R1 670B NVIDIA GB300 NVL72 ~1.7x higher ~3.6x lower latency Not disclosed in summary 1.65s (Jalapeño) vs. 5.99s (GB300)
Kimi K2.5 1T NVIDIA GB300 NVL72 ~1.5x higher ~3.4x lower latency Not disclosed in summary Not disclosed in summary

Four technical conditions qualify these numbers:

Condition 1: Rated Thermal Design Power Denominator

The reported throughput-per-watt ratios were calculated by dividing raw token output by the rated maximum thermal design power (TDP) of each accelerator package:

  • OpenAI Jalapeño: 700W rated TDP.
  • NVIDIA GB200: 1,200W rated TDP.
  • NVIDIA GB300: 1,400W rated TDP.

Author calculation shows the arithmetic impact of rated TDP denominators: 1,200W divided by 700W equals 1.71x, while 1,400W divided by 700W equals 2.0x. Baseline power ratings alone account for the majority of the reported multiplier. While OpenAI reported that Jalapeño sustained dynamic power remained below 550W during testing, corresponding dynamic measurements for GB200 and GB300 were not disclosed.

Condition 2: Absence of Speculative Decoding

All benchmarks evaluated sequential token-by-token decoding without speculative decoding. Because speculative decoding increases compute intensity, it disproportionately enhances GPU throughput.

Condition 3: Fixed Synthetic Prompt Ratios

The test suite evaluated exclusively an 8,000/1,000 token single-turn profile. Results from the AgentX benchmark, which simulates multi-turn conversational agents with sprawling KV caches, were not available.

Condition 4: Generational Alignment

SemiAnalysis observed that comparing Jalapeño to Blackwell is architecturally mismatched. Because Jalapeño utilizes HBM4 memory, its true peer is NVIDIA Rubin. SemiAnalysis estimates that on facility-level power (all-in utility MW), Jalapeño exceeds Rubin throughput per watt, but total cost per token is roughly comparable between the two architectures. Furthermore, these tests evaluated early A0 stepping silicon; a refined B0 revision currently in fabrication is projected to improve energy efficiency by an additional 25%.

Slide 6: Cross-Accelerator Memory Bandwidth Comparison


Section 4: Industry Landscape: Memory Bandwidth and Fabrication Parity

Comparing publicly verifiable hardware specifications across leading AI accelerators illustrates architectural positioning:

Accelerator Platform Memory Capacity Memory Technology Memory Bandwidth Evaluation Unit Source Citation
NVIDIA Vera Rubin 288GB HBM4 19.2 TB/s Per discrete GPU NVIDIA official product page
NVIDIA GB200 NVL72 13.4TB HBM3E 576 TB/s rack total (~8.0 TB/s per GPU) 72-GPU rack system NVIDIA official product page
NVIDIA GB300 NVL72 20.0TB HBM3E Up to 576 TB/s rack total (~8.0 TB/s per GPU) 72-GPU rack system NVIDIA official product page
Google Cloud TPU7x 192GiB HBM ~7.37 TB/s Per discrete chip Google Cloud documentation
Amazon Trainium3 144GB HBM3e 4.9 TB/s Per discrete chip AWS product specification
OpenAI Jalapeño Unspecified HBM4 15.4 TB/s Per multi-chip package SemiAnalysis technical teardown

Jalapeño memory bandwidth of 15.4 TB/s per package places it near NVIDIA Rubin (19.2 TB/s). However, both chips depend on TSMC N3P lithography and HBM4 packaging lines. While custom ASICs optimize internal dataflow, they remain bound to the global semiconductor supply chain.

In rack deployment, Jalapeño clusters utilize 128 accelerators per server rack, co-developed with Celestica. Separately, Foxconn and OpenAI announced an agreement on November 21, 2025, to co-design and manufacture next-generation AI data center racks in United States facilities. However, that agreement covers generalized infrastructure and does not mention Jalapeño.

Furthermore, technology reporting indicates that OpenAI has explicitly reaffirmed its commitment to purchasing and deploying commercial merchant accelerators, including NVIDIA GPUs, across training and inference workloads. Jalapeño represents architectural diversification rather than total vendor replacement.

Slide 7: Multiplier Arithmetic and Supply Chain Constraints


Section 5: Analytical Interpretation: What Changes and What Remains Constant

Synthesizing public disclosures highlights critical strategic takeaways:

  • Efficiency Is Multiplicative: A reported 1.9x throughput-per-watt advantage is the product of silicon architecture multiplied by rated TDP baselines. Altering the denominator to actual facility-level power reshapes comparative efficiency.
  • Latency Gains Are Real but Conditional: Single-turn prompt benchmarks prove that co-designing memory buses with proprietary model kernels dramatically compresses response times. Whether these gains persist under multi-turn agent workloads remains unverified.
  • Merchant GPUs Remain Essential: Custom ASICs cannot instantly supplant merchant GPUs due to packaging constraints and software ecosystem maturity.
  • Unverified Variables: Pricing economics, long-context conversational performance (AgentX), and wafer allocation volumes remain undisclosed.

Slide 8: Enterprise Accelerator Audit Framework


Executive Implementation Framework: Auditing Accelerator Performance Claims

Enterprise technical evaluators reviewing vendor performance metrics should enforce a five-step audit protocol:

  1. Audit the Power Denominator: Determine whether efficiency claims divide by rated TDP, dynamic chip draw, or total facility utility power (including cooling and power distribution losses).
  2. Examine Algorithmic Conditions: Verify whether benchmarks utilized speculative decoding, structured sparsity, or low-bit quantization formats.
  3. Inspect Workload Topology: Check input context and generation lengths. Single-turn tests with short outputs conceal KV-cache eviction bottlenecks.
  4. Demand Generational Parity: Ensure new silicon is compared against contemporary architectures rather than prior-generation baselines.
  5. Verify Data Provenance: Ascertain whether benchmarks represent vendor self-reporting or independently reproduced runs.

Frequently Asked Questions

Q: Is Jalapeño fundamentally faster than NVIDIA GPUs? A: OpenAI benchmarks demonstrated lower latency and higher throughput per rated watt against Blackwell GB200 and GB300 under strict single-turn, 8,000/1,000-token conditions. However, against Rubin hardware utilizing speculative decoding, comparative cost per token is estimated to be roughly equivalent.

Q: Will OpenAI cease procuring NVIDIA accelerators? A: No. OpenAI has explicitly stated that it will continue large-scale procurement of NVIDIA GPUs for both model training and operational inference.

Q: Is Foxconn manufacturing Jalapeño server racks? A: Official documentation does not link Foxconn to Jalapeño. Foxconn partnered with OpenAI for general AI rack designs, whereas Celestica is the engineering partner for Jalapeño server systems.

Q: Why do development timelines vary between 9 and 16 months? A: OpenAI measured the 9-month window from formal architecture tapeout authorization, whereas SemiAnalysis tracked the 16-month window from initial team recruitment.


Primary Reference Documentation

Back to articles