Consuloji: Deep Dives into Overseas AI日本語

Why 89% AI Agent Pilots Stall at 14% Production: Cascading Errors and Three Guardrails

Published: November 18, 2026

Listen or watch on YouTube

Why visual AI agents pass short demos yet hit a 14% enterprise wall—unpacking negative transfer from human logs, hybrid +23.05pt gains, and 62% security barriers.

Japanese Edition: Read the original Japanese article on Substack

Video Episode (Japanese Audio): Watch on YouTube

Autonomous vision-language-action (VLA) models and computer-using agents (CUAs) that operate software strictly through screenshots or camera feeds—paired with human-style mouse and keyboard inputs rather than internal APIs or source code—are spreading rapidly across gaming, automated software testing, 3D CAD, humanoid robotics, and medical laboratory automation. On the surface, eliminating custom integration pipelines promises universal software automation: any legacy application or desktop workflow that a human can see and click should, in theory, become automatable overnight.

Yet enterprise deployment data from late 2025 and early 2026 reveals a stark paradox. While 79% to 89% of surveyed organizations are piloting or adopting AI agents and generative AI quality engineering, only 14% to 15% have scaled these systems into full enterprise-wide production (Zylos Research, January 2026; Capgemini World Quality Report 2025, November 13, 2025). Short demonstrations look remarkably fluent, yet production environments expose compounding errors over long task horizons, severe “negative transfer” when training naively on human interaction logs, and an unmanaged security boundary that 62% of practitioners cite as their top concern.

This analysis synthesizes primary data from twelve peer-reviewed preprints, technical reports, and industry surveys published between May 2025 and November 2026 to trace why pure screen-based autonomy stalls in practice—and outlines three architectural guardrails required to cross the 14% production divide.

Slide 1: Visual AI Agents — 79-89% Pilots vs. 14-15% Enterprise Production


1. The Pilot-to-Production Chasm: 89% Testing vs. 14–15% Scale

Industry surveys across both general GUI automation and software quality engineering point to the same structural bottleneck: moving from controlled pilot demos to enterprise-wide production deployment.

Primary Source (Publication Date) Domain Scope Adoption / Pilot Share Enterprise-Wide Production Top Production Blockers Cited
Capgemini World Quality Report 2025 (November 13, 2025) Quality Engineering & GenAI QA 89% (52% piloting, 37% partial production) 15% Lack of AI/ML expertise; tactical, fragmented adoption
Zylos Research (January 9, 2026) GUI / Computer-Using AI Agents 79% (organizations adopting or piloting) 14% Security concerns (62%), reliability (32%), full-autonomy trust (27%)

In Capgemini’s annual quality engineering report, 89% of organizations had moved forward with generative AI initiatives—split between 52% in active pilots and 37% running limited production deployments—yet only 15% achieved enterprise-wide scaling. Capgemini attributed this drop-off to two operational gaps: shortages of specialized AI and machine learning engineering talent, and tactical, siloed rollouts that leave automation fragmented across teams rather than integrated into core delivery pipelines.

Zylos Research’s January 2026 analysis of GUI computer-using agents uncovered an almost identical ceiling: 79% adoption or experimentation narrowing to just 14% scaled deployment. When examining practitioner sentiment, 62% identified security as a primary barrier, 32% flagged reliability and error accumulation over multi-step workflows, and only 27% expressed full trust in completely autonomous agents operating unassisted. In short, what succeeds across a five-step sandbox demo breaks down when confronted with real-world latency, unexpected modal dialogs, and unrestricted operating system permissions.

Slide 2: Three Walls Separating 89% Pilots from 15% Production


2. How Pure Vision-Action Models Scaled to 65.12 ms Latency

To understand why expectations around visual agents surged despite these enterprise blockers, it is necessary to examine how rapidly pure screen-to-action models improved in 2026.

  • miHoYo “Gamer” (arXiv:2611.18397v1 / Project Page, November 13, 2026): Developed by miHoYo’s Honkai AI R&D Team, Gamer operates strictly through visual screen input and device-level keyboard and mouse actions without accessing game internals or source code. Built on Qwen3-VL-8B and Qwen3.5-35B-A3B backbones, the team conducted continual pre-training (CPT) on tens of thousands of hours of gameplay footage, followed by supervised fine-tuning (SFT) on approximately 1.2 million time-aligned reasoning (“Think”) traces and 223,000 memory samples, and finally time-aware group relative policy optimization (TA-GRPO). Across seven puzzle categories totaling 40 tasks in Honkai: Star Rail, Gamer achieved a 49.7% average success rate—nearly tripling the 17.1% baseline of GPT-6.1-Sol—while climbing from 3/16 to 15/16 on exploration benchmarks. Crucially, its Streaming-vLLM inference engine observes frames at 5 Hz (every 200 ms) while emitting control actions at 30 Hz (every 33 ms), cutting end-to-end latency on NVIDIA H200 GPUs from 1,306.00 ms down to 65.12 ms (92.16 Hz control throughput).
  • Qwen-CUA (arXiv:2608.02352, August 2026): Extending pure visual operation to general desktop tasks, Qwen-CUA uses a 397B-A17B Mixture-of-Experts architecture that relies exclusively on the most recent 20 screenshots and standard mouse-keyboard actions—with zero DOM trees or API hooks. Training required an industrial-scale reinforcement learning infrastructure spanning nearly 100,000 vCPUs in parallel virtualized environments across roughly 40,000 verifiable tasks.

Pure visual control carries an undeniable advantage: like a new restaurant worker operating an unfamiliar point-of-sale touchscreen purely by reading button labels without knowing the backend network protocol, a visual agent requires zero custom API adapters. Yet that exact reliance on surface pixels introduces extreme fragility whenever layouts shift, pop-ups interrupt a sequence, or tasks stretch across dozens of interdependent steps.

Slide 3: Pure Vision VLA Models and 65.12ms Inference Acceleration


3. Why Imitating Human Logs Backfires: Negative Transfer and Long-Horizon Decay

A widespread assumption in enterprise AI projects is that recording enough employees performing desktop tasks will naturally train a reliable automation agent. Controlled experiments from 2025 and 2026 demonstrate the opposite.

  1. Negative Transfer in ProCUA-SFT (arXiv:2606.17321, June 2026): When researchers applied supervised fine-tuning to UI-TARS 7B using AgentNet—one of the largest public datasets of human GUI demonstrations (22.5K trajectories)—OSWorld benchmark performance did not improve; it collapsed from 26.3% down to 8%–10%. Human recordings are filled with cursor hesitation, backtracking, idiosyncratic window switching, and inconsistent recovery habits that introduce severe noise into policy learning. By contrast, when the researchers automatically generated and verified 93,000 synthetic trajectories (3.1 million action steps) across 2,484 application combinations and trained for a single epoch, OSWorld success jumped to 45.0%—an 18.7 percentage point gain over the base model.
  2. Data Quality Precedes Reinforcement Learning in OS-Helix (arXiv:2610.22234v1, October 2026): Table 4a of the OS-Helix study isolates the ordering effect on OSWorld-Verified. Applying reinforcement learning directly to the 46.50% base model yielded only 48.20% (a marginal +1.70pt lift). However, applying high-quality SFT first lifted accuracy to 62.75%, and layering RL on top of that clean foundation pushed performance to 66.44%.
  3. Horizon Explosion in VideoCAD (arXiv:2505.24838v2, May 2025): Real engineering tools involve far longer sequences than consumer web browsing. The MIT-led VideoCAD dataset captures 41,005 interaction videos in the 3D CAD platform Onshape, including action sequences reaching 186 steps—more than 4 times the average temporal horizon of the next-longest UI dataset, WebLINX. Even a 99% per-step accuracy degrades steeply when compounded across 186 spatial reasoning actions, where a single misplaced click invalidates downstream geometry.

Verifying these primary tables firsthand also exposes a common trap in secondary AI summaries. Readers often encounter 63.87% and 66.44% cited side-by-side for OS-Helix and assume they measure the same benchmark. Cross-checking the original paper confirms that 63.87% belongs to WindowsAgentArena, whereas 66.44% is the OSWorld-Verified result—and miHoYo’s pre-training scale is explicitly reported as “tens of thousands of hours” rather than an exact rounded figure. Separating benchmark conditions at the source level is essential before extrapolating research metrics to internal business workflows.

Slide 4: Negative Transfer from Human Logs and Long-Horizon Compounding


4. Dual-System Control and the Remaining Lab-to-Production Gap

Interestingly, physical robotics and laboratory automation have converged on the exact same architectural split to manage long-horizon compounding errors and slow multimodal reasoning.

  • Figure AI “Helix” (Official Announcement): Because general vision-language models think too slowly for real-time balance and dexterous manipulation, Helix decouples cognition from execution. Its System 2 module interprets natural-language instructions and visual scenes at 7 to 9 Hz, while its System 1 visuomotor policy translates those latent intentions into continuous upper-body control at 200 Hz. Just as a cyclist uses conscious thought to read a street map at intersections while unconscious reflexes continuously adjust steering and balance, this dual-rate architecture enables two humanoid robots to collaboratively sort unfamiliar household items.
  • Robo-MLT (arXiv:2611.13039v1, November 2026): Applying dual-system control to medical laboratory workflows, Robo-MLT pairs a high-level progress tracker with a low-level action generator so the robot can replan after human technicians interrupt or swap test tubes mid-protocol. Crucially, however, the authors explicitly bound their achievement: evaluation was limited to two laboratory protocols, a single workspace, and one type of external intervention, and the system has not been deployed in a production laboratory.

Even with sophisticated dual-system architectures, bridging controlled benchmarks and live operational environments remains an unresolved engineering boundary.

Slide 5: Dual-System Control and the Distance to Production Environments


5. Why Production Teams Retreat to Hybrid GUI + API/MCP Architectures

Inside enterprise software and IT workflows, engineering teams are moving away from pure pixel-only automation toward hybrid architectures—and the quantitative evidence explains why.

On the WindowsAgentArena benchmark in the OS-Helix study (arXiv:2610.22234v1), a pure GUI-only base agent achieved 40.82%. Simply unifying GUI actions with structured code execution (Bash) inside a hybrid base model jumped success to 56.41%, and full hybrid RL training (OS-Helix-27B-RL) reached 63.87%—a +23.05 percentage point improvement over visual control alone. Executing deterministic file lookups, text parsing, or calculations directly via shell commands eliminates dozens of brittle mouse clicks and visual OCR errors.

The commercial software testing market reflects the same shift. Automated QA provider Momentic, which raised a $15 million Series A (Momentic Blog, November 2025) and validated its system across 1,396 labeled bugs, pairs browser-level visual test execution with Model Context Protocol (MCP) servers that connect directly to developers’ coding agents. Rather than forcing an agent to debug a codebase through a screen capture, Momentic routes UI verification through visual channels and code remediation through structured protocol endpoints. Across the broader ecosystem, MCP adoption has scaled past 97 million monthly SDK downloads (Zylos Research MCP Overview, January 7, 2026).

Think of an office relocation: an operations team does not send movers walking into every room to flip circuit breakers by hand when a building engineer already has direct access to the electrical panel schematic. Structured interfaces (APIs, CLIs, and MCP) handle deterministic system state, reserving visual GUI interaction strictly for legacy screens or true visual verification.

Slide 6: Hybrid Architecture — +23.05pt Gain Over Pure GUI Operation


6. The 62% Security Boundary and Three Guardrails Before Production

Technical accuracy alone does not clear an enterprise security review. As Zylos Research highlights, granting an autonomous agent raw desktop screen vision and OS-level mouse-keyboard control breaks legacy Identity and Access Management (IAM) models (Zylos Research, January 2026). Traditional IAM verifies who logged into a session, not why a stochastic model is about to click “Delete” or “Send External Wire” after misreading a prompt-injected banner on a webpage. That governance gap explains why 62% of practitioners flag security as a primary obstacle and only 27% trust full autonomy.

Combined with Capgemini’s finding that 52% piloting collapses to 15% enterprise-wide scale due to skills shortages and fragmented point deployments, organizations can only cross the 14% production wall by establishing three concrete guardrails before rollout:

  1. Enforce a Strict Hybrid Boundary: Offload deterministic data retrieval, file manipulation, and system state changes to APIs, CLIs, and MCP endpoints, restricting pixel-level GUI automation strictly to legacy interfaces without APIs or end-to-end visual assertions (capturing the +23.05pt stability advantage while slashing maintenance overhead).
  2. Audit Trajectory Quality and Long-Horizon Recovery: Reject vendor claims built on short demos or unfiltered human screen recordings that risk 8%–10% negative transfer. Benchmark candidates under internal multi-step conditions—measuring state tracking, modal pop-up handling, and rollback recovery across realistic workflow lengths.
  3. Deploy Sandboxed Least-Privilege Execution with Human Approval Gates: Isolate visual agents inside dedicated virtual machines or containers with scoped credentials, full action audit trails, and mandatory human-in-the-loop confirmation before any irreversible action (external messaging, financial transactions, or record deletion).

Moving from 89% experimentation to durable production value rarely starts with end-to-end unassisted autonomy. Teams that establish reversible, semi-autonomous boundaries first turn promising demos into auditable enterprise infrastructure.

Slide 7: 62% Security Boundary and Enterprise Scaling Blockers

Slide 8: Three Operational Guardrails Before Deploying Visual Agents


Primary Sources Verified in This Analysis

Back to articles