Still Running Agentic Pilots? Here’s What 5 Companies Did to Ship AI Agents to Production

Five companies ran agentic AI pilots last year. All five had working demos by Q3. Only two shipped to production. The other three are still “refining” their proof of concept, waiting for the right moment to scale. That moment will not arrive on its own.

This piece is for the operations leader or IT director who has a working pilot sitting in a sandbox, budget approval pending, and a growing suspicion that the demo-to-production gap is wider than anyone told you.

Why Pilots Stall

A recent Salesforce piece on production AI agents highlights a pattern we see repeatedly in mid-market engagements: teams that shipped made a single architectural decision early, while teams that stalled kept optimizing the wrong layer.

The pilots that never graduate share three characteristics:

• They measure success by model accuracy instead of business throughput
• They treat exceptions as edge cases to solve later rather than core design constraints
• They assume the humans in the loop will adapt to the agent instead of designing the agent around existing workflows

The teams that shipped flipped each of those assumptions. They accepted 85% accuracy as sufficient when paired with robust exception routing. They built the exception path first. They shadowed the people who would supervise the agent before writing a single prompt.

The Decision That Separates Shippers from Refiners

In every successful production deployment we have observed, there is a moment—usually around week six of the pilot—where the team faces a choice: keep improving the happy path, or start building the scaffolding that catches failures.

Teams that choose the happy path stay in pilot mode indefinitely. The agent gets incrementally better at the 70% of cases it already handles well. Leadership sees impressive demos. But the 30% of cases that require human judgment never get a proper handoff mechanism, so the agent cannot be trusted unsupervised, so it never ships.

Teams that choose the scaffolding accept that the agent will be imperfect. They spend weeks seven through twelve building:

• Confidence thresholds that trigger human review
• Audit trails that let supervisors understand why the agent acted
• Fallback queues that route failures to the right person, not a generic inbox
• Monitoring dashboards that surface degradation before customers notice

This scaffolding typically costs 3–5x what the original pilot cost. It is also the only path to production.

What the Successful 20% Built First

The companies that shipped shared a counterintuitive priority: they invested more in the “agent fails” path than the “agent succeeds” path.

Exception Routing Before Optimization

One B2B services company deployed an agent to handle routine contract amendments. Their pilot achieved 91% accuracy on test data. Instead of pushing to 95%, they spent six weeks building a tiered review system: amendments under $50K with high confidence scores auto-processed; amendments between $50K and $200K queued for single-reviewer approval; anything above $200K or below the confidence threshold went to a two-person review.

The agent still runs at 91% accuracy. It processes 340 amendments per week that previously required analyst time. The 9% it flags for review take less time to process than the old system because the agent pre-populates the analysis.

Human-in-the-Loop as a Feature, Not a Crutch

A common mistake: treating human oversight as a temporary phase you will eliminate once the model improves. The companies that shipped treat human oversight as a permanent architectural component.

The production mindset changes what you build. You design the review interface to be fast. You track reviewer throughput as a system metric. You feed reviewer corrections back into the model. The human is not a workaround—the human is part of the system.

Integration Depth Over Model Sophistication

Pilots often run on exported data or API snapshots. Production requires real-time integration with systems of record—CRM, ERP, ticketing, billing. In most mid-market environments, this integration work takes 8–12 weeks and surfaces data quality issues that were invisible in the sandbox.

One manufacturing company discovered their pilot’s 88% accuracy dropped to 61% in production because their ERP contained 15 years of inconsistent product codes. The agent worked fine; the data did not. They spent three months on data remediation before relaunching.

The Production Checklist

Before you present a go-live date to leadership, verify you have answers—not plans, answers—to these questions:

• What happens when the agent’s confidence score falls below threshold? Who gets notified, how fast, and what do they see?
• How do you detect model drift? What is your threshold for retraining versus rollback?
• What is the blast radius of a bad decision? If the agent makes an error at 2 AM, how many downstream systems are affected before a human notices?
• Who owns the agent after go-live? Not the vendor. Not “IT.” A named person with time allocated.
• What does the agent do when a downstream system is unavailable? Queue, fail, or proceed with stale data?

If any of those answers is “we’ll figure it out after launch,” you are not ready for production.

The Real Cost of Waiting

There is a counterargument: if the technology is improving rapidly, maybe waiting is rational. Let the models get better, let best practices emerge, ship next year instead of this quarter.

This logic has a flaw. The companies that ship now are building institutional knowledge—how to supervise agents, how to structure exception handling, how to integrate AI into existing workflows. That knowledge compounds. The company that ships an 85%-accurate agent in Q2 and iterates for eight months will outperform the company that ships a 92%-accurate agent in Q4 with no operational experience.

The gap is not model quality. The gap is organizational muscle memory. You cannot buy that from a vendor. You have to build it by shipping.

The pattern across companies that moved from pilot to production is consistent: they stopped optimizing the demo and started building the system around it. They accepted imperfection as a design constraint. They invested in failure handling before success handling. They named an owner and gave them authority.

Your pilot probably works. The question is whether you have built everything that lets it fail safely. That is the work that separates the teams still refining from the teams already compounding their lead.