← All Posts
AIArchitectureLLMs

Building AI Agents at Scale: What Actually Works

Sean Lobjoit··3 min read
// View summary on LinkedInopenLinkedIn();

Everyone is building AI agents. Very few of them work in production.

After deploying agent systems across cloud environments for enterprise clients, the pattern is clear: the gap between "impressive demo" and "production system" is almost entirely architectural.

The Demo Trap

Most agent demos fail in production for the same reasons:

  • No observability: you can't debug what you can't see
  • Unbounded execution: agents that can loop forever (and will)
  • State management debt: conversation context stored in ways that don't survive restarts
  • Tool call sprawl: 40+ tools per agent, creating a combinatorial reasoning problem

What Actually Works

1. Keep the tool surface small

An agent with 5 well-defined tools will outperform an agent with 30 tools almost every time. Each tool is a decision point for the LLM. The more decisions, the more opportunity for drift.

Design tools around actions, not APIs. create_support_ticket beats post_to_jira_api.

2. Build for observability from day one

Every agent invocation should emit:

  • The input prompt (with context length)
  • The reasoning trace (if your model supports it)
  • Every tool call with inputs and outputs
  • Total tokens consumed and latency

This isn't optional. You will need this to debug production failures.

3. Put humans in the loop at the right points

The agents that succeed in production aren't the ones that do the most autonomously. They're the ones that know when to stop and ask.

Build explicit escalation paths. An agent that surfaces uncertainty is infinitely more valuable than one that confidently hallucinates forward.

4. Treat context like memory, not a dumping ground

The context window is not a database. Stuffing 100k tokens of raw data into every agent call is expensive, slow, and degrades reasoning quality.

Use retrieval. Summarise aggressively. Pass only what's needed for the next decision.

The Cloud Infrastructure Layer

Agents need reliable infrastructure underneath them:

  • Idempotent tool implementations: agent retries will happen
  • Rate limiting and circuit breakers on every external API call
  • Async execution for long-running tasks: synchronous agent loops time out
  • Persistent state stores: DynamoDB, Firestore, or Postgres, not in-memory

The Bottom Line

AI agents are a legitimate architectural pattern. They're also easy to get spectacularly wrong.

Start small. One agent, one task, deep observability. Expand from there once you understand the failure modes.

The engineers building reliable agents aren't the ones with the most tools. They're the ones who understand where the boundaries are.

← Back to all posts