AI Agent Operations Platform
An AI sales/support agent built like infrastructure — services over gRPC and a message queue — to explore what agent operations look like when they're engineered, not scripted.
Business Context
The situation
An R&D build exploring a real question: when a company runs AI agents across sales and support, what does the operations layer underneath actually need? Most agent projects are a single process holding an LLM and some tools. This one treats the agent as one component inside a distributed system — separate services for the gateway, auth, task execution, and notifications, communicating over gRPC and an async message bus.
The hidden cost
- A single-process agent has no isolation — auth, long-running tasks, and notifications all share one failure domain
- Long agent runs block the request path when there's no async layer
- Without typed service contracts, an agent platform's internals drift and break silently
The Problem
What actually had to be solved
Stand up an agent platform where long-running agent tasks are decoupled from the request path, services have typed contracts, and the whole thing runs as reproducible infrastructure — so the operational concerns of running agents (isolation, async execution, delivery) are first-class.
Why the existing approach failed
A monolithic agent script couples the API, the agent runtime, and delivery into one process. It works in a demo and strains the moment agent runs get long or traffic gets bursty. The point of this R&D was to see those concerns separated cleanly.
Constraints
- Agent runs can be long — they can't hold an HTTP request open
- Services need contracts that don't silently drift
- The whole system must be reproducible locally and in deployment
Before → After
How the workflow changed
Before
- Request hits one process
- Agent runs inline
blocks the request
- Tool calls in-process
- Notify from same process
shared failure domain
After
- Gateway receives request
- Auth service validates
- Task queued (RabbitMQ)
decoupled from request
- Agent runs in task service
- Tools + retrieval (MCP · Pinecone)
- Notification service delivers
- Realtime update (WebSocket)
Architecture
How the system is built
Split the platform into four services — gateway, auth, task, and notification — talking over gRPC with shared proto contracts, and hand long-running agent work to a task service via a RabbitMQ queue. Each service carries the agent runtime it needs (LangGraph, Pinecone retrieval, MCP tools). The whole stack — services plus MongoDB and RabbitMQ — is orchestrated with Docker Compose so it comes up identically anywhere.
Architecture — click any node
Client
API
Orchestration
AI
Integration
Data
Observability
Client
Channels
WebSocket
- Purpose
- Where requests and live updates flow.
- Responsibility
- Client I/O and realtime push.
AI vs. Engineering
Where AI helps — and where it doesn't
AI Layer
The LangGraph agent runtime plans and executes with MCP tools and Pinecone retrieval. AI owns the reasoning and tool-selection — the part where a fixed script would be brittle.
Engineering Layer
The distributed system around it: a gRPC service mesh with shared proto contracts, a RabbitMQ queue that moves long agent runs off the request path, an isolated auth boundary, WebSocket realtime, and a Docker-Compose stack that's reproducible by construction.
Engineering Decisions
The hard calls, with their tradeoffs
Decision 01
gRPC + shared protos between services
Choice — Type every inter-service call with shared proto contracts instead of ad-hoc REST.
Why — In an agent platform the internals change constantly — typed contracts catch drift at the boundary instead of in production.
Tradeoff — Proto tooling and codegen to maintain, in exchange for interfaces that can't silently break.
Decision 02
RabbitMQ for agent execution
Choice — Queue agent tasks rather than run them inside the request.
Why — Agent runs are long and bursty; a durable queue decouples them from the request path and gives retries and dead-lettering for free.
Tradeoff — A broker to operate, in exchange for execution that doesn't block requests or lose work on restart.
Decision 03
Where we'd START a real client — honestly
Choice — This R&D runs full microservices; a first client engagement would usually start as a modular monolith.
Why — Microservices earn their cost at scale and with independent teams. Reaching for them too early is exactly the over-engineering we avoid — the value here was learning the seams, not prescribing them for everyone.
Tradeoff — Fewer moving parts early, split into services only when scale or team boundaries actually demand it.
Cost & Reliability
Built to run, not just to work
Cost strategy
Queuing means compute is spent only when tasks run, not on idle request threads. The honest cost lesson is the third decision above: microservices add operational cost, so we recommend them only when scale justifies them.
Reliability
Durable RabbitMQ queues survive consumer restarts and give retries/dead-lettering; the agent graph runs with bounded steps to prevent runaway loops; service isolation keeps an auth or notification failure from taking down agent execution.
Security
A dedicated auth service (bcrypt, JWT, Google OAuth) forms the identity boundary, isolated from the services that run agent work.
By the Numbers
What we can point to
4 over gRPC
VerifiedServices
Gateway, auth, task, notification.
RabbitMQ queue
VerifiedAsync execution
Agent tasks decoupled from the request path.
LangGraph · MCP · Pinecone
VerifiedAgent stack
Reasoning, tools, and retrieval per service.
Docker Compose
VerifiedReproducibility
Services + MongoDB + RabbitMQ come up identically.
Business Impact
System → workflow → outcome
System
A distributed agent platform: gRPC services + a durable task queue.
Workflow
Request → auth → queued task → agent run (tools + retrieval) → delivery → realtime update.
Operational change
Long agent runs stop blocking requests, and each concern fails in isolation instead of together.
Business impact
Shows how agent operations are engineered for real load — and, just as importantly, when that engineering is worth it and when it isn't.
Reflection
Lessons & what we'd do next
Lessons
- Typed gRPC contracts caught interface drift that REST would have let slip.
- The most valuable takeaway was knowing when NOT to split into services.
Risks
- Microservices carry real operational overhead — justified only at genuine scale.
- A message bus adds a component that must be monitored and sized.
Improve next
- Add distributed tracing across the gRPC calls and queue hops.
- Publish agent-run evaluation metrics (success, tool errors, latency) per task.
Technology
The verified stack
Services
Messaging
AI
Data
Auth
Ops
Could this be relevant to your business?
If you're moving AI agents from a prototype into something that has to handle real load — or you want an honest read on whether you need microservices at all — this is the kind of architecture work we do.
Related Work
More systems we've built
Autonomous Browser Agent + Harness
An autonomous browser agent with a real harness: durable, resumable state and a human approval gate before it's ever allowed to act.
Voice AI Companion
A production-shaped voice pipeline — speech in, reasoned reply out — with memory, safety, and observability as first-class services, not afterthoughts.