Event-Driven AI Architecture Controls Latency, Retries and Cloud Costs
AI systems expose a weakness in request-response architecture: model work often takes longer than the connection carrying the request can stay open. A queue-first design gives that work room to finish, while helping teams cap model usage, recover from failures and avoid paying for idle capacity. The practical shift is to put serverless components around AI workloads, with compute matched to each part of the pipeline.
Let the request end before the model does
A conventional web handler assumes it can return a result quickly. That assumption breaks when a request starts a multi-step agent, summarizes a long document or generates embeddings in bulk. A model call can run for tens of seconds or several minutes; the client or an API gateway may time out first. The client then retries, potentially launching the same billable work again.
Amazon API Gateway illustrates the tension. Its integration timeout was 29 seconds until AWS raised the limit in June 2024 for Regional and private REST APIs, citing large language model workloads. AWS also said the change could require a lower account-level throttle quota. Extending the wait can therefore reduce capacity available to other API traffic. Lambda has a separate maximum invocation duration of 15 minutes, which still rules out some longer jobs.
For work that can finish asynchronously, the API can validate the input, save a job record and publish an event to a queue or topic such as Amazon SQS, Google Pub/Sub, Azure Service Bus or Kafka. It can return a job ID with an HTTP 202 response, leaving workers to process the event after the client has disconnected. The application can deliver the result through a webhook or WebSocket, or let the client check job status later.
That design changes the meaning of a lost connection. A browser closing no longer cancels a task that has already entered the queue. It also gives teams a place to manage load before jobs reach a model endpoint.
Match compute to uneven demand
AI workloads rarely arrive at a steady rate. Document ingestion may run overnight, copilots may peak during business hours, and report generation may happen once a week. Keeping accelerators ready for every possible burst can mean paying for substantial idle capacity.
Cast AI’s 2026 report examined roughly 23,000 Kubernetes clusters across AWS, Azure and Google Cloud. It reported average GPU utilization of 5%, compared with 8% for CPUs. Cast AI sells optimization software, and the sample covered clusters that were not optimized, so those figures are best read as a warning about potential waste rather than a universal benchmark. The report put the on-demand monthly cost of an idle H100 at about $8,850. Its highest-utilization example was a fleet of 136 H200s serving inference at 49% utilization.
Workers that scale with queue depth can better follow demand. On Kubernetes, KEDA can scale deployments from queue metrics; managed container platforms may support scaling to zero when no work is waiting. Queue depth is often a more useful signal than CPU usage for GPU-bound tasks: a worker can appear quiet to a CPU monitor while its accelerator is fully occupied.
Scaling down has limits. Starting a worker and loading a multi-gigabyte model can take tens of seconds, which may be too slow for latency-sensitive requests. Systems need to balance startup delay against idle cost, and sustained inference traffic may be cheaper on reserved capacity that stays busy.
Put a boundary around retries and quotas
Model providers commonly limit usage by requests and tokens per minute. A batch job that ignores those limits can slow or disrupt customer-facing traffic when both workloads share an account quota. Retries can compound the problem by submitting work that already succeeded or by repeatedly sending a request that cannot currently be accepted.
Consider a nightly run over 100,000 documents, with 20,000 input tokens per document. That is 2 billion input tokens. At an assumed price of $3 per million tokens, one pass costs $6,000; processing each document three times would raise the total to $18,000. These figures depend on the stated token price and input volume, but they show how quickly a retry bug can turn into a large bill.
A queue makes concurrency an explicit control. Teams can limit the number of workers consuming events so batch processing stays within a model quota. Failed messages can move to a dead-letter queue after three unsuccessful receives, while a visibility timeout gives workers room to retry with backoff.
Idempotency keys address a different failure mode: duplicate delivery. Standard SQS queues provide at-least-once delivery, so a consumer can receive the same message more than once. A key based on a document ID and prompt version lets the system recognize work it has already completed and avoid paying for the same model call twice.
Persist multi-step work
Agent pipelines often combine retrieval, model calls, tool use and human review. If all those steps run in one function, a failure near the end can discard the progress and expense accumulated earlier. Workflow engines such as AWS Step Functions, Azure Durable Functions and Temporal persist execution state so a retry can resume from the failed step.
Long-running workflows matter when a person must review an output or a process waits on an external system. AWS Step Functions Standard workflows can run for up to a year, making them suitable for work that may pause for days. A durable record of each step can also help teams answer operational questions: what information entered the workflow, what result came back, and who approved it.
There is a cost tradeoff. Standard Step Functions billing is based on state transitions, so a workflow that makes a separate transition for each item in a 10,000-document batch can add expense. Batching operations where the task allows keeps orchestration overhead in check.
Use serverless around the model
Serverless is a strong fit for ingestion, routing, orchestration, guardrail checks and post-processing. It is not a universal answer for inference. AWS Lambda does not offer GPU compute, so teams serving their own models need GPU instances or a managed inference endpoint.
Retrieval-augmented generation shows where managed services can reduce infrastructure work. Document storage, chunking, embedding and indexing can run through services such as Amazon S3, a managed knowledge base and serverless vector search. Those components may not require teams to operate GPU machines themselves, although embedding workloads still need suitable compute somewhere in the service stack.
The right boundary depends on traffic and latency targets. A workload with intermittent demand may benefit from workers that scale down between jobs. A high-volume inference service that keeps accelerators busy may be more economical on dedicated or reserved capacity. Cold starts also matter: loading a large model can take long enough to undermine a fast response requirement.
Make failures visible across the pipeline
Event-driven systems can fail without producing a clear application error. A message might be lost between stages, leaving a customer to report a missing result before operators notice. A correlation ID carried through each queue, worker and workflow gives teams a way to trace one job across the full path.
Monitoring should include queue age and dead-letter depth alongside error rates. A growing backlog can signal a capacity shortfall or a downstream quota problem even while individual workers appear healthy. Those signals make it easier to spot stalled work before it becomes a customer complaint.
Before releasing an AI feature, identify each synchronous model call and compare its p99 latency with the relevant gateway timeout. Put queues and concurrency caps between batch jobs and shared model quotas, then attach an idempotency key to every message that can trigger a billable request. With those controls in place, latency, throttling and duplicate work become manageable parts of system design—and the architecture can scale with the actual shape of AI demand.
