Topic · 4 stories
Running AI workloads: model serving and local LLMs, streaming, latency, semantic caching, AI gateways and observability for non-deterministic software.
Per-call LLM logs can’t explain why an agent took forty steps or cost ten times the usual. What an agent trace should capture, how to keep one trace across agents and pauses, what not to record, and how to sample.
A long agent run that crashes either starts over, repeating model calls and side effects, or is lost. How durable execution records and replays steps, where step boundaries go, and why idempotency is still your job.
Ollama has no authentication for local requests, and vLLM’s --api-key covers only some paths. What an exposed inference server gives away, and the setup that closes it: loopback binding, an allowlisting proxy and closed ports.
Tokens stream on localhost, then arrive late and in clumps behind a production proxy. How nginx, HAProxy, Caddy and Cloudflare can hold them, how to survive idle timeouts, and a probe that measures per-event latency and fails CI when events are held.
