Eight seconds staring at a spinner kill conversion. The same eight seconds arriving in chunks feel instant. Streaming AI responses cuts perceived latency from 8s to under 1s because the first token arrives fast and the rest flows in. Server-Sent Events deliver that with less complexity than WebSockets in most cases.

SSE: HTTP with events

SSE uses text/event-stream as content type, events separated by blank lines, and auto-reconnect in the browser's EventSource. Each event fits in a few bytes; proxies and load balancers handle the flow without strain. Idle connections die inside intermediate proxies. A keepalive comment (: ping every 15s) keeps the path open. LLM output flows server to client; SSE covers everything through an ordinary HTTP request.

When WebSockets earn their keep

  • Genuine bidirectional flow: voice agents, collaborative sessions, users interrupting mid-sentence
  • Message latency below 100ms changes the perceived experience
  • Multiplexing several logical streams over one connection

Streaming on the backend

Your server forwards provider deltas as they arrive and flushes buffers without waiting for accumulation. Nginx holds chunks by default and wrecks the experience: proxy_buffering off; on the location block or an X-Accel-Buffering: no response header fixes it. Gzip compression on partial responses also holds bytes back; switch it off on that endpoint.

Cancellation

The user hit stop? The client aborts the fetch, the server detects the closed connection (request.on('close') in Express, generator cancellation in FastAPI), and cancels the upstream provider call. Providers bill tokens generated up to the cut, so the cut has to happen on both sides. Log cancellations with the session ID. Without that chain, every abandoned session pays for tokens nobody read.

Errors mid-stream

The provider drops the connection halfway through a completion. Ship what arrived plus one final structured error event; clients render the partial with a clear failure marker and a regenerate button. When the provider supports continuation, resume from the last received chunk. Silence until timeout is the bad alternative.

Framework notes

  • Next.js route handlers stream a ReadableStream without extra libraries
  • Express needs Content-Type: text/event-stream, Cache-Control: no-cache, and Connection: keep-alive headers, plus res.flush() after each chunk
  • FastAPI ships StreamingResponse out of the box, fed by async generators