How to Resume an AI Chat After the User Goes Offline

Streaming makes AI chat feel conversational. The first words appear quickly, and the response grows in front of you instead of hiding behind a loading spinner.

But streaming also couples two things that should not necessarily share a lifetime: the model’s work and the user’s Internet connection. This coupling is mostly invisible on a stable desktop connection but on mobile, it becomes a reliability problem. A user can lock their phone, switch apps, enter an elevator, or move between Wi-Fi and their cell network, causing their browser to drop its connection midway through a generation. The model may still be producing a perfectly good response, but the client can’t hear it anymore.

We can build a resumable streaming system to handle this case. The generation continues after the client disconnects, its output is persisted as it arrives, and the client can reconnect without starting the model request over.

The central idea is simple:

Treat generation as durable work and streaming as a temporary view into that work.

Once those two lifecycles are separated, intermittent connectivity becomes a recoverable transport problem instead of a failed AI request.

Why ordinary streaming fails

A straightforward streaming endpoint usually works like this:

1. The client sends a request.
2. The server starts a model generation.
3. Tokens arrive from the model.
4. The server forwards them to the client.
5. The connection closes when the response finishes.

This design is an easy way to start building a streaming endpoint. The server reads a chunk from the model and immediately writes that chunk to the HTTP response.

This approach fails when the connection disappears, though. Depending on the implementation, the server may cancel the model request, abandon the response, or continue generating output that no client can recover. Retrying from the browser often starts a second model request, which is both expensive and semantically wrong. Given the probabilistic nature of AI, even at a low temperature the second response may diverge from the first.

Also, a disconnect is not evidence that the user wants to cancel. On a phone, it may not even mean the user intentionally left the page.

To make it possible to resume chats when the device reconnects, we need three operations to be independent:

• Starting a generation.
• Observing a generation’s output.
• Canceling a generation.

That separation changes which part of the system owns the model request. The generation can run on the server independently of the connection observing it because closing an HTTP stream detaches the observer but does not terminate the producer. Cancellation uses its own explicit signal.

At a high level, the system can look like this:

The model request, stored output, and observation stream have related lifecycles, but they no longer need to share the same lifecycle.

Give each generation a durable identity

Every model request receives a stable generation ID. That ID exists independently of any particular HTTP connection.

Conceptually, the server stores a record like this:

generation
  id
  status: running | complete | failed | cancelled
  chunks: ordered output received so far
  final metadata
  expiration time
Copy to clipboard

Each item in the stored stream can have a small, explicit event shape:

1type GenerationEvent =
2  | { sequence: number; type: "content"; content: string }
3  | { sequence: number; type: "complete"; messageId: string }
4  | { sequence: number; type: "error"; code: string }
5  | { sequence: number; type: "cancelled" };
Copy to clipboard

Content is not the only state worth replaying. A reconnecting client also needs to learn whether a generation completed, failed, or was cancelled while it was offline.

The initial request starts the work and returns a stream associated with that ID. If the stream disconnects, the generation keeps running. A later request can attach to the same ID and ask for everything after the last output the client successfully processed.

This changes the meaning of a streaming request. It no longer means "perform this work while I hold this socket open." It means "let me observe this durable operation from a particular point."

Persist output while it is in progress

Resumption only works if the server retains output that arrived while the client was away.

Redis works well as short-lived storage for in-progress generations because this state is temporary, frequently updated, and read during reconnects. Each generation can expire automatically after a reasonable recovery window.

A simplified write path looks like this:

model emits chunk
      ↓
assign sequence number
      ↓
persist chunk in Redis
      ↓
publish or forward chunk to connected clients
Copy to clipboard

The order matters. If the server sends a chunk before persisting it, the connection can fail in the gap between those two actions. The client may have seen output that the server cannot replay, leaving both sides with different histories.

Persisting first does not eliminate every failure window. A process can persist chunk 43 and crash before publishing its notification. That is acceptable because Redis remains the source of truth: the chunk will be found during replay. This is another reason the protocol favors recoverable, at-least-once delivery over trying to make live publication exactly once.

The stored stream also needs a terminal event. "No new chunks right now" is different from "this generation is finished." Completion, failure, and cancellation should all be explicit states that reconnecting clients can observe.

Resume from a cursor

The client tracks the most recent event it has successfully applied. Then, when it reconnects, it sends that position back to the server.

The cursor can be a monotonically increasing sequence number. For example, a reconnect request might carry:

1GET /generations/gen_123/stream
2Last-Event-ID: 42
Copy to clipboard

The exact representation matters less than the contract. The client is saying:

This is the state I already have. Continue from here.

Sequence numbers avoid the ambiguity of trying to locate the client by its partial text. Text is not a safe cursor: a model can repeat the same word, sentence, or larger passage, making it unclear which occurrence the client has already received. A sequence number gives every event one unambiguous position regardless of its content.

On reconnect, the server:

1. Loads the generation state.
2. Validates the client’s cursor.
3. Replays any stored chunks after that cursor.
4. Subscribes the connection to new chunks if the generation is still running.
5. Sends the terminal event immediately if the generation has already finished.

The client experiences this as one continuous answer, even though several network connections may have carried it.

Here is the full recovery path with concrete positions:

1. The client starts a generation and receives events 1 through 30.
2. The phone goes offline, but the server-side generation continues.
3. The model produces events 31 through 44, which are persisted even though the client is absent.
4. The client reconnects with the generation ID and cursor 30.
5. The server subscribes the new connection to live notifications, buffers them, and captures 44 as the replay high-water mark.
6. It replays events 31 through 44 from Redis in sequence order.
7. It discards buffered events at or below 44 as duplicates, then processes newer buffered events in order.
8. Live delivery continues without starting another model request.

Prefer at-least-once delivery without gaps

The most subtle race happens when a client reconnects while the model is still generating.

Imagine that Redis contains chunks 1 through 42. The reconnecting client needs chunks after 30, so the server begins replaying 31 through 42. Meanwhile, the model writes chunks 43 and 44.

If the server first replays stored output and only then subscribes to live output, it can miss chunks produced in between. If it subscribes first, it may receive chunk 43 live while also finding it in a subsequent storage read.

The protocol therefore has to tolerate overlap while preserving sequence order. Simply applying events as they arrive is not enough. If live event 43 arrives before replayed events 31 through 42, advancing the cursor to 43 would cause the client to treat the missing replay events as stale.

We can handle that boundary with a replay high-water mark:

1. Subscribe to notifications for new output and buffer them temporarily.
2. Read the highest sequence number currently persisted and record it as the replay high-water mark.
3. Replay persisted events after the client cursor through that high-water mark, applying them in sequence order.
4. Drain the buffered notifications in sequence order, discarding events at or below the high-water mark as duplicates.
5. Continue with live delivery, applying an event only when it is the next expected sequence.

Subscribing before capturing the high-water mark closes both sides of the race. An event persisted before the subscription is included in the replay range. An event persisted afterward produces a notification that is buffered until replay finishes. Sequence numbers then make any overlap easy to deduplicate.

Networks make duplicate delivery normal. A server may write a chunk successfully just before the connection dies, and neither side can infer from the closed connection whether the client received it. The client therefore applies only events newer than its current cursor:

1if (event.sequence <= lastAppliedSequence) {
2  return;
3}
4
5if (event.sequence !== lastAppliedSequence + 1) {
6  buffer(event);
7  return;
8}
9
10switch (event.type) {
11  case "content":
12    append(event.content);
13    break;
14  case "complete":
15    finalize(event.messageId);
16    break;
17  case "error":
18    showError(event.code);
19    break;
20  case "cancelled":
21    showCancelled();
22    break;
23}
24
25lastAppliedSequence = event.sequence;
26drainContiguousBufferedEvents();
Copy to clipboard

This is much easier to reason about than trying to guarantee that every event crosses an unreliable connection exactly once. The useful guarantee is:

An event may be delivered more than once, but the client applies it at most once and in sequence order.

Stale cursors need an explicit policy

A reconnect request can refer to state the server no longer has. The generation may have expired from Redis, the client may present a cursor beyond the server’s latest event, or stored output may have been compacted into a final message.

These cases should not silently restart the model. Instead, the server should tell the client what happened:

• The completed message is available; replace the local partial message with it.
• The cursor is inconsistent; resynchronize from the server’s canonical copy.
• The recovery window expired; offer an explicit retry.
• The generation failed or was cancelled; show that terminal state.

Restarting automatically risks additional token costs and a second response that does not match the text the user already saw.

Partial messages are real product state

Streaming interfaces often treat partial text as pixels that happen to be on the screen. Once reconnection is possible, partial output becomes part of the protocol.

The client needs to know:

• Which message is being generated?
• How much of it has been applied?
• Is the message still active?
• Has the final server state replaced the partial version?

This becomes especially important if the app itself reloads. A new client instance may need to reconstruct the in-progress message from local state, server state, or both.

The server should remain authoritative. The partial text on the client is a local view, not a recovery cursor. On reconnect, the client presents the generation ID and last applied sequence number; the server can then replay missing events or replace the local partial message with its canonical final copy.

Reconnection is also a UI state

A technically resumable stream can still feel broken if the interface handles reconnection badly. When connectivity disappears, immediately replacing partial text with an error is usually the wrong signal. The generation may be progressing normally.

Instead, the UI can preserve the response and move through a small set of understandable states:

Generating → Reconnecting → Generating → Complete
Copy to clipboard

These principles help:

• Keep the partial text visible.
• Distinguish reconnecting from generation failure.
• Retry automatically with bounded backoff.
• Do not show duplicated text during replay.
• Let the user cancel while reconnecting.
• Offer a manual retry only when automatic recovery is no longer possible.

The best version is deliberately uneventful. The user briefly sees a reconnecting indicator, the stream catches up, and the response continues.

Cancellation must be explicit

Separating generation from the connection raises an important operational question: when should the work stop? If a disconnected client can no longer cancel, the system might keep spending model tokens on output nobody wants. But treating every disconnect as cancellation defeats the purpose of resumability.

The answer is to make cancellation a separate, explicit operation and combine it with bounded retention:

• The Stop button sends a cancellation request for the generation ID.
• A lost connection does not cancel the generation.
• In-progress generations have maximum runtime limits.
• Stored output expires after the recovery window.
• Orphaned work is monitored and cleaned up.

This gives mobile users time to return without allowing abandoned work to live forever.

The invariants that make the system easier to reason about

The implementation has several moving pieces, but a few invariants keep the behavior coherent:

1. A client disconnect never means cancellation.
2. Persisted output is authoritative.
3. Every output event has a stable order.
4. Delivery may be duplicated but must not contain gaps.
5. Reconnecting never starts a second model request implicitly.
6. Every generation eventually reaches a terminal state.
7. Temporary recovery state has a bounded lifetime.

These rules are more valuable than any particular choice of transport or datastore. Redis, Server-Sent Events, WebSockets, and ordinary streaming HTTP can all participate in a resumable design. The key is defining lifetimes and ownership clearly.

Failure modes to test

The happy path is straightforward. Most of the engineering work lives at the boundaries:

• A disconnect during the first chunk
• A disconnect after completion but before the terminal event arrives
• A reconnect while replay and live output overlap
• Multiple tabs attaching to the same generation
• Cancellation racing with completion
• Redis eviction during an active stream
• Application deploys while generations are running
• A client whose local partial text disagrees with server state

These cases deserve fault-injection tests, not just unit tests. Deliberately terminate connections at different points in the stream and verify that the final message is complete, ordered, and contains no duplicated text.

Metrics are equally important. Track reconnect attempts, successful resumptions, cursor mismatches, replay sizes, expired generations, orphaned work, and duplicate events. A recovery system can appear healthy in ordinary request metrics while quietly failing the users who need it most.

Streaming is a view, not the job

The deeper lesson is not specific to AI chat. Long-running work should not depend on the continued existence of the connection that started it. A network stream is only one way to observe progress. It is not a reliable place to store progress, and it should not define the lifetime of the underlying operation.

That distinction becomes increasingly important as AI products perform longer and more agentic tasks. A generation lasting a few seconds can sometimes get away with fragile request-response semantics. Work lasting minutes or involving tools, retries, and multiple stages cannot.

The product goal is simple: a writer should be able to lock their phone midway through a generation, return later, and watch the response continue instead of starting over. Making that experience feel uneventful requires treating the generation as durable work, Redis as a short-lived event log, and every network connection as replaceable.