AI API Streaming vs Complete Responses: Handle Partial Output

Artificial Intelligence / API integration / Implementation guide

Choose AI API streaming when users benefit from reading text as it arrives. Choose a complete-response workflow when software needs a finished, validated result before proceeding. In either case, check the provider’s completion metadata: receiving some text, an HTTP 200, or a closed connection does not establish that the task succeeded.

The useful decision is where incomplete output may appear. A drafting assistant can show a provisional paragraph. A system creating an invoice record should wait until it has the complete data and has checked it. Streaming delivery and permission to use a result are separate decisions.

01 / Choose streaming around the user’s next action

Streaming can let a reader begin before the answer is finished. It does not by itself promise a shorter total generation time, a better answer, or a lower bill. Evaluate whether earlier visibility actually helps the task, especially when an answer must be reviewed as a whole.

The transport need not dictate the interface. Claude’s streaming guide documents a final-message helper that accumulates streamed events into a complete message. An application can therefore stream upstream while displaying a waiting state until its own checks finish.

Likewise, Gemini documents streamGenerateContent alongside its ordinary generation method. Its REST streaming examples use alt=sse. Check the selected endpoint and SDK rather than assuming every provider enables streaming with the same parameter.

For a hypothetical customer-support assistant, an agent might read a draft while it forms, but the Send button remains a separate deliberate action. For a hypothetical ticket classifier, early fragments add little value because the routing system needs the complete classification. These are design examples, not measured product results.

02 / Decode complete events rather than arbitrary network chunks

A network read is not a reliable application-message boundary. A read may contain part of an event or several events. Avoid parsing each received byte chunk as a complete JSON object or treating every event payload as ordinary answer text.

The WHATWG server-sent events specification defines a UTF-8 text/event-stream format. Blank lines delimit event dispatch, and an event can contain multiple data lines. Use a suitable SSE parser or a supported SDK to handle framing and incremental decoding.

Provider event shapes still matter after SSE parsing. Claude’s event documentation describes message events, indexed content blocks, deltas, pings, and a final message_stop. Its tool-input deltas can contain partial JSON. Do not concatenate every event’s data into a text transcript.

• Decode incoming bytes without losing characters split across reads.
• Keep framing, provider-event interpretation, and rendering as distinct responsibilities.
• Associate deltas with the correct message and content block.
• Route non-text events to their appropriate handler.
• Handle newly introduced event types deliberately without misclassifying them as text.

Keep a small captured or synthetic event fixture with the integration, stripped of credentials and private content. Split it at awkward byte boundaries in parser tests. A parser that works only when each read happens to contain one event has not verified the actual boundary problem.

03 / Separate provisional text from accepted results

Give the application an explicit lifecycle. “Generating” means the result is still provisional. “Checking” means generation has ended but acceptance checks remain. “Ready” means those checks passed. “Interrupted,” “cancelled,” and “rejected” should be distinguishable from a usable completed answer.

create a new attempt linked to the user request
state = generating
collect recognized response events
optionally render provisional text

if cancelled or interrupted:
    preserve an explicit incomplete state
else:
    state = checking
    inspect completion metadata
    validate the complete result
    if accepted:
        persist once
        state = ready
    else:
        state = rejected

For structured data, previews can be useful for debugging but should not become production records field by field. A partial object may have a plausible name while still lacking the identifier that determines which account it belongs to. Our JSON mode vs structured outputs guide covers the separate contract and value-validation checks.

Treat displayed text as untrusted output. OWASP’s output-handling guidance calls for validation and context-appropriate encoding before model output reaches downstream systems. Use a controlled renderer; do not inject generated fragments directly as executable HTML. Apply the same rule to the completed response.

If a final check rejects the output, the interface should stop presenting the provisional draft as accepted. Decide whether to retain it with a clear warning, replace it with an explanation, or restrict it to an authorized review view. Make that behaviour part of the product specification.

04 / Check why generation ended before accepting it

Transport completion and task completion answer different questions. A stream can end cleanly while the model stopped for a reason your workflow must handle. Conversely, an interrupted connection may leave readable text without the metadata needed to accept it.

For Claude, the stop-reason reference distinguishes natural turn completion, output-limit truncation, tool use, and refusal, among other states. A tool-use stop asks the application to handle a proposed tool call; it is not a finished user-facing answer. Apply the current provider-specific meanings.

Claude’s error reference explicitly warns that an SSE error can occur after HTTP 200. Monitor both the initial request and the ongoing stream. A dashboard counting only non-200 responses will miss that class of failure.

Do not repair a truncated data object merely by closing its braces. The missing content may be essential. Keep separate handling for a refusal, a size limit, a provider error, a parsing failure, and a failed business-rule check.

05 / Make interruption, retry, and cancellation visible

When a stream breaks, stop the generating indicator and identify the attempt as incomplete. A retry should have its own attempt identity under the same user request. Avoid silently appending a fresh generation to the old fragment: it may repeat, contradict, or restart the answer.

Do not assume reconnecting resumes at the last word. Resume support must be documented for the specific API and operation. Otherwise offer a clearly labelled fresh attempt or an application-controlled recovery path. Preserve enough state to explain which result was ultimately accepted.

Browser cancellation also needs an end-to-end plan. MDN documents AbortController for cancelling a fetch and response-body consumption. Your backend must separately handle its upstream request and clean up the attempt. Hiding text or aborting a browser read does not establish how much work the provider performed or what it will bill.

Before retrying, inspect SDK retry behaviour and prevent duplicate downstream operations. If generation proposes sending a message or updating a record, tie execution to a stable operation identifier and the required authorization. A second generation must not silently repeat an already completed action.

Use a bounded recovery policy with a deadline. Our AI API rate-limit guide explains admission and retry budgets. For streaming, add the question that ordinary request handling can miss: did this failed attempt already produce provisional output or trigger later work?

06 / Check the backend, proxy, and browser together

If output appears all at once, examine each boundary: provider to backend, backend to proxy, and proxy to browser. An upstream SDK can stream correctly while application code waits for a full string before sending anything onward.

NGINX’s proxy documentation describes response buffering and how disabling it passes data onward as received. Its X-Accel-Buffering handling is configuration-dependent. Inspect the actual route and hosting setup before changing anything; a header copied into application code is not proof that every intermediary honours it.

Check idle timeouts, maximum request duration, flush behaviour, and any middleware that transforms responses. Compare timestamps at the backend and browser. Keep credentials on the appropriate trusted server boundary and send the browser only the data it is authorized to receive.

Use separate measurements for time to first useful content, time to completion, and time to accepted result. A ping or empty event should not count as useful answer text. Record interruption and cancellation rates alongside these timings, with request IDs and integration versions where available.

07 / Verify failure cases before launch

Use controlled fixtures and a small approved live check. The following is a verification plan, not a claim that a benchmark was performed:

• Split events and multi-byte characters across reads; confirm reconstruction.
• Deliver several events together; confirm every event is handled once.
• Inject an error after initial success; confirm no accepted result is saved.
• End the connection without expected completion metadata; show interruption.
• Simulate truncation, refusal, and tool use; confirm separate handling.
• Cancel from the browser; inspect backend cleanup and attempt state.
• Retry after partial output; confirm text and actions are not duplicated.
• Check the deployed proxy path, mobile layout, keyboard controls, and final rendering.

Choose acceptance criteria tied to your task. For a drafting tool, a clearly marked incomplete draft may be recoverable. For an automated record import, the same interruption should leave the record uncommitted. Release only when those outcomes are explicit and observable.

Frequently asked questions