Streaming

Enable streaming

Set stream: true in the same request. The OpenAI-compatible gateway passes through the upstream event stream.

curl -N "$MODEL_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $MODEL_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{
  "model": "YOUR_MODEL",
  "messages": [
    {
      "role": "user",
      "content": "Explain what this function does: function add(a, b) { return a + b; }"
    }
  ],
  "stream": true
}'

Read the stream

Use the streaming reader in your model client, or parse the server-sent events returned by the API. Text arrives in deltas; tool arguments may also arrive in parts. Assemble the complete response before treating it as finished.

Interrupted streams

Handle connection failures and cancellation separately from a completed answer. Retrying may create another model request and another charge. A partial stream is not evidence that the upstream did no work.