LlmStream

is a pull-based stream of tokens and tool calls from an LLM response. Wraps a ResponseStream and parses provider-specific streaming formats. Use poll for text-only streaming or poll-event for streaming with tool call support.

stop-reason, input-tokens and output-tokens hold what LLMResponse would, filled in as the provider reports them; until then, or if the stream is cut off first, they are "" and 0.

(match (LLM.chat-stream &config &req)
  (Result.Success stream)
    (do
      (while-do true
        (match (LlmStream.poll-event &stream)
          (Maybe.Nothing) (break)
          (Maybe.Just evt)
            (match-ref &evt
              (StreamEvent.Text tok) (IO.print tok)
              (StreamEvent.ToolCallEvent tc)
                (IO.println &(String.concat
                  &[@"tool: " @(ToolCall.name tc)])))))
      (LlmStream.close stream))
  (Result.Error e) (IO.errorln &e))

close

defn

(Fn [LlmStream] ())

                        (close s)
                    

closes the underlying connection.

copy

instantiate

(Fn [(Ref LlmStream a)] LlmStream)

copies a LlmStream.

delete

instantiate

(Fn [LlmStream] ())

deletes a LlmStream. Should usually not be called manually.

init

instantiate

(Fn [ResponseStream, String, String, String, String, String, (Array StreamEvent), Bool, String, Int, Int] LlmStream)

creates a LlmStream.

input-tokens

instantiate

(Fn [(Ref LlmStream a)] (Ref Int a))

gets the input-tokens property of a LlmStream.

linebuf

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the linebuf property of a LlmStream.

output-tokens

instantiate

(Fn [(Ref LlmStream a)] (Ref Int a))

gets the output-tokens property of a LlmStream.

pending

instantiate

(Fn [(Ref LlmStream a)] (Ref (Array StreamEvent) a))

gets the pending property of a LlmStream.

poll

defn

(Fn [(Ref LlmStream a)] (Maybe String))

                        (poll s)
                    

returns the next token from the LLM stream, or Nothing when complete. For text-only streaming. Use poll-event for streaming with tool call support.

A provider chunk carrying several text parts (Gemini) yields them joined in wire order, tagged with soh-prefix only when every one of them is a thought. Use poll-event to keep reasoning and answer text apart.

poll-event

defn

(Fn [(Ref LlmStream a)] (Maybe StreamEvent))

                        (poll-event s)
                    

returns the next StreamEvent from the LLM stream, or Nothing when complete. Unlike poll, this handles tool calls from streaming responses — incremental tool call data is accumulated internally and emitted as complete StreamEvent.ToolCallEvent values. Parallel tool calls (multiple tool calls in a single response) are fully supported. Events are returned one at a time, in wire order; a chunk carrying several parts (Gemini) yields one event per part rather than only the first.

Gemini does not provide tool call IDs, so ToolCall values from Gemini streams will have an empty string as their ID.

Once it returns Nothing, stop-reason, input-tokens and output-tokens say why the response ended and what it cost.

(while-do true
  (match (LlmStream.poll-event &stream)
    (Maybe.Nothing) (break)
    (Maybe.Just evt)
      (match-ref &evt
        (StreamEvent.Text tok) (IO.print tok)
        (StreamEvent.ToolCallEvent tc)
          (println* "tool: " (ToolCall.name tc)))))

prn

instantiate

(Fn [(Ref LlmStream a)] String)

converts a LlmStream to a string.

provider

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the provider property of a LlmStream.

rs

instantiate

(Fn [(Ref LlmStream a)] (Ref ResponseStream a))

gets the rs property of a LlmStream.

set-input-tokens

instantiate

(Fn [LlmStream, Int] LlmStream)

sets the input-tokens property of a LlmStream.

set-input-tokens!

instantiate

(Fn [(Ref LlmStream a), Int] ())

sets the input-tokens property of a LlmStream in place.

set-linebuf

instantiate

(Fn [LlmStream, String] LlmStream)

sets the linebuf property of a LlmStream.

set-linebuf!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the linebuf property of a LlmStream in place.

set-output-tokens

instantiate

(Fn [LlmStream, Int] LlmStream)

sets the output-tokens property of a LlmStream.

set-output-tokens!

instantiate

(Fn [(Ref LlmStream a), Int] ())

sets the output-tokens property of a LlmStream in place.

set-pending

instantiate

(Fn [LlmStream, (Array StreamEvent)] LlmStream)

sets the pending property of a LlmStream.

set-pending!

instantiate

(Fn [(Ref LlmStream a), (Array StreamEvent)] ())

sets the pending property of a LlmStream in place.

set-provider

instantiate

(Fn [LlmStream, String] LlmStream)

sets the provider property of a LlmStream.

set-provider!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the provider property of a LlmStream in place.

set-rs

instantiate

(Fn [LlmStream, ResponseStream] LlmStream)

sets the rs property of a LlmStream.

set-rs!

instantiate

(Fn [(Ref LlmStream a), ResponseStream] ())

sets the rs property of a LlmStream in place.

set-stop-reason

instantiate

(Fn [LlmStream, String] LlmStream)

sets the stop-reason property of a LlmStream.

set-stop-reason!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the stop-reason property of a LlmStream in place.

set-stream-done

instantiate

(Fn [LlmStream, Bool] LlmStream)

sets the stream-done property of a LlmStream.

set-stream-done!

instantiate

(Fn [(Ref LlmStream a), Bool] ())

sets the stream-done property of a LlmStream in place.

set-tc-args

instantiate

(Fn [LlmStream, String] LlmStream)

sets the tc-args property of a LlmStream.

set-tc-args!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the tc-args property of a LlmStream in place.

set-tc-id

instantiate

(Fn [LlmStream, String] LlmStream)

sets the tc-id property of a LlmStream.

set-tc-id!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the tc-id property of a LlmStream in place.

set-tc-name

instantiate

(Fn [LlmStream, String] LlmStream)

sets the tc-name property of a LlmStream.

set-tc-name!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the tc-name property of a LlmStream in place.

stop-reason

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the stop-reason property of a LlmStream.

str

instantiate

(Fn [(Ref LlmStream a)] String)

converts a LlmStream to a string.

stream-done

instantiate

(Fn [(Ref LlmStream a)] (Ref Bool a))

gets the stream-done property of a LlmStream.

tc-args

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the tc-args property of a LlmStream.

tc-id

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the tc-id property of a LlmStream.

tc-name

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the tc-name property of a LlmStream.

update-input-tokens

instantiate

(Fn [LlmStream, (Ref (Fn [Int] Int a) b)] LlmStream)

updates the input-tokens property of a LlmStream using a function f.

update-linebuf

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the linebuf property of a LlmStream using a function f.

update-output-tokens

instantiate

(Fn [LlmStream, (Ref (Fn [Int] Int a) b)] LlmStream)

updates the output-tokens property of a LlmStream using a function f.

update-pending

instantiate

(Fn [LlmStream, (Ref (Fn [(Array StreamEvent)] (Array StreamEvent) a) b)] LlmStream)

updates the pending property of a LlmStream using a function f.

update-provider

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the provider property of a LlmStream using a function f.

update-rs

instantiate

(Fn [LlmStream, (Ref (Fn [ResponseStream] ResponseStream a) b)] LlmStream)

updates the rs property of a LlmStream using a function f.

update-stop-reason

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the stop-reason property of a LlmStream using a function f.

update-stream-done

instantiate

(Fn [LlmStream, (Ref (Fn [Bool] Bool a) b)] LlmStream)

updates the stream-done property of a LlmStream using a function f.

update-tc-args

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the tc-args property of a LlmStream using a function f.

update-tc-id

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the tc-id property of a LlmStream using a function f.

update-tc-name

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the tc-name property of a LlmStream using a function f.