llm

LlmStream

is a pull-based stream of tokens and tool calls from an LLM response. Wraps a ResponseStream and parses provider-specific streaming formats. Use poll for text-only streaming or poll-event for streaming with tool call support.

(match (LLM.chat-stream &config &req)
  (Result.Success stream)
    (do
      (while-do true
        (match (LlmStream.poll-event &stream)
          (Maybe.Nothing) (break)
          (Maybe.Just evt)
            (match-ref &evt
              (StreamEvent.Text tok) (IO.print tok)
              (StreamEvent.ToolCallEvent tc)
                (IO.println &(String.concat
                  &[@"tool: " @(ToolCall.name tc)])))))
      (LlmStream.close stream))
  (Result.Error e) (IO.errorln &e))

close

defn

(Fn [LlmStream] ())

                        (close s)

closes the underlying connection.

copy

instantiate

(Fn [(Ref LlmStream a)] LlmStream)

copies a LlmStream.

delete

instantiate

(Fn [LlmStream] ())

deletes a LlmStream. Should usually not be called manually.

init

instantiate

(Fn [ResponseStream, String, String, String, String, String, (Array StreamEvent), Bool] LlmStream)

creates a LlmStream.

linebuf

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the linebuf property of a LlmStream.

pending

instantiate

(Fn [(Ref LlmStream a)] (Ref (Array StreamEvent) a))

gets the pending property of a LlmStream.

poll

defn

(Fn [(Ref LlmStream a)] (Maybe String))

                        (poll s)

returns the next token from the LLM stream, or Nothing when complete. For text-only streaming. Use poll-event for streaming with tool call support.

A provider chunk carrying several text parts (Gemini) yields them joined in wire order, tagged with soh-prefix only when every one of them is a thought. Use poll-event to keep reasoning and answer text apart.

poll-event

defn

(Fn [(Ref LlmStream a)] (Maybe StreamEvent))

                        (poll-event s)

returns the next StreamEvent from the LLM stream, or Nothing when complete. Unlike poll, this handles tool calls from streaming responses — incremental tool call data is accumulated internally and emitted as complete StreamEvent.ToolCallEvent values. Parallel tool calls (multiple tool calls in a single response) are fully supported. Events are returned one at a time, in wire order; a chunk carrying several parts (Gemini) yields one event per part rather than only the first.

Gemini does not provide tool call IDs, so ToolCall values from Gemini streams will have an empty string as their ID.

(while-do true
  (match (LlmStream.poll-event &stream)
    (Maybe.Nothing) (break)
    (Maybe.Just evt)
      (match-ref &evt
        (StreamEvent.Text tok) (IO.print tok)
        (StreamEvent.ToolCallEvent tc)
          (println* "tool: " (ToolCall.name tc)))))

prn

instantiate

(Fn [(Ref LlmStream a)] String)

converts a LlmStream to a string.

provider

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the provider property of a LlmStream.

rs

instantiate

(Fn [(Ref LlmStream a)] (Ref ResponseStream a))

gets the rs property of a LlmStream.

set-linebuf

instantiate

(Fn [LlmStream, String] LlmStream)

sets the linebuf property of a LlmStream.

set-linebuf!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the linebuf property of a LlmStream in place.

set-pending

instantiate

(Fn [LlmStream, (Array StreamEvent)] LlmStream)

sets the pending property of a LlmStream.

set-pending!

instantiate

(Fn [(Ref LlmStream a), (Array StreamEvent)] ())

sets the pending property of a LlmStream in place.

set-provider

instantiate

(Fn [LlmStream, String] LlmStream)

sets the provider property of a LlmStream.

set-provider!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the provider property of a LlmStream in place.

set-rs

instantiate

(Fn [LlmStream, ResponseStream] LlmStream)

sets the rs property of a LlmStream.

set-rs!

instantiate

(Fn [(Ref LlmStream a), ResponseStream] ())

sets the rs property of a LlmStream in place.

set-stream-done

instantiate

(Fn [LlmStream, Bool] LlmStream)

sets the stream-done property of a LlmStream.

set-stream-done!

instantiate

(Fn [(Ref LlmStream a), Bool] ())

sets the stream-done property of a LlmStream in place.

set-tc-args

instantiate

(Fn [LlmStream, String] LlmStream)

sets the tc-args property of a LlmStream.

set-tc-args!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the tc-args property of a LlmStream in place.

set-tc-id

instantiate

(Fn [LlmStream, String] LlmStream)

sets the tc-id property of a LlmStream.

set-tc-id!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the tc-id property of a LlmStream in place.

set-tc-name

instantiate

(Fn [LlmStream, String] LlmStream)

sets the tc-name property of a LlmStream.

set-tc-name!

instantiate

(Fn [(Ref LlmStream a), String] ())

sets the tc-name property of a LlmStream in place.

str

instantiate

(Fn [(Ref LlmStream a)] String)

converts a LlmStream to a string.

stream-done

instantiate

(Fn [(Ref LlmStream a)] (Ref Bool a))

gets the stream-done property of a LlmStream.

tc-args

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the tc-args property of a LlmStream.

tc-id

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the tc-id property of a LlmStream.

tc-name

instantiate

(Fn [(Ref LlmStream a)] (Ref String a))

gets the tc-name property of a LlmStream.

update-linebuf

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the linebuf property of a LlmStream using a function f.

update-pending

instantiate

(Fn [LlmStream, (Ref (Fn [(Array StreamEvent)] (Array StreamEvent) a) b)] LlmStream)

updates the pending property of a LlmStream using a function f.

update-provider

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the provider property of a LlmStream using a function f.

update-rs

instantiate

(Fn [LlmStream, (Ref (Fn [ResponseStream] ResponseStream a) b)] LlmStream)

updates the rs property of a LlmStream using a function f.

update-stream-done

instantiate

(Fn [LlmStream, (Ref (Fn [Bool] Bool a) b)] LlmStream)

updates the stream-done property of a LlmStream using a function f.

update-tc-args

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the tc-args property of a LlmStream using a function f.

update-tc-id

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the tc-id property of a LlmStream using a function f.

update-tc-name

instantiate

(Fn [LlmStream, (Ref (Fn [String] String a) b)] LlmStream)

updates the tc-name property of a LlmStream using a function f.