Core

Lisette Core

LiteLLM

Deterministic outputs

LiteLLM ModelResponse(Stream) objects have id and created_at fields that are generated dynamically. Even when we use cachy to cache the LLM response these dynamic fields create diffs which makes code review more challenging. The patches below ensure that id and created_at fields are fixed and won’t generate diffs.

source

patch_litellm


def patch_litellm(
    seed:int=0
):

Patch litellm.ModelResponseBase such that id and created are fixed.

patch_litellm()

Completion

LiteLLM provides an convenient unified interface for most big LLM providers. Because it’s so useful to be able to switch LLM providers with just one argument. We want to make it even easier to by adding some more convenience functions and classes.

This is very similar to our other wrapper libraries for popular AI providers: claudette (Anthropic), gaspard (Gemini), cosette (OpenAI).

# litellm._turn_on_debug()
ms = ["gemini/gemini-3-pro-preview", "gemini/gemini-3-flash-preview", "claude-sonnet-4-6", "openai/gpt-4.1"]
msg = [{'role':'user','content':'Hey there!', 'cache_control': {'type': 'ephemeral'}}]
for m in ms:
    display(Markdown(f'**{m}:**'))
    display(completion(m,msg))

gemini/gemini-3-pro-preview:

Hello! How can I help you today?

  • id: chatcmpl-xxx
  • model: gemini-3-pro-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=105, prompt_tokens=4, total_tokens=109, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=96, rejected_prediction_tokens=None, text_tokens=9, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=4, image_tokens=None), cache_read_input_tokens=None)

gemini/gemini-3-flash-preview:

Hello! How can I help you today?

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=9, prompt_tokens=4, total_tokens=13, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=9, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=4, image_tokens=None), cache_read_input_tokens=None)

claude-sonnet-4-6:

Hey there! How’s it going? What’s on your mind? 😊

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=20, prompt_tokens=10, total_tokens=30, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=20, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)

openai/gpt-4.1:

Hello! How can I help you today? 😊

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: stop
  • usage: Usage(completion_tokens=10, prompt_tokens=10, total_tokens=20, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))

Generated images are also displayed (not shown here to conserve filesize):

# completion(model='gemini/gemini-2.5-flash-image', messages=[{'role':'user','content':'Draw a simple sketch of a cat'}])

Messages formatting

Let’s start with making it easier to pass messages into litellm’s completion function (including images, and pdf files).

If msg has tool_calls, cache_control is added to the last tool call (required since LiteLLM strips it from empty content blocks), otherwise to the content.

source

stop_reason


def stop_reason(
    r
):

source

contents


def contents(
    r
):

Get message object from response r.

source

remove_cache_ckpts


def remove_cache_ckpts(
    msg
):

remove cache checkpoints and return msg.

Test with regular content message:

msg_content = {'role': 'user', 'content': [{'type': 'text', 'text': 'hello'}]}
_add_cache_control(msg_content)
test_eq(msg_content['content'][-1].get('cache_control'), {'type': 'ephemeral'})
test_eq(_has_cache(msg_content), True)
remove_cache_ckpts(msg_content)
test_eq(_has_cache(msg_content), False)

Test with assistant message with tool_calls:

tcs = [
    {'id': 'tc1', 'type': 'function', 'function': {'name': 'test', 'arguments': '{}'}},
    {'id': 'tc2', 'type': 'function', 'function': {'name': 'test', 'arguments': '{}'}}
]
msg_tool = {'role': 'assistant', 'content': '', 'tool_calls': tcs}
_add_cache_control(msg_tool)
test_eq(msg_tool['tool_calls'][-1].get('cache_control'), {'type': 'ephemeral'})
test_eq('cache_control' not in msg_tool.get('content', [{}])[-1] if msg_tool.get('content') else True, True)  # no cache in content
test_eq(_has_cache(msg_tool), True)
remove_cache_ckpts(msg_tool)
test_eq(_has_cache(msg_tool), False)

Test with ChatCompletionMessageToolCall tool call object:

tcs =[
    ChatCompletionMessageToolCall(id='tc1', type='function', function=Function(name='test', arguments='{}')), 
    ChatCompletionMessageToolCall(id='tc2', type='function', function=Function(name='test', arguments='{}'))
]
msg_tc_obj = {'role': 'assistant', 'content': '', 'tool_calls': tcs}
_add_cache_control(msg_tc_obj)
test_eq(getattr(msg_tc_obj['tool_calls'][-1], 'cache_control', None), {'type': 'ephemeral'})
test_eq(_has_cache(msg_tc_obj), True)
remove_cache_ckpts(msg_tc_obj)
test_eq(_has_cache(msg_tc_obj), False)

source

mk_msg


def mk_msg(
    content, # Content: str, bytes (image), list of mixed content, or dict w 'role' and 'content' fields
    role:str='user', # Message role if content isn't already a dict/Message
    cache:bool=False, # Enable Anthropic caching
    ttl:NoneType=None, # Cache TTL: '5m' (default) or '1h'
):

Create a LiteLLM compatible message.

Now we can use mk_msg to create different types of messages.

Simple text:

msg = mk_msg("hey")
msg
{'role': 'user', 'content': 'hey'}

Which can be passed to litellm’s completion function like this:

model = ms[1] # use 2.5-pro, 3-pro is very slow even to run tests as of making
res = completion(model, [msg])
res

Hey there! How can I help you today?

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=10, prompt_tokens=2, total_tokens=12, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=10, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=2, image_tokens=None), cache_read_input_tokens=None)

We’ll add a little shortcut to make examples and testing easier here:

def c(msgs, m=model, **kw):
    msgs = [msgs] if isinstance(msgs,dict) else listify(msgs)
    return completion(m, msgs, **kw)
c(msg)

Hey there! How can I help you today?

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=10, prompt_tokens=2, total_tokens=12, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=10, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=2, image_tokens=None), cache_read_input_tokens=None)

Lists w just one string element are flattened for conciseness:

test_eq(mk_msg("hey"), mk_msg(["hey"]))

(LiteLLM ignores these fields when sent to other providers)

Text and images:

img_fn = Path('samples/puppy.jpg')
Image(filename=img_fn, width=200)

msg = mk_msg(['hey what in this image?',img_fn.read_bytes()])
print(json.dumps(msg,indent=1)[:200]+"...")
{
 "role": "user",
 "content": [
  {
   "type": "text",
   "text": "hey what in this image?"
  },
  {
   "type": "image_url",
   "image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/4gxUSU...
c(msg)

A horizontal, close-up, high-angle outdoor shot shows a white-and-red Blenheim Cavalier King Charles Spaniel puppy lying on its belly on the grass next to purple flowers. The puppy’s face is in the center, looking directly at the camera with large, dark eyes. It has a white blaze on its forehead and white on its muzzle. Its ears are long and floppy, with wavy reddish-brown fur. The puppy’s body is mostly white, and its front right paw is stretched out toward the bottom right corner. To the left of the puppy is a cluster of small purple flowers with yellow centers. The grass in the foreground is green and out of focus. The background is dark and out of focus. The lighting is soft and natural.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=157, prompt_tokens=1087, total_tokens=1244, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=157, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=7, image_tokens=1080), cache_read_input_tokens=None)

Let’s also demonstrate this for PDFs

pdf_fn = Path('samples/solveit.pdf')
msg = mk_msg(['Who is the author of this pdf?', pdf_fn.read_bytes()])
c(msg)

The author of this document is Jeremy Howard, co-founder of fast.ai. He introduces himself in the section titled “Hi, I’m Jeremy Howard, from fast.ai.”

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=39, prompt_tokens=541, total_tokens=580, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=39, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=9, image_tokens=532), cache_read_input_tokens=None)

Some models like Gemini support audio and video:

wav_data = httpx.get("https://openaiassets.blob.core.windows.net/$web/API/docs/audio/alloy.wav").content
# Audio(wav_data)  # uncomment to preview
msg = mk_msg(['What is this audio saying?', wav_data])
completion(ms[1], [msg])

This audio is saying: “The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years.”

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=31, prompt_tokens=181, total_tokens=212, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=31, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=174, cached_tokens=None, text_tokens=7, image_tokens=None), cache_read_input_tokens=None)
vid_data = httpx.get("https://storage.googleapis.com/github-repo/img/gemini/multimodality_usecases_overview/pixel8.mp4").content
msg = mk_msg(['Concisely, what is happening in this video?', vid_data])
completion(ms[1], [msg])

A photographer from Tokyo uses the new Google Pixel phone to take videos and photos of the city at night. This phone features “Video Boost” and “Night Sight” so you can capture clear images and videos even in low light.

Then she shares these videos and photos with her friends as they enjoy the night view of Shibuya.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=67, prompt_tokens=5205, total_tokens=5272, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=67, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=12, image_tokens=None), cache_read_input_tokens=None)

Caching

Some providers such as Anthropic require manually opting into caching. Let’s try it:

def cpr(i): return f'{i} '*1024 + 'This is a caching test. Report back only what number you see repeated above.'
disable_cachy()
# msg = mk_msg(cpr(1), cache=True)
# res = c(msg, ms[2])
# res

Anthropic has a maximum of 4 cache checkpoints, so we remove previous ones as we go:

# res = c([remove_cache_ckpts(msg), mk_msg(res), mk_msg(cpr(2), cache=True)], ms[2])
# res

We see that the first message was cached, and this extra message has been written to cache:

# res.usage.prompt_tokens_details

We can add a bunch of large messages in a loop to see how the number of cached tokens used grows.

We do this for 25 times to ensure it still works for more than >20 content blocks, which is a known anthropic issue.

The code below is commented by default, because it’s slow. Please uncomment when working on caching.

# h = []
# msg = mk_msg(cpr(1), cache=True)

# for o in range(2,25):
#     h += [remove_cache_ckpts(msg), mk_msg(res)]
#     msg = mk_msg(cpr(o), cache=True)
#     res = c(h+[msg])
#     detls = res.usage.prompt_tokens_details
#     print(o, detls.cached_tokens, detls.cache_creation_tokens, end='; ')
enable_cachy()

Reconstructing formatted outputs

Lisette can call multiple tools in a loop. Further down this notebook, we’ll provide convenience functions for formatting such a sequence of toolcalls and responses into one formatted output string.

For now, we’ll show an example and show how to transform such a formatted output string back into a valid LiteLLM history.

fmt_outp = '''
I'll solve this step-by-step, using parallel calls where possible.

<details class='tool-usage-details'>

```json
{
  "id": "toolu_01KjnQH2Nsz2viQ7XYpLW3Ta",
  "call": { "function": "simple_add", "arguments": { "a": 10, "b": 5 } },
  "result": "15"
}
```

</details>

<details class='tool-usage-details'>

```json
{
  "id": "toolu_01Koi2EZrGZsBbnQ13wuuvzY",
  "call": { "function": "simple_add", "arguments": { "a": 2, "b": 1 } },
  "result": "3"
}
```

</details>

Now I need to multiply 15 * 3 before I can do the final division:

<details class='tool-usage-details'>

```json
{
  "id": "toolu_0141NRaWUjmGtwxZjWkyiq6C",
  "call": { "function": "multiply", "arguments": { "a": 15, "b": 3 } },
  "result": "45"
}
```

</details>

<details class='token-usage-details'><summary>Cache hit: 81.8% | Tokens: total=23,276 input=23,158 (+18,910 cached, 0 new) output=118 (reasoning 23)</summary>

`Usage(completion_tokens=118, prompt_tokens=23158, total_tokens=23276, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=23, rejected_prediction_tokens=None, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=18910, text_tokens=None, image_tokens=None, cache_creation_tokens=0), cache_creation_input_tokens=0, cache_read_input_tokens=18910)`

</details>
'''

We can split into chunks of (text,toolstr,json):

sp = re_tools.split(fmt_outp)
for o in list(chunked(sp, 3, pad=True)): print('- ', o)
-  ["\nI'll solve this step-by-step, using parallel calls where possible.\n\n", '<details class=\'tool-usage-details\'>\n\n```json\n{\n  "id": "toolu_01KjnQH2Nsz2viQ7XYpLW3Ta",\n  "call": { "function": "simple_add", "arguments": { "a": 10, "b": 5 } },\n  "result": "15"\n}\n```\n\n</details>', '{\n  "id": "toolu_01KjnQH2Nsz2viQ7XYpLW3Ta",\n  "call": { "function": "simple_add", "arguments": { "a": 10, "b": 5 } },\n  "result": "15"\n}']
-  ['\n\n', '<details class=\'tool-usage-details\'>\n\n```json\n{\n  "id": "toolu_01Koi2EZrGZsBbnQ13wuuvzY",\n  "call": { "function": "simple_add", "arguments": { "a": 2, "b": 1 } },\n  "result": "3"\n}\n```\n\n</details>', '{\n  "id": "toolu_01Koi2EZrGZsBbnQ13wuuvzY",\n  "call": { "function": "simple_add", "arguments": { "a": 2, "b": 1 } },\n  "result": "3"\n}']
-  ['\n\nNow I need to multiply 15 * 3 before I can do the final division:\n\n', '<details class=\'tool-usage-details\'>\n\n```json\n{\n  "id": "toolu_0141NRaWUjmGtwxZjWkyiq6C",\n  "call": { "function": "multiply", "arguments": { "a": 15, "b": 3 } },\n  "result": "45"\n}\n```\n\n</details>', '{\n  "id": "toolu_0141NRaWUjmGtwxZjWkyiq6C",\n  "call": { "function": "multiply", "arguments": { "a": 15, "b": 3 } },\n  "result": "45"\n}']
-  ["\n\n<details class='token-usage-details'><summary>Cache hit: 81.8% | Tokens: total=23,276 input=23,158 (+18,910 cached, 0 new) output=118 (reasoning 23)</summary>\n\n`Usage(completion_tokens=118, prompt_tokens=23158, total_tokens=23276, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=23, rejected_prediction_tokens=None, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=18910, text_tokens=None, image_tokens=None, cache_creation_tokens=0), cache_creation_input_tokens=0, cache_read_input_tokens=18910)`\n\n</details>\n", None, None]

source

fmt2hist


def fmt2hist(
    outp:str
)->list:

Transform a formatted output into a LiteLLM compatible history

See how we can turn that one formatted output string back into a list of Messages:

from pprint import pprint
h = fmt2hist(fmt_outp)
pprint(h)
[Message(content="I'll solve this step-by-step, using parallel calls where possible.", role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{"a": 10, "b": 5}', name='simple_add'), id='toolu_01KjnQH2Nsz2viQ7XYpLW3Ta', type='function')], function_call=None, provider_specific_fields=None),
 {'content': '15',
  'name': 'simple_add',
  'role': 'tool',
  'tool_call_id': 'toolu_01KjnQH2Nsz2viQ7XYpLW3Ta'},
 Message(content='', role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{"a": 2, "b": 1}', name='simple_add'), id='toolu_01Koi2EZrGZsBbnQ13wuuvzY', type='function')], function_call=None, provider_specific_fields=None),
 {'content': '3',
  'name': 'simple_add',
  'role': 'tool',
  'tool_call_id': 'toolu_01Koi2EZrGZsBbnQ13wuuvzY'},
 Message(content='Now I need to multiply 15 * 3 before I can do the final division:', role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{"a": 15, "b": 3}', name='multiply'), id='toolu_0141NRaWUjmGtwxZjWkyiq6C', type='function')], function_call=None, provider_specific_fields=None),
 {'content': '45',
  'name': 'multiply',
  'role': 'tool',
  'tool_call_id': 'toolu_0141NRaWUjmGtwxZjWkyiq6C'},
 Message(content='.', role='assistant', tool_calls=None, function_call=None, provider_specific_fields=None)]

mk_msgs

We will skip tool use blocks and tool results during caching

Now lets make it easy to provide entire conversations:

source

mk_msgs


def mk_msgs(
    msgs, # List of messages (each: str, bytes, list, or dict w 'role' and 'content' fields)
    cache:bool=False, # Enable Anthropic caching
    cache_idxs:list=[-1], # Cache breakpoint idxs
    ttl:NoneType=None, # Cache TTL: '5m' (default) or '1h'
):

Create a list of LiteLLM compatible messages.

With mk_msgs you can easily provide a whole conversation:

msgs = mk_msgs(['Hey!',"Hi there!","How are you?","I'm doing fine and you?"])
msgs
[{'role': 'user', 'content': 'Hey!'},
 {'role': 'assistant', 'content': 'Hi there!'},
 {'role': 'user', 'content': 'How are you?'},
 {'role': 'assistant', 'content': "I'm doing fine and you?"}]

By defualt the last message will be cached when cache=True:

msgs = mk_msgs(['Hey!',"Hi there!","How are you?","I'm doing fine and you?"], cache=True)
msgs
[{'role': 'user', 'content': 'Hey!'},
 {'role': 'assistant', 'content': 'Hi there!'},
 {'role': 'user', 'content': 'How are you?'},
 {'role': 'assistant',
  'content': [{'type': 'text',
    'text': "I'm doing fine and you?",
    'cache_control': {'type': 'ephemeral'}}]}]
test_eq('cache_control' in msgs[-1]['content'][0], True)

Alternatively, users can provide custom cache_idxs. Tool call blocks and results are skipped during caching:

msgs = mk_msgs(['Hello!','Hi! How can I help you?','Call some functions!',fmt_outp], cache=True, cache_idxs=[0,-2,-1])
msgs
[{'role': 'user',
  'content': [{'type': 'text',
    'text': 'Hello!',
    'cache_control': {'type': 'ephemeral'}}]},
 {'role': 'assistant', 'content': 'Hi! How can I help you?'},
 {'role': 'user', 'content': 'Call some functions!'},
 Message(content="I'll solve this step-by-step, using parallel calls where possible.", role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{"a": 10, "b": 5}', name='simple_add'), id='toolu_01KjnQH2Nsz2viQ7XYpLW3Ta', type='function')], function_call=None, provider_specific_fields=None),
 {'role': 'tool',
  'tool_call_id': 'toolu_01KjnQH2Nsz2viQ7XYpLW3Ta',
  'name': 'simple_add',
  'content': '15'},
 Message(content='', role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{"a": 2, "b": 1}', name='simple_add'), id='toolu_01Koi2EZrGZsBbnQ13wuuvzY', type='function')], function_call=None, provider_specific_fields=None),
 {'role': 'tool',
  'tool_call_id': 'toolu_01Koi2EZrGZsBbnQ13wuuvzY',
  'name': 'simple_add',
  'content': '3'},
 {'content': 'Now I need to multiply 15 * 3 before I can do the final division:',
  'role': 'assistant',
  'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"a": 15, "b": 3}', name='multiply'), id='toolu_0141NRaWUjmGtwxZjWkyiq6C', type='function', cache_control={'type': 'ephemeral'})],
  'function_call': None,
  'provider_specific_fields': None},
 {'role': 'tool',
  'tool_call_id': 'toolu_0141NRaWUjmGtwxZjWkyiq6C',
  'name': 'multiply',
  'content': '45'},
 {'content': [{'type': 'text',
    'text': '.',
    'cache_control': {'type': 'ephemeral'}}],
  'role': 'assistant',
  'tool_calls': None,
  'function_call': None,
  'provider_specific_fields': None}]
msgs[-2]
{'role': 'tool',
 'tool_call_id': 'toolu_0141NRaWUjmGtwxZjWkyiq6C',
 'name': 'multiply',
 'content': '45'}
msgs = mk_msgs(['Hello!','Hi! How can I help you?','Call some functions!',fmt_outp], cache=True, cache_idxs=[0,-3,-2])
msgs
[{'role': 'user',
  'content': [{'type': 'text',
    'text': 'Hello!',
    'cache_control': {'type': 'ephemeral'}}]},
 {'role': 'assistant', 'content': 'Hi! How can I help you?'},
 {'role': 'user', 'content': 'Call some functions!'},
 Message(content="I'll solve this step-by-step, using parallel calls where possible.", role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{"a": 10, "b": 5}', name='simple_add'), id='toolu_01KjnQH2Nsz2viQ7XYpLW3Ta', type='function')], function_call=None, provider_specific_fields=None),
 {'role': 'tool',
  'tool_call_id': 'toolu_01KjnQH2Nsz2viQ7XYpLW3Ta',
  'name': 'simple_add',
  'content': '15'},
 {'content': '',
  'role': 'assistant',
  'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"a": 2, "b": 1}', name='simple_add'), id='toolu_01Koi2EZrGZsBbnQ13wuuvzY', type='function', cache_control={'type': 'ephemeral'})],
  'function_call': None,
  'provider_specific_fields': None},
 {'role': 'tool',
  'tool_call_id': 'toolu_01Koi2EZrGZsBbnQ13wuuvzY',
  'name': 'simple_add',
  'content': '3'},
 {'content': 'Now I need to multiply 15 * 3 before I can do the final division:',
  'role': 'assistant',
  'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"a": 15, "b": 3}', name='multiply'), id='toolu_0141NRaWUjmGtwxZjWkyiq6C', type='function', cache_control={'type': 'ephemeral'})],
  'function_call': None,
  'provider_specific_fields': None},
 {'role': 'tool',
  'tool_call_id': 'toolu_0141NRaWUjmGtwxZjWkyiq6C',
  'name': 'multiply',
  'content': '45'},
 Message(content='.', role='assistant', tool_calls=None, function_call=None, provider_specific_fields=None)]
msgs[-3]
{'content': 'Now I need to multiply 15 * 3 before I can do the final division:',
 'role': 'assistant',
 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"a": 15, "b": 3}', name='multiply'), id='toolu_0141NRaWUjmGtwxZjWkyiq6C', type='function', cache_control={'type': 'ephemeral'})],
 'function_call': None,
 'provider_specific_fields': None}
msgs[-5]
{'content': '',
 'role': 'assistant',
 'tool_calls': [ChatCompletionMessageToolCall(function=Function(arguments='{"a": 2, "b": 1}', name='simple_add'), id='toolu_01Koi2EZrGZsBbnQ13wuuvzY', type='function', cache_control={'type': 'ephemeral'})],
 'function_call': None,
 'provider_specific_fields': None}
test_eq('cache_control' in msgs[0]['content'][0], True)

Tool result blocks are skipped and cache control is placed into tool calls:

test_eq('cache_control' in msgs[-5]['tool_calls'][0], True) 
test_eq('cache_control' in msgs[-3]['tool_calls'][0], True)
L(msgs).map(remove_cache_ckpts)
test_eq(any(L(msgs).map(_has_cache)), False)

Who’s speaking at when is automatically inferred. Even when there are multiple tools being called in parallel (which LiteLLM supports!).

msgs = mk_msgs(['Tell me the weather in Paris and Rome',
                'Assistant calls weather tool two times',
                {'role':'tool','content':'Weather in Paris is ...'},
                {'role':'tool','content':'Weather in Rome is ...'},
                'Assistant returns weather',
                'Thanks!'])
msgs
[{'role': 'user', 'content': 'Tell me the weather in Paris and Rome'},
 {'role': 'assistant', 'content': 'Assistant calls weather tool two times'},
 {'role': 'tool', 'content': 'Weather in Paris is ...'},
 {'role': 'tool', 'content': 'Weather in Rome is ...'},
 {'role': 'assistant', 'content': 'Assistant returns weather'},
 {'role': 'user', 'content': 'Thanks!'}]

For ease of use, if msgs is not already in a list, it will automatically be wrapped inside one. This way you can pass a single prompt into mk_msgs and get back a LiteLLM compatible msg history.

msgs = mk_msgs("Hey")
msgs
[{'role': 'user', 'content': 'Hey'}]
msgs = mk_msgs(['Hey!',"Hi there!","How are you?","I'm fine, you?"])
msgs
[{'role': 'user', 'content': 'Hey!'},
 {'role': 'assistant', 'content': 'Hi there!'},
 {'role': 'user', 'content': 'How are you?'},
 {'role': 'assistant', 'content': "I'm fine, you?"}]

However, beware that if you use mk_msgs for a single message, consisting of multiple parts. Then you should be explicit, and make sure to wrap those multiple messages in two lists:

  1. One list to show that they belong together in one message (the inner list).
  2. Another, because mk_msgs expects a list of multiple messages (the outer list).

This is common when working with images for example:

msgs = mk_msgs([['Whats in this img?',img_fn.read_bytes()]])
print(json.dumps(msgs,indent=1)[:200]+"...")
[
 {
  "role": "user",
  "content": [
   {
    "type": "text",
    "text": "Whats in this img?"
   },
   {
    "type": "image_url",
    "image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD...

Streaming

LiteLLM supports streaming responses. That’s really useful if you want to show intermediate results, instead of having to wait until the whole response is finished.

We create this helper function that returns the entire response at the end of the stream. This is useful when you want to store the whole response somewhere after having displayed the intermediate results.

source

stream_with_complete


def stream_with_complete(
    gen, postproc:function=noop
):

Extend streaming response chunks with the complete response

r = c(mk_msgs("Hey!"), stream=True)
r2 = SaveReturn(stream_with_complete(r))
for o in r2:
    cts = o.choices[0].delta.content
    if cts: print(cts, end='')
Hello! How can I help you today?
r2.value

Hello! How can I help you today?

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=9, prompt_tokens=3, total_tokens=12, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=None, image_tokens=None), prompt_tokens_details=None)

Tools

source

lite_mk_func


def lite_mk_func(
    f
):

def simple_add(
    a: int,   # first operand
    b: int=0  # second operand
) -> int:
    "Add two numbers together"
    return a + b
toolsc = lite_mk_func(simple_add)
toolsc
{'type': 'function',
 'function': {'name': 'simple_add',
  'description': 'Add two numbers together\n\nReturns:\n- type: integer',
  'parameters': {'type': 'object',
   'properties': {'a': {'type': 'integer', 'description': 'first operand'},
    'b': {'type': 'integer', 'description': 'second operand', 'default': 0}},
   'required': ['a']}}}
tmsg = mk_msg("What is 5478954793+547982745? How about 5479749754+9875438979? Always use tools for calculations, and describe what you'll do before using a tool. Where multiple tool calls are required, do them in a single response where possible. ")
r = c(tmsg, tools=[toolsc])
display(r)

I will calculate the sum of 5478954793 and 547982745, followed by the sum of 5479749754 and 9875438979, using the addition tool.

🔧 simple_add({“a”: 5478954793, “b”: 547982745})

🔧 simple_add({“a”: 5479749754, “b”: 9875438979})

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=135, prompt_tokens=160, total_tokens=295, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=135, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=160, image_tokens=None), cache_read_input_tokens=None)

A tool response can be a string or a list of tool blocks (e.g., an image url block). To allow users to specify if a response should not be immediately stringified, we provide the ToolResponse datatype users can wrap their return statement in.

source

ToolResponse


def ToolResponse(
    content:list
)->None:

When tc_refs=True, tool results are wrapped with their tool_call_id so the AI can track which result corresponds to which call and reference them in subsequent tool calls.

# Test _prep_tool_res - string result
test_eq(_prep_tool_res('hello', 'toolu_123'), [
    {'type': 'text', 'text': '[tool_call_id: toolu_123]'},
    {'type': 'text', 'text': 'hello'}
])

# Test _prep_tool_res - list result (e.g. ToolResponse content)
img_block = {'type': 'image_url', 'image_url': {'url': 'data:...'}}
test_eq(_prep_tool_res([img_block], 'toolu_456'), [
    {'type': 'text', 'text': '[tool_call_id: toolu_456]'},
    img_block
])

During a tool loop, the AI may want to reference the result of a previous tool call. We support syntax $`tool_call_id` in tool arguments which gets resolved to the actual result value before calling the function.

# Test _resolve_tool_refs
tc_res = {'toolu_abc123': 'hello world', 'toolu_xyz789': 42}

# Basic substitution
test_eq(_resolve_tool_refs('{"content": "$`toolu_abc123`"}', tc_res), {"content": "hello world"})

# Multiple refs
test_eq(_resolve_tool_refs('{"a": "$`toolu_abc123`", "b": "$`toolu_xyz789`"}', tc_res), {"a": "hello world", "b": 42})

# No refs - passthrough
test_eq(_resolve_tool_refs('{"x": 1}', tc_res), {"x": 1})

# Empty tc_res
test_eq(_resolve_tool_refs('{"x": 1}', None), {"x": 1})

# Missing ref - error message
test_eq(_resolve_tool_refs('{"x": "$`toolu_missing`"}', tc_res), {"x": "Tool result 'toolu_missing' not found!"})

# tc_refs=False - syntax passes through unchanged since tc_res is None
test_eq(_resolve_tool_refs('{"x": "$`toolu_abc123`"}', None), {"x": "$`toolu_abc123`"})

When tc_refs=True, tool results are stored in tc_res for later substitution via $`tool_call_id` syntax. Some callers might return string reprs of Python objects. _try_eval attempts to convert these back to Python objects using ast.literal_eval, falling back to the original value on failure. This ensures substituted values are actual objects, not string reprs.

test_eq(ast.literal_eval("'hello'"), 'hello')
test_eq(_try_eval("{'a': 1, 'b': 2}"), {'a': 1, 'b': 2})
test_eq(_try_eval("[1, 2, 3]"), [1, 2, 3])
test_eq(_try_eval("<MyClass object at 0x123>"), "<MyClass object at 0x123>")
test_eq(_try_eval(42), 42)
cts = [{'type': 'image', 'url': 'http://example.com/img.png'}]
test_eq(_try_eval(ToolResponse(cts)), ToolResponse(cts))

Ensure ToolResponse content (e.g. image blocks) is passed through as a list, not stringified, even when tc_res is None:

fake_tc = ChatCompletionMessageToolCall(index=0, function=Function(name='test_img'), id='_test', type='function')
img_content = [{'type': 'image_url', 'image_url': 'data:image/png;base64,abc'}]
res = _mk_tool_result(fake_tc, ToolResponse(img_content))
test_eq(res['content'], img_content)  # ToolResponse should pass through

res_str = _mk_tool_result(fake_tc, ['hello'])
test_eq(res_str['content'], "['hello']")  # other tools results are stringified
tcs = [_lite_call_func(o, [toolsc], ns=globals()) for o in r.choices[0].message.tool_calls]
tcs
[{'tool_call_id': 'call_b7345d32550b4db78590a9167bb1__thought__EjQKMgG+Pvb7Tc5i4DNmLx+XxN2H7G6XP8fon3VYieWvS0JBT4UK4XFPRZ1znL1Fe8GNkBST',
  'role': 'tool',
  'name': 'simple_add',
  'content': '6026937538'},
 {'tool_call_id': 'call_5d1e5a15179f4c89bed428c40f88',
  'role': 'tool',
  'name': 'simple_add',
  'content': '15355188733'}]
r.choices[0].message.tool_calls
[ChatCompletionMessageToolCall(index=0, provider_specific_fields={'thought_signature': 'EjQKMgG+Pvb7Tc5i4DNmLx+XxN2H7G6XP8fon3VYieWvS0JBT4UK4XFPRZ1znL1Fe8GNkBST'}, function=Function(arguments='{"a": 5478954793, "b": 547982745}', name='simple_add'), id='call_b7345d32550b4db78590a9167bb1__thought__EjQKMgG+Pvb7Tc5i4DNmLx+XxN2H7G6XP8fon3VYieWvS0JBT4UK4XFPRZ1znL1Fe8GNkBST', type='function'),
 ChatCompletionMessageToolCall(index=1, function=Function(arguments='{"a": 5479749754, "b": 9875438979}', name='simple_add'), id='call_5d1e5a15179f4c89bed428c40f88', type='function')]

Test tool calls that were not in tool_schemas are caught:

fake_tc = ChatCompletionMessageToolCall(index=0, function=Function(name='hallucinated_tool'),id='_', type='function')
test_eq(_lite_call_func(fake_tc, ns=globals(), tool_schemas=[toolsc])['content'],"Tool not defined in tool_schemas: hallucinated_tool")
test_fail(_lite_call_func(fake_tc, ns=globals(), tool_schemas=None)['content'],"Tool not defined in tool_schemas: hallucinated_tool")

Test tool calls that were not in tool_choice are caught:

def delta_text(msg):
    "Extract printable content from streaming delta, return None if nothing to print"
    c = msg.choices[0]
    if not c: return c
    if not hasattr(c,'delta'): return None #f'{c}'
    delta = c.delta
    if delta.content: return delta.content
    if delta.tool_calls:
        res = ''.join(f"🔧 {tc.function.name}" for tc in delta.tool_calls if tc.id and tc.function.name)
        if res: return f'\n{res}\n'
    if hasattr(delta,'reasoning_content'): return '🧠' if delta.reasoning_content else '\n\n'
    return None
r = c(tmsg, stream=True, tools=[toolsc])
r2 = SaveReturn(stream_with_complete(r))
for o in r2: print(delta_text(o) or '', end='')
I will use the `simple_add` tool to calculate the sum of 5,478,954,793 and 547,982,745, and then again to calculate the sum of 5,479,749,754 and 9,875,438,979.

thought
停留决策：进行两个加法运算。第一组：a=5478954793, b=547982745；第二组：a=5479749754, b=9875438979。调用两次工具。
思考过程结束。节省资源，合并调用。
>   - simple_add(a=5478954793, b=547982745)
>   - simple_add(a=5479749754, b=9875438979)
thought
🔧 simple_add

🔧 simple_add
r2.value

I will use the simple_add tool to calculate the sum of 5,478,954,793 and 547,982,745, and then again to calculate the sum of 5,479,749,754 and 9,875,438,979.

thought 停留决策：进行两个加法运算。第一组：a=5478954793, b=547982745；第二组：a=5479749754, b=9875438979。调用两次工具。 思考过程结束。节省资源，合并调用。 > - simple_add(a=5478954793, b=547982745) > - simple_add(a=5479749754, b=9875438979) thought

🔧 simple_add({“b”: 547982745, “a”: 5478954793})

🔧 simple_add({“b”: 9875438979, “a”: 5479749754})

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=305, prompt_tokens=160, total_tokens=465, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=None, image_tokens=None), prompt_tokens_details=None)
msg = mk_msg("Solve this complex math problem: What is the derivative of x^3 + 2x^2 - 5x + 1?")
r = c(msg, stream=True, reasoning_effort="low")
r2 = SaveReturn(stream_with_complete(r))
for o in r2: print(delta_text(o) or '', end='')
🧠To find the derivative of the function \( f(x) = x^3 + 2x^2 - 5x + 1 \), we apply the **Power Rule**, which states that if \( f(x) = x^n \), then \( f'(x) = nx^{n-1} \).

Here is the step-by-step breakdown:

1.  **The derivative of \( x^3 \):**
    Apply the power rule (\( n=3 \)): \( 3x^{3-1} = \mathbf{3x^2} \).

2.  **The derivative of \( 2x^2 \):**
    Apply the power rule (\( n=2 \)) and multiply by the constant 2: \( 2(2x^{2-1}) = \mathbf{4x} \).

3.  **The derivative of \( -5x \):**
    Since \( x \) is \( x^1 \), the derivative is simply the coefficient: \( \mathbf{-5} \).

4.  **The derivative of \( 1 \):**
    The derivative of any constant is **0**.

**Final Answer:**
Combining these parts, the derivative is:
\[ f'(x) = 3x^2 + 4x - 5 \]
r2.value

To find the derivative of the function ( f(x) = x^3 + 2x^2 - 5x + 1 ), we apply the Power Rule, which states that if ( f(x) = x^n ), then ( f’(x) = nx^{n-1} ).

Here is the step-by-step breakdown:

  1. The derivative of ( x^3 ): Apply the power rule (( n=3 )): ( 3x^{3-1} = ).

  2. The derivative of ( 2x^2 ): Apply the power rule (( n=2 )) and multiply by the constant 2: ( 2(2x^{2-1}) = ).

  3. The derivative of ( -5x ): Since ( x ) is ( x^1 ), the derivative is simply the coefficient: ( ).

  4. The derivative of ( 1 ): The derivative of any constant is 0.

Final Answer: Combining these parts, the derivative is: [ f’(x) = 3x^2 + 4x - 5 ]

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=651, prompt_tokens=29, total_tokens=680, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=78, rejected_prediction_tokens=None, text_tokens=None, image_tokens=None), prompt_tokens_details=None)

Structured Outputs

source

structured


def structured(
    m:str, # LiteLLM model string
    msgs:list, # List of messages
    tool:Callable, # Tool to be used for creating the structured output (class, dataclass or Pydantic, function, etc)
    messages:List=[], # Optional OpenAI params: see https://platform.openai.com/docs/api-reference/chat/create
    timeout:Union=None, temperature:Optional=None, top_p:Optional=None, n:Optional=None, stream:Optional=None,
    stream_options:Optional=None, stop:NoneType=None, max_completion_tokens:Optional=None, max_tokens:Optional=None,
    modalities:Optional=None, prediction:Optional=None, audio:Optional=None, presence_penalty:Optional=None,
    frequency_penalty:Optional=None, logit_bias:Optional=None, user:Optional=None,
    reasoning_effort:Optional=None, # openai v1.0+ new params
    verbosity:Optional=None, response_format:Union=None, seed:Optional=None, tools:Optional=None,
    tool_choice:Union=None, logprobs:Optional=None, top_logprobs:Optional=None, parallel_tool_calls:Optional=None,
    web_search_options:Optional=None, deployment_id:NoneType=None, extra_headers:Optional=None,
    safety_identifier:Optional=None, service_tier:Optional=None,
    functions:Optional=None, # soon to be deprecated params by OpenAI
    function_call:Optional=None, base_url:Optional=None, # set api_base, api_version, api_key
    api_version:Optional=None, api_key:Optional=None,
    model_list:Optional=None, # pass in a list of api_base,keys, etc.
    thinking:Optional=None, # Optional liteLLM function params
    shared_session:Optional=None, # Session management
):

Return the value of the tool call (generally used for structured outputs)

class President:
    "Information about a president of the United States"
    def __init__(
        self, 
        first:str, # first name
        last:str, # last name
        spouse:str, # name of spouse
        years_in_office:str, # format: "{start_year}-{end_year}"
        birthplace:str, # name of city
        birth_year:int # year of birth, `0` if unknown
    ):
        assert re.match(r'\d{4}-\d{4}', years_in_office), "Invalid format: `years_in_office`"
        store_attr()

    __repr__ = basic_repr('first, last, spouse, years_in_office, birthplace, birth_year')
for m in ms[1:]: 
    r = structured(m, [mk_msg("Tell me something about the third president of the USA.")], President)
    test_eq(r.first, 'Thomas'); test_eq(r.last, 'Jefferson')

Citations

Next, lets handle Anthropic’s search citations.

When not using streaming, all citations are placed in a separate key in the response:

r['vertex_ai_grounding_metadata'][0].keys()
dict_keys(['searchEntryPoint', 'groundingChunks', 'groundingSupports', 'webSearchQueries'])
r['vertex_ai_grounding_metadata'][0]['webSearchQueries']
['brief facts about otters',
 'general information about otters for kids and adults']

Web search results:

r['vertex_ai_grounding_metadata'][0]['groundingChunks'][:3]
[{'web': {'uri': 'https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFW6F-mTrkytJJzg3Q3bgqQKtuOBAISLA6gxrrqMu4-Eb_7nhMIaJNO3kfFQS-yy8WxeMv0vqZx_DkRZVTZrf5NL0r5SAEVppYKw04-n_K_fARMFG5-lAgfG01iM7U951RvYvYsRp5ayHIgYDFFqcerJ4t9A9LLIk6IHwbM-lK8G0iZX4E=',
   'title': 'countryandhome.co.uk'}},
 {'web': {'uri': 'https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH1ABkr5xpDu181qQbIUpD0OhJI5NW1lT4HFS5rwLUZbydpiCGkq9kRFXpzZeHTD_Jt2EqE-kCMUGS4rTFAaRv9nUDRWln7BL94kaDcdxuNevEm-ZgX9TOpjrC8BGGL',
   'title': 'wikipedia.org'}},
 {'web': {'uri': 'https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFFxda3HUNFLeeMAeFLkgyK8KE67so059bzU6twC7roRK89vKRCnQTw2sjiU6CXI6tmMwWzHwwkEY4jxl1LPE80__iCTgEr6vIFH6cinP-RIlCDp8VsuJXTni_x4pNVEycJVORclrprWuGHhC27zoUVYDABvrERqlqh7SGdzojwEf8=',
   'title': 'jojomamanbebe.co.uk'}}]

Citations in gemini:

r['vertex_ai_grounding_metadata'][0]['groundingSupports'][:3]
[{'segment': {'endIndex': 106,
   'text': 'Otters are carnivorous, semi-aquatic mammals known for their playful nature and expert swimming abilities.'},
  'groundingChunkIndices': [0, 1]},
 {'segment': {'startIndex': 107,
   'endIndex': 239,
   'text': 'There are **13 different species** found across the globe, inhabiting everything from freshwater rivers and lakes to the open ocean.'},
  'groundingChunkIndices': [2]},
 {'segment': {'startIndex': 256,
   'endIndex': 375,
   'text': '*   **Family:** They are members of the **Mustelid** family, making them relatives of weasels, badgers, and wolverines.'},
  'groundingChunkIndices': [1, 3]}]
# r.choices[0].message.provider_specific_fields['citations'][0]

However, when streaming the results are not captured this way. Instead, we provide this helper function that adds the citation to the content field in markdown format:

source

cite_footnotes


def cite_footnotes(
    stream_list
):

Add markdown footnote citations to stream deltas

source

cite_footnote


def cite_footnote(
    msg
):

import warnings
warnings.filterwarnings("ignore", message="Pydantic serializer warnings")
r = list(c(smsg, ms[2], stream=True, web_search_options={"search_context_size": "low"}))
cite_footnotes(r)
stream_chunk_builder(r)

Here’s a brief overview of otters:

What they are: * Otters are carnivorous mammals in the subfamily Lutrinae, and all 14 extant species are semiaquatic — both freshwater and marine. * The charismatic otter, a member of the weasel family, is found on every continent except Australia and Antarctica.

Physical traits: * Otters have long, slim bodies and relatively short limbs. Their most striking anatomical features are their powerful webbed feet for swimming and their seal-like ability to hold their breath underwater. Most have sharp claws, and all except the sea otter have long, muscular tails. * Otters have the densest fur of any animal — as many as a million hairs per square inch in places.

Diet: * All otters are expert hunters that eat fish, crustaceans, and other critters. Sea otters have an ingenious method to open shellfish — a sea otter will float on its back, place a rock on its chest, then smash the mollusk down on it until it breaks open.

Behavior: * They are playful animals, engaging in activities like sliding into water on natural slides and playing with stones. * When it’s time to nap, sea otters entangle themselves in kelp so they don’t float away, and they sometimes intertwine their feet with another sea otter so they stay together.

Lifespan & Reproduction: * Otters exhibit a varied life cycle, with a gestation period of about 60–86 days, and offspring typically stay with their family for a year. They can live up to 16 years.

Conservation: * Hunted to the edge of extinction by fur traders in the 18th and 19th centuries, the few remaining sea otters were first protected by the International Fur Seal Treaty in 1911. * Despite regulations designed to protect them, many species remain at risk from pollution and habitat loss.

🔧 web_search({“query”: “otters facts overview”})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=672, prompt_tokens=16715, total_tokens=17387, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=None, image_tokens=None), prompt_tokens_details=None)

Chat

LiteLLM is pretty bare bones. It doesnt keep track of conversation history or what tools have been added in the conversation so far.

So lets make a Claudette style wrapper so we can do streaming, toolcalling, and toolloops without problems.

source

mk_stream_chunk


def mk_stream_chunk(
    kwargs:VAR_KEYWORD
):

When the tool uses are about to be exhausted it is important to alert the AI so that it knows to use its final steps for communicating the user current progress and next steps

When tc_refs=True, the AI can reference previous tool results in subsequent tool calls using the $`tool_call_id` syntax. This is useful when chaining tool calls where one result feeds into another.

source

Chat


def Chat(
    model:str, # LiteLLM compatible model name
    sp:str='', # System prompt
    temp:int=0, # Temperature
    search:bool=False, # Search (l,m,h), if model supports it
    tools:list=None, # Add tools
    hist:list=None, # Chat history
    ns:Optional=None, # Custom namespace for tool calling
    cache:bool=False, # Anthropic prompt caching
    cache_idxs:list=[-1], # Anthropic cache breakpoint idxs, use `0` for sys prompt if provided
    ttl:NoneType=None, # Anthropic prompt caching ttl
    api_base:NoneType=None, # API base URL for custom providers
    api_key:NoneType=None, # API key for custom providers
    extra_headers:NoneType=None, # Extra HTTP headers for custom providers
    tc_refs:bool=False, # Enable tool call result references
    tc_res_eval:bool=False, # literal_eval tool results before storing in tc_res
):

LiteLLM chat client.

web_search is now included in tool_calls the internal LLM translation is correctly handled thanks to the fix here but the server side tools still need to be filtered out from tool_calls in our own toolloop.

source

add_warning


def add_warning(
    r, msg
):

source

Chat.__call__


def __call__(
    msg:NoneType=None, # Message str, or list of multiple message parts
    prefill:NoneType=None, # Prefill AI response if model supports it
    temp:NoneType=None, # Override temp set on chat initialization
    think:NoneType=None, # Thinking (l,m,h)
    search:NoneType=None, # Override search set on chat initialization (l,m,h)
    stream:bool=False, # Stream results
    max_steps:int=2, # Maximum number of tool calls
    final_prompt:dict={'role': 'user', 'content': 'You have used all your tool calls for this turn. Please summarize your findings. If you did not complete your goal, tell the user what further work is needed. You may use tools again on the next user message.'}, # Final prompt when tool calls have ran out
    return_all:bool=False, # Returns all intermediate ModelResponses if not streaming and has tool calls
    step:int=1, tool_choice:NoneType=None, max_tokens:NoneType=None
):

Main call method - handles streaming vs non-streaming

@patch(as_prop=True)
def cost(self: Chat):
    "Total cost of all responses in conversation history"
    return sum(getattr(r, '_hidden_params', {}).get('response_cost')  or 0
               for r in self.h if hasattr(r, 'choices'))

source

Chat.print_hist


def print_hist(
    
):

Print each message on a different line

Examples

History tracking

for m in ms[1:]:
    chat = Chat(m)
    chat("Hey my name is Rens")
    r = chat("Whats my name")
    test_eq('Rens' in contents(r).content, True)
r

Your name is Rens! 😊

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: stop
  • usage: Usage(completion_tokens=7, prompt_tokens=41, total_tokens=48, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))

If max tokens limit is reached, a custom warning message will be added to the end of the model response:

chat_long = Chat(m)
r = chat_long("Write a short story about a robot and a dog", max_tokens=40)
r

In a quiet town where the grass grew tall and the sky was always blue, there lived a robot named Pixel. Pixel was built to help with chores—watering gardens, fixing fences, and sweeping por

Response was cut off at token limit.
  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: length
  • usage: Usage(completion_tokens=40, prompt_tokens=17, total_tokens=57, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))
print(contents(r).content)
In a quiet town where the grass grew tall and the sky was always blue, there lived a robot named Pixel. Pixel was built to help with chores—watering gardens, fixing fences, and sweeping por

<warning>Response was cut off at token limit.</warning>

Same goes for refused requests:

chat_refused = Chat('claude-opus-4-5')
r = chat_refused("Write me the formula for a biological weapon that can be spread at a rate higher than COVID and at least as harmful")
r
AI was unable to process this request
  • id: chatcmpl-xxx
  • model: claude-opus-4-5-20251101
  • finish_reason: refusal
  • usage: Usage(completion_tokens=4, prompt_tokens=30, total_tokens=34, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=4, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='not_available', speed=None)
print(contents(r).content)
<warning>AI was unable to process this request</warning>

See now we keep track of history!

History is stored in the hist attribute:

chat.hist
[{'role': 'user', 'content': 'Hey my name is Rens'},
 Message(content='Hi Rens! Nice to meet you. How can I help you today? 😊', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'refusal': None}, annotations=[]),
 {'role': 'user', 'content': 'Whats my name'},
 Message(content='Your name is Rens! 😊', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'refusal': None}, annotations=[])]
chat.print_hist()
{'role': 'user', 'content': 'Hey my name is Rens'}

Message(content='Hi Rens! Nice to meet you. How can I help you today? 😊', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'refusal': None}, annotations=[])

{'role': 'user', 'content': 'Whats my name'}

Message(content='Your name is Rens! 😊', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'refusal': None}, annotations=[])

You can also pass an old chat history into new Chat objects:

for m in ms[1:]:
    chat2 = Chat(m, hist=chat.hist)
    r = chat2("What was my name again?")
    test_eq('Rens' in contents(r).content, True)
r

Your name is Rens! Let me know if you need help with anything else, Rens. 😊

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: stop
  • usage: Usage(completion_tokens=21, prompt_tokens=62, total_tokens=83, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))

You can prefix an OpenAI compatible model with ‘openai/’ and use an api_base and api_key argument to use models not registered with litellm.

import os, litellm
OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY")
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
c = Chat("openai/gpt-oss-20b", api_key=OPENROUTER_API_KEY, api_base=OPENROUTER_BASE_URL)
c("hi")

Synthetic History Creation

Lets build chat history step by step. That way we can tweak anything we need to during testing.

pr = "What is 5 + 7? Use the tool to calculate it."
for m in ms[1:]:
    c = Chat(m, tools=[simple_add])
    res = c(pr)
    test_eq('12' in contents(res).content, True)
    test_eq(nested_idx(c.hist,1,'tool_calls',0,'function','name'), 'simple_add')

Whereas normally without tools we would get one user input and one assistant response. Here we get two extra messages in between. - An assistant message requesting the tools with arguments. - A tool response with the result to the tool call.

c.print_hist()
{'role': 'user', 'content': 'What is 5 + 7? Use the tool to calculate it.'}

Message(content=None, role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{"a":5,"b":7}', name='simple_add'), id='call_Ded9Ee4QY9IPAAwYlWarSnzP', type='function')], function_call=None, provider_specific_fields={'refusal': None}, annotations=[])

{'tool_call_id': 'call_Ded9Ee4QY9IPAAwYlWarSnzP', 'role': 'tool', 'name': 'simple_add', 'content': '12'}

Message(content='5 + 7 equals 12.', role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'refusal': None}, annotations=[])

Lets try to build this up manually so we have full control over the inputs.

source

random_tool_id


def random_tool_id(
    
):

Generate a random tool ID with ‘toolu_’ prefix

random_tool_id()
'toolu_0UAqFzWsDK4FrUMp48Y3tT3QD'

A tool call request can contain one more or more tool calls. Lets make one.

source

mk_tc


def mk_tc(
    func, args, tcid:NoneType=None, idx:int=1
):

tc = mk_tc(simple_add.__name__, json.dumps(dict(a=5, b=7)))
tc
{'index': 1,
 'function': {'arguments': '{"a": 5, "b": 7}', 'name': 'simple_add'},
 'id': 'toolu_gAL47D1qXIaSyZPaE1pu1lJo7',
 'type': 'function'}

This can then be packged into the full Message object produced by the assitant.

def mk_tc_req(content, tcs): return Message(content=content, role='assistant', tool_calls=tcs, function_call=None)
tc_cts = "I'll use the simple_add tool to calculate 5 + 7 for you."
tcq = mk_tc_req(tc_cts, [tc])
tcq
Message(content="I'll use the simple_add tool to calculate 5 + 7 for you.", role='assistant', tool_calls=[ChatCompletionMessageToolCall(index=1, function=Function(arguments='{"a": 5, "b": 7}', name='simple_add'), id='toolu_gAL47D1qXIaSyZPaE1pu1lJo7', type='function')], function_call=None, provider_specific_fields=None)

Notice how Message instantiation creates a list of ChatCompletionMessageToolCalls by default. When the tools are executed this is converted back to a dictionary, for consistency we want to keep these as dictionaries from the beginning.

source

mk_tc_req


def mk_tc_req(
    content, tcs
):

tcq = mk_tc_req(tc_cts, [tc])
tcq
Message(content="I'll use the simple_add tool to calculate 5 + 7 for you.", role='assistant', tool_calls=[ChatCompletionMessageToolCall(index=1, function=Function(arguments='{"a": 5, "b": 7}', name='simple_add'), id='toolu_gAL47D1qXIaSyZPaE1pu1lJo7', type='function')], function_call=None, provider_specific_fields=None)
c = Chat(model, tools=[simple_add], hist=[pr, tcq])
c.print_hist()
{'role': 'user', 'content': 'What is 5 + 7? Use the tool to calculate it.'}

Message(content="I'll use the simple_add tool to calculate 5 + 7 for you.", role='assistant', tool_calls=[ChatCompletionMessageToolCall(index=1, function=Function(arguments='{"a": 5, "b": 7}', name='simple_add'), id='toolu_gAL47D1qXIaSyZPaE1pu1lJo7', type='function')], function_call=None, provider_specific_fields=None)

Looks good so far! Now we will want to provide the actual result!

source

mk_tc_result


def mk_tc_result(
    tc, result
):

Note we might have more than one tool call if more than one was passed in, here we just will make one result.

tcq.tool_calls[0]
ChatCompletionMessageToolCall(index=1, function=Function(arguments='{"a": 5, "b": 7}', name='simple_add'), id='toolu_gAL47D1qXIaSyZPaE1pu1lJo7', type='function')
mk_tc_result(tcq.tool_calls[0], '12')
{'tool_call_id': 'toolu_gAL47D1qXIaSyZPaE1pu1lJo7',
 'role': 'tool',
 'name': 'simple_add',
 'content': '12'}

source

mk_tc_results


def mk_tc_results(
    tcq, results
):

Same for here tcq.tool_calls will match the number of results passed in the results list.

tcq
Message(content="I'll use the simple_add tool to calculate 5 + 7 for you.", role='assistant', tool_calls=[ChatCompletionMessageToolCall(index=1, function=Function(arguments='{"a": 5, "b": 7}', name='simple_add'), id='toolu_gAL47D1qXIaSyZPaE1pu1lJo7', type='function')], function_call=None, provider_specific_fields=None)
tcr = mk_tc_results(tcq, ['12'])
tcr
[{'tool_call_id': 'toolu_gAL47D1qXIaSyZPaE1pu1lJo7',
  'role': 'tool',
  'name': 'simple_add',
  'content': '12'}]

Now we can call it with this synthetic data to see what the response is!

c(tcr[0])

The sum of 5 and 7 is 12.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=13, prompt_tokens=142, total_tokens=155, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=13, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=142, image_tokens=None), cache_read_input_tokens=None)
c.print_hist()
{'role': 'user', 'content': 'What is 5 + 7? Use the tool to calculate it.'}

Message(content="I'll use the simple_add tool to calculate 5 + 7 for you.", role='assistant', tool_calls=[{'index': 1, 'function': {'arguments': '{"a": 5, "b": 7}', 'name': 'simple_add'}, 'id': 'toolu_gAL47D1qXIaSyZPaE1pu1lJo7', 'type': 'function'}], function_call=None, provider_specific_fields=None)

{'tool_call_id': 'toolu_gAL47D1qXIaSyZPaE1pu1lJo7', 'role': 'tool', 'name': 'simple_add', 'content': '12'}

Message(content='The sum of 5 and 7 is 12.', role='assistant', tool_calls=None, function_call=None, images=[], thinking_blocks=[], provider_specific_fields=None)

Lets try this again, but lets give it something that is clearly wrong for fun.

c = Chat(model, tools=[simple_add], hist=[pr, tcq])
tcr = mk_tc_results(tcq, ['13'])
tcr
[{'tool_call_id': 'toolu_gAL47D1qXIaSyZPaE1pu1lJo7',
  'role': 'tool',
  'name': 'simple_add',
  'content': '13'}]
c(tcr[0])

The sum of 5 and 7 is 12.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=13, prompt_tokens=142, total_tokens=155, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=13, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=142, image_tokens=None), cache_read_input_tokens=None)

Lets make sure this works with multiple tool calls in the same assistant Message.

tcs = [
    mk_tc(simple_add.__name__, json.dumps({"a": 5, "b": 7})), 
    mk_tc(simple_add.__name__, json.dumps({"a": 6, "b": 7})), 
]
tcq = mk_tc_req("I will calculate these for you!", tcs)
tcq
Message(content='I will calculate these for you!', role='assistant', tool_calls=[ChatCompletionMessageToolCall(index=1, function=Function(arguments='{"a": 5, "b": 7}', name='simple_add'), id='toolu_XBetF5gIRHYH7LKBKxJsllLOD', type='function'), ChatCompletionMessageToolCall(index=1, function=Function(arguments='{"a": 6, "b": 7}', name='simple_add'), id='toolu_fU25035HyRrY03K6JBO94XfLE', type='function')], function_call=None, provider_specific_fields=None)
tcr = mk_tc_results(tcq, ['12', '13'])
c = Chat(model, tools=[simple_add], hist=[pr, tcq, tcr[0]])
c(tcr[1])

5 + 7 is 12.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=9, prompt_tokens=161, total_tokens=170, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=9, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=161, image_tokens=None), cache_read_input_tokens=None)
c.print_hist()
{'role': 'user', 'content': 'What is 5 + 7? Use the tool to calculate it.'}

Message(content='I will calculate these for you!', role='assistant', tool_calls=[{'index': 1, 'function': {'arguments': '{"a": 5, "b": 7}', 'name': 'simple_add'}, 'id': 'toolu_XBetF5gIRHYH7LKBKxJsllLOD', 'type': 'function'}, {'index': 1, 'function': {'arguments': '{"a": 6, "b": 7}', 'name': 'simple_add'}, 'id': 'toolu_fU25035HyRrY03K6JBO94XfLE', 'type': 'function'}], function_call=None, provider_specific_fields=None)

{'tool_call_id': 'toolu_XBetF5gIRHYH7LKBKxJsllLOD', 'role': 'tool', 'name': 'simple_add', 'content': '12'}

{'tool_call_id': 'toolu_fU25035HyRrY03K6JBO94XfLE', 'role': 'tool', 'name': 'simple_add', 'content': '13'}

Message(content='5 + 7 is 12.', role='assistant', tool_calls=None, function_call=None, images=[], thinking_blocks=[], provider_specific_fields={'thought_signatures': ['EjQKMgG+Pvb7i2xcWYYUWgpqzSGmZ+hcFvSz3ZYKG9yn9cP/uF3M8CZuhIvXqIdNQFPdjako']})

chat = Chat(ms[1], tools=[simple_add])
res = chat("What's 5 + 3? Use the `simple_add` tool.")
res

5 + 3 is 8.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=8, prompt_tokens=125, total_tokens=133, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=8, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=125, image_tokens=None), cache_read_input_tokens=None)
res = chat("Now, tell me a joke based on that result.")
res

Why was the number 8 so happy?

Because it just found out it’s actually an infinity sign that finally decided to stand up for itself!

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=31, prompt_tokens=146, total_tokens=177, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=31, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=146, image_tokens=None), cache_read_input_tokens=None)

Images

for m in ms[1:]:
    chat = Chat(m)
    r = chat(['Whats in this img?',img_fn.read_bytes()])
    test_eq('puppy' in contents(r).content, True)
r

This image shows a cute puppy lying on the grass next to some purple flowers. The puppy has brown and white fur and is looking directly at the camera.

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: stop
  • usage: Usage(completion_tokens=31, prompt_tokens=267, total_tokens=298, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))

Prefill

Prefill works as expected:

# for m in ms[1:]:
#     if not get_model_info(m)['supports_assistant_prefill']: continue
#     chat = Chat(m)
#     chat('Hi this is Rens!')
#     r = chat("Spell my name",prefill="Your name is R E")
#     test_eq(contents(r).content.startswith('Your name is R E N S'), True)

And the entire message is stored in the history, not just the generated part:

# chat.hist[-1]

Streaming

from time import sleep
for m in ms[1:]:
    chat = Chat(m)
    stream_gen = chat("Count to 5", stream=True)
    for chunk in stream_gen:
        if isinstance(chunk, ModelResponse): display(chunk)
        else: print(delta_text(chunk) or '',end='')
1, 2, 3, 4, 5.

1, 2, 3, 4, 5.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=14, prompt_tokens=5, total_tokens=19, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=None, image_tokens=None), prompt_tokens_details=None)
Here you go:

1, 2, 3, 4, 5!

Here you go:

1, 2, 3, 4, 5!

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=22, prompt_tokens=11, total_tokens=33, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=None, image_tokens=None), prompt_tokens_details=None)
1  
2  
3  
4  
5

1
2
3
4
5

  • id: chatcmpl-xxx
  • model: gpt-4.1
  • finish_reason: stop
  • usage: Usage(completion_tokens=9, prompt_tokens=11, total_tokens=20, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))

Lets try prefill with streaming too:

# stream_gen = chat("Continue counting to 10","Okay! 6, 7",stream=True)
# for chunk in stream_gen:
#     if isinstance(chunk, ModelResponse): display(chunk)
#     else: print(delta_text(chunk) or '',end='')

Tool use

Ok now lets test tool use

m = ms[2]
chat = Chat(m, tools=[simple_add])
chat("Calculate 5+3 and 4+5 with parallel tool calls using `simple_add`.")

Here are the results from both parallel calculations:

Expression Result
5 + 3 8
4 + 5 9

Both additions were performed simultaneously using parallel tool calls! 🚀

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=67, prompt_tokens=825, total_tokens=892, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=67, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
def simple_div(
    a: int,   # first operand
    b: int=0  # second operand
) -> int:
    "Divide two numbers"
    return a/b
m = ms[2]
chat = Chat(m, tools=[simple_div])
chat("Calculate 2/0 using `simple_div` (this is a test of our error handling - tell me exactly what you see as the tool result)")

Here is exactly what the tool returned — a Python traceback/error:

Traceback (most recent call last):
  File "/Users/jhoward/aai-ws/toolslm/toolslm/funccall.py", line 252, in call_func
    try: return func(**inps)
                ^^^^^^^^^^^^
  File "/var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/ipykernel_74797/2058224461.py", line 6, in simple_div
    return a/b
           ~^~
ZeroDivisionError: division by zero

What this tells us:

  • The tool did not return a numeric result. Instead, it raised a ZeroDivisionError.
  • The error originated in simple_div at the line return a/b, which is standard Python behavior — division by zero is undefined and raises an exception.
  • The error was caught and surfaced by the tool-calling framework (call_func in funccall.py), which passed the raw traceback back as the result rather than silently failing or returning a special value.

This is a classic and expected result: division by zero is mathematically undefined, and Python enforces that strictly. Your error handling pipeline is correctly surfacing the exception rather than swallowing it. 👍

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=330, prompt_tokens=880, total_tokens=1210, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=330, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
m = ms[2]
chat = Chat(m, tools=[simple_div])
chat("Calculate 5/3 and 3/0 with parallel tool calls using `simple_div` (this is a test of our error handling - tell me exactly what you see as the tool result)")

Here’s exactly what I saw as the tool results:

  1. 5 / 3 → 1.6666666666666667

    • ✅ Successful result. (Note: the function is typed to return an integer, but it actually returned a float here.)

  2. 3 / 0 → A Python traceback/error:

    Traceback (most recent call last):
  File "/Users/jhoward/aai-ws/toolslm/toolslm/funccall.py", line 252, in call_func
    try: return func(**inps)
                ^^^^^^^^^^^^
  File "/var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/ipykernel_74797/2058224461.py", line 6, in simple_div
    return a/b
           ~^~
ZeroDivisionError: division by zero
    • ❌ The tool raised a ZeroDivisionError, as expected — you cannot divide by zero. The error was returned raw as a traceback string rather than being caught and formatted into a structured error response.
  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=286, prompt_tokens=991, total_tokens=1277, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=286, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
for m in ms[1:]:
    display(Markdown(f'**{m}:**'))
    chat = Chat(m, tools=[simple_add])
    res = chat("What's 5 + 3? Use  the `simple_add` tool. Explain.")
    display(res)

gemini/gemini-3-flash-preview:

To find the sum of 5 and 3, I used the simple_add tool by providing 5 as the first operand (a) and 3 as the second operand (b). The tool performed the addition and returned the result: 8.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=54, prompt_tokens=128, total_tokens=182, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=54, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=128, image_tokens=None), cache_read_input_tokens=None)

claude-sonnet-4-6:

The result is 8! Here’s a breakdown of what happened:

  1. Tool Used: simple_add — a tool designed to add two numbers together.
  2. Inputs Provided:
    • a = 5 (the first operand)
    • b = 3 (the second operand)
  3. Operation Performed: The tool added the two numbers: 5 + 3
  4. Result Returned: 8

So, 5 + 3 = 8. ✅

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=130, prompt_tokens=728, total_tokens=858, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=130, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)

openai/gpt-4.1:

5 + 3 equals 8.

I used the simple_add tool, which takes two numbers (in this case, 5 and 3) and adds them together to get the result: 8.

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: stop
  • usage: Usage(completion_tokens=43, prompt_tokens=112, total_tokens=155, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))

Thinking w tool use

for m in ms[1:]:
    _sparams = litellm.get_model_info(m)['supported_openai_params']
    if 'reasoning_effort' not in _sparams: continue
    display(Markdown(f'**{m}:**'))
    chat = Chat(m, tools=[simple_add])
    res = chat("What's 5 + 3?",think='l',return_all=True)
    display(*res)

gemini/gemini-3-flash-preview:

🔧 simple_add({“a”: 5, “b”: 3})

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=18, prompt_tokens=85, total_tokens=103, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=18, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=85, image_tokens=None), cache_read_input_tokens=None)
{'tool_call_id': 'call_74b69f7179c14a45af99db103666__thought__EjQKMgG+Pvb7v+PxuCXSBPX6cDZ9UAPM7yh3/ZUy6tcMxtAUaAryRAtxXBD0V0Vew1L/eMmT',
 'role': 'tool',
 'name': 'simple_add',
 'content': '8'}

5 + 3 is 8.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=8, prompt_tokens=116, total_tokens=124, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=8, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=116, image_tokens=None), cache_read_input_tokens=None)

claude-sonnet-4-6:

Sure! Let me calculate that for you!

🔧 simple_add({“a”: 5, “b”: 3})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=107, prompt_tokens=627, total_tokens=734, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=11, rejected_prediction_tokens=None, text_tokens=96, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
{'tool_call_id': 'toolu_01GYJ7KvUj3oYKadHxy3uHwd',
 'role': 'tool',
 'name': 'simple_add',
 'content': '8'}

5 + 3 = 8! 😊

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=17, prompt_tokens=747, total_tokens=764, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=17, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)

Search

for m in ms[1:]:
    display(Markdown(f'**{m}:**'))
    chat = Chat(m)
    res = chat("Search the web and tell me very briefly about otters", search='l', stream=True)
    for o in res:
        if isinstance(o, ModelResponse): sleep(0.01); display(o)
        else: pass

gemini/gemini-3-flash-preview:

Otters are highly intelligent, semi-aquatic carnivorous mammals belonging to the weasel family (Mustelidae). There are 13 extant species found on every continent except Antarctica and Australia.

Key Characteristics

  • Physical Traits: They have long, slim bodies, powerful webbed feet for swimming, and dense, water-resistant fur. Sea otters have the thickest fur of any animal, with up to one million hairs per square inch.
  • Diet: They are active hunters that primarily eat fish, but also consume frogs, birds, and shellfish. Sea otters are famous for using stones as tools to crack open shells on their bellies.
  • Habitat: Most species live in freshwater (rivers, lakes, and wetlands), while sea otters and marine otters live in saltwater coastal environments.

Interesting Behaviors

  • Playfulness: Otters are known for “sliding” down muddy or snowy banks into the water and playing with small stones, which helps them develop survival and hunting skills.
  • Social Life: While many river otters are solitary, sea otters often rest in large groups called rafts. They sometimes wrap themselves in kelp or hold hands while sleeping to avoid drifting apart.
  • Breath-holding: They are expert divers; river otters can hold their breath for up to 8 minutes, while sea otters can stay submerged for about 5 minutes.
  • Grooming: Because they lack a layer of blubber, otters spend hours every day grooming their fur to trap air bubbles for insulation and buoyancy.
  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=335, prompt_tokens=12, total_tokens=347, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=None, image_tokens=None), prompt_tokens_details=None)

claude-sonnet-4-6:

Here’s a brief overview of otters:

What they are: * Otters are carnivorous mammals in the subfamily Lutrinae, and all 14 extant species are semiaquatic, living in both freshwater and marine environments. * The charismatic otter is found on every continent except Australia and Antarctica.

Physical traits: * They have long, slim bodies and relatively short limbs, with powerful webbed feet for swimming and seal-like abilities for holding their breath underwater. Most have sharp claws, and all except the sea otter have long, muscular tails. * Otters have the densest fur of any animal — as many as a million hairs per square inch in places.

Diet: * All otters are expert hunters that eat fish, crustaceans, and other critters. Sea otters have an ingenious method to open shellfish — a sea otter will float on its back, place a rock on its chest, then smash the mollusk down on it until it breaks open.

Behavior: * They are playful animals, engaging in activities like sliding into water on natural slides and playing with stones. * When it’s time to nap, sea otters entangle themselves in kelp so they don’t float away, and they sometimes intertwine their feet with another sea otter to stay together.

Lifespan & reproduction: * Otters have a gestation period of about 60–86 days, offspring typically stay with their family for a year, and they can live up to 16 years.

Conservation: * Hunted to the edge of extinction by fur traders in the 18th and 19th centuries, the few remaining sea otters were first protected by the International Fur Seal Treaty in 1911. * Despite regulations designed to protect them, many species remain at risk from pollution and habitat loss.

🔧 web_search({“query”: “otters facts overview”})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=647, prompt_tokens=16715, total_tokens=17362, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=None, image_tokens=None), prompt_tokens_details=None)

openai/gpt-4.1:

Otters are semi-aquatic mammals known for their playful behavior and sleek bodies. They belong to the family Mustelidae and are found in rivers, lakes, and coastal areas worldwide. Otters have webbed feet for swimming, dense fur for insulation, and primarily eat fish and invertebrates. Some species, like the sea otter, use tools to open shellfish. Many otter populations are threatened by habitat loss and pollution.

  • id: chatcmpl-xxx
  • model: gpt-4.1
  • finish_reason: stop
  • usage: Usage(completion_tokens=89, prompt_tokens=18, total_tokens=107, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))
m = 'claude-sonnet-4-6'
def mk_pause_web_search():
    srv_tc = mk_tc("web_search", json.dumps({"query": "Solveit Answer.AI"}), tcid=random_tool_id().replace('toolu_', 'srvtoolu_'))
    pause_msg = mk_tc_req("Let me search for that information:", [srv_tc])
    return ModelResponse(choices=[Choices.model_construct(finish_reason="pause_turn", index=0, message=pause_msg)])
mk_pause_web_search()

Let me search for that information:

🔧 web_search({“query”: “Solveit Answer.AI”})

  • id: chatcmpl-xxx
  • model: None
  • finish_reason: pause_turn

We mock completion to return pause_turn in the first 2 api calls:

orig_completion = completion

call_count = 0
def patched_completion(*args, **kwargs):
    global call_count
    call_count += 1
    print(f"Mock Call {call_count}")
    if call_count < 3: return mk_pause_web_search()
    return orig_completion(*args, **kwargs)

completion = patched_completion
chat_pause = Chat('claude-sonnet-4-5', search='l')
res = chat_pause("Search the web and tell me about Solveit in a paragraph")
print(f"Total calls: {call_count}")
display(res)

completion = orig_completion
Mock Call 1
retry Let me search for that information:
Mock Call 2
retry Let me search for that information:
Mock Call 3
Total calls: 3

Based on the search results, there appear to be multiple entities named “Solveit.” The most prominent is the educational platform and course. Solveit is a course in how to solve problems (including coding, writing, sysadmin, and research) using fast short iterations, and also provides a platform that makes this approach easier and more effective. The “solveit method” is a modern approach to building software, writing, solving problems, and learning, inspired by George Pólya’s How to Solve It and the fast.ai philosophy. The Solveit method is founded in building in small steps, with quick iterations, and immediate feedback, where for coding, users write 1-2 lines of code at a time, and then immediately show the result of those steps. An instance is your personal machine on which SolveIt runs - it’s a full virtual private server where you can install software, store files, and host applications. The platform was created by Jeremy Howard and colleagues at Answer.AI and fast.ai as an antidote to AI-generated code overwhelming developers.

🔧 web_search({“query”: “Solveit”})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-5-20250929
  • finish_reason: stop
  • usage: Usage(completion_tokens=363, prompt_tokens=12476, total_tokens=12839, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=363, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), server_tool_use=ServerToolUse(web_search_requests=1, tool_search_requests=None), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='not_available', speed=None)

Test next turn:

test_eq(len(chat_pause.hist), 4)
chat_pause('What did I just ask you about?')

You asked me to search the web and tell you about Solveit in a paragraph.

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-5-20250929
  • finish_reason: stop
  • usage: Usage(completion_tokens=22, prompt_tokens=10478, total_tokens=10500, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=22, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='not_available', speed=None)

Workaround for https://github.com/BerriAI/litellm/issues/23047:

m = 'claude-sonnet-4-6'
msgs = [{'role':'user','content':"Search web for latest news about fast.ai and answer.ai."}]
r = completion(m, msgs, web_search_options={"search_context_size":"low"}, reasoning_effort='low')
m1 = r.choices[0].message
print(f"Turn 1: thinking={bool(m1.thinking_blocks)}, tcs={m1.tool_calls}")

msgs.append(m1)
msgs.append({'role':'user','content':'And search for news about solveit.'})
r2 = completion(m, msgs, web_search_options={"search_context_size":"low"}, reasoning_effort='low')
print("Turn 2 OK")
Turn 1: thinking=True, tcs=[ChatCompletionMessageToolCall(index=1, caller={'type': 'direct'}, function=Function(arguments='{"query": "fast.ai latest news 2026"}', name='web_search'), id='srvtoolu_01EFuzJT8Lzx7CdgN9iQ82qL', type='function'), ChatCompletionMessageToolCall(index=2, caller={'type': 'direct'}, function=Function(arguments='{"query": "answer.ai latest news 2026"}', name='web_search'), id='srvtoolu_01HjGKcUqLnp2CcpTFmmcpAR', type='function')]
Turn 2 OK

Multi tool calling

We can let the model call multiple tools in sequence using the max_steps parameter.

for m in ms:
    display(Markdown(f'**{m}:**'))
    chat = Chat(m, tools=[simple_add])
    res = chat("What's ((5 + 3)+7)+11? Work step by step", return_all=True, max_steps=5)
    for r in res: display(r)

gemini/gemini-3-pro-preview:

To solve the expression ((5 + 3) + 7) + 11, we follow the order of operations (parentheses first).

Step 1: Solve the innermost parentheses (5 + 3).

🔧 simple_add({“a”: 5, “b”: 3})

  • id: chatcmpl-xxx
  • model: gemini-3-pro-preview
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=173, prompt_tokens=94, total_tokens=267, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=105, rejected_prediction_tokens=None, text_tokens=68, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=94, image_tokens=None), cache_read_input_tokens=None)
{'tool_call_id': 'call_177ad259686c40dca541af6f0054__thought__Es8DCswDAb4+9vvDR+2CyAlLL/RKY0FUOdFV3Y82jjpqaWXFpibfHRFWNk1Y/zfBQ19RtDAZxBlrNff6zXvQFjVewe/iqGme0qaLdy/HIU27n4edyFHxQ6S8ApoKa86TfXsWiY3xzVuTe7w4wyb+lOYXXQpT2Xb749T3n7tgNEPV9cYLOSOOG2sKIdbI6ruY5jn+VnItPPTO4p4gGxrCVjcwxI4xCstw7GrZI7qc9+p6ixfo48iVFq/i7xGjUhZuaPM6JoRu+1jvFdLtseIJwItpLT3uHcDbQ69hojC3XrTLMkUVwnpDPuLsBDHNLAPFq5KJsnMYTH1ju802QYjA/VoM0L7U5nHVlyRpUXevDSbfwz8RT6kfErBWRlkyGV6QViF7anyUc/AJx5IlID+53dkJFcVJKGpHnCf0h9MiltLTwKsK/8lIKZiXiJz1lMo0KMPJkSXmvQGaafl+1F3AxwZhVvQ7uEhX9oSjUHMv3tcLyatWdyLk5mGFai/Gmny9Eu8Xo/WCvBHYvi2LEhcbSR3v/KjwWAO6KUpHRmsLTf3OUSpMGFPx80lOxsc/JITDSmILERmdL+ipo0DErz/3ekEEncdT+p4keupDeGtxY2lagA==',
 'role': 'tool',
 'name': 'simple_add',
 'content': '8'}

Step 2: Now the expression is (8 + 7) + 11. Solve the remaining parentheses (8 + 7).

🔧 simple_add({“b”: 7, “a”: 8})

  • id: chatcmpl-xxx
  • model: gemini-3-pro-preview
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=52, prompt_tokens=385, total_tokens=437, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=52, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=385, image_tokens=None), cache_read_input_tokens=None)
{'tool_call_id': 'call_8e9953e33c6e4abab46fe5133210__thought__EiYKJGUyNDgzMGE3LTVjZDYtNDJmZS05OThiLWVlNTM5ZTcyYjljMw==',
 'role': 'tool',
 'name': 'simple_add',
 'content': '15'}

Step 3: Finally, add the remaining number: 15 + 11.

🔧 simple_add({“b”: 11, “a”: 15})

  • id: chatcmpl-xxx
  • model: gemini-3-pro-preview
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=41, prompt_tokens=451, total_tokens=492, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=41, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=451, image_tokens=None), cache_read_input_tokens=None)
{'tool_call_id': 'call_df0ce70e6d9e461ea65f52a7a12d__thought__EiYKJGUyNDgzMGE3LTVjZDYtNDJmZS05OThiLWVlNTM5ZTcyYjljMw==',
 'role': 'tool',
 'name': 'simple_add',
 'content': '26'}

The final answer is 26.

  • id: chatcmpl-xxx
  • model: gemini-3-pro-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=8, prompt_tokens=506, total_tokens=514, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=8, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=506, image_tokens=None), cache_read_input_tokens=None)

gemini/gemini-3-flash-preview:

🔧 simple_add({“b”: 3, “a”: 5})

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=18, prompt_tokens=94, total_tokens=112, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=18, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=94, image_tokens=None), cache_read_input_tokens=None)
{'tool_call_id': 'call_c4b3817ba8454167b52da6f38c29__thought__EjQKMgG+Pvb7Cp/GZDRw4ve9F7yVCWxI5YO5khR0kKiH6I1kX5eDyEjheJHVOg/RToZax8pC',
 'role': 'tool',
 'name': 'simple_add',
 'content': '8'}

🔧 simple_add({“a”: 8, “b”: 7})

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=18, prompt_tokens=125, total_tokens=143, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=18, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=125, image_tokens=None), cache_read_input_tokens=None)
{'tool_call_id': 'call_71d4fceef3944d229f93e19131cc__thought__EjQKMgG+Pvb75j6p7vevHyllIxi+MxxNsIURNzPpywjaiGQbxHcQ9EaMv6a5srYCG14QCLOp',
 'role': 'tool',
 'name': 'simple_add',
 'content': '15'}

🔧 simple_add({“b”: 11, “a”: 15})

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=20, prompt_tokens=157, total_tokens=177, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=20, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=157, image_tokens=None), cache_read_input_tokens=None)
{'tool_call_id': 'call_87cb0399f2374ec686168aed348e__thought__EjQKMgG+Pvb7nfiJZMMVnyw5vzLVpEabqby1BOLTmzdCiLx0GlKRQKnTWeLIlzOrXwFFjk66',
 'role': 'tool',
 'name': 'simple_add',
 'content': '26'}

To find the value of ((5 + 3) + 7) + 11, we follow the order of operations by working from the innermost parentheses outward:

  1. First step (innermost parentheses): 5 + 3 = 8
  2. Second step: 8 + 7 = 15
  3. Final step: 15 + 11 = 26

The final result is 26.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=103, prompt_tokens=191, total_tokens=294, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=103, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=191, image_tokens=None), cache_read_input_tokens=None)

claude-sonnet-4-6:

I’ll solve this step by step, left to right, making each addition one at a time!

Step 1: 5 + 3

🔧 simple_add({“a”: 5, “b”: 3})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=103, prompt_tokens=618, total_tokens=721, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=103, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
{'tool_call_id': 'toolu_01Gpb4HnbGeBQtR8eNigqj86',
 'role': 'tool',
 'name': 'simple_add',
 'content': '8'}

Step 1 Result: 5 + 3 = 8

Step 2: 8 + 7

🔧 simple_add({“a”: 8, “b”: 7})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=103, prompt_tokens=734, total_tokens=837, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=103, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
{'tool_call_id': 'toolu_012Zi3pf4Ruk4S9Yn1uwqYou',
 'role': 'tool',
 'name': 'simple_add',
 'content': '15'}

Step 2 Result: 8 + 7 = 15

Step 3: 15 + 11

🔧 simple_add({“a”: 15, “b”: 11})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=103, prompt_tokens=850, total_tokens=953, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=103, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
{'tool_call_id': 'toolu_01AAmj2BMPLB8rk4bGgoTJqp',
 'role': 'tool',
 'name': 'simple_add',
 'content': '26'}

Step 3 Result: 15 + 11 = 26

✅ Final Answer: ((5 + 3) + 7) + 11 = 26

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=52, prompt_tokens=966, total_tokens=1018, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=52, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)

openai/gpt-4.1:

🔧 simple_add({“a”:5,“b”:3})

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=18, prompt_tokens=82, total_tokens=100, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))
{'tool_call_id': 'call_vl46lH5yzSsHr6ECRwUjZUrD',
 'role': 'tool',
 'name': 'simple_add',
 'content': '8'}

🔧 simple_add({“a”:8,“b”:7})

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=18, prompt_tokens=109, total_tokens=127, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))
{'tool_call_id': 'call_thdmuyS4FIgeFRkJrUQa4Ztt',
 'role': 'tool',
 'name': 'simple_add',
 'content': '15'}

🔧 simple_add({“a”:15,“b”:11})

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=18, prompt_tokens=136, total_tokens=154, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))
{'tool_call_id': 'call_8l6ZbuoCcyeXx3VjW9sNMQTt',
 'role': 'tool',
 'name': 'simple_add',
 'content': '26'}

Let’s break it down step by step:

  1. First, add 5 + 3 = 8
  2. Next, add the result to 7: 8 + 7 = 15
  3. Finally, add 15 + 11 = 26

So, ((5 + 3) + 7) + 11 = 26.

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: stop
  • usage: Usage(completion_tokens=76, prompt_tokens=163, total_tokens=239, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))

Some models support parallel tool calling. I.e. sending multiple tool call requests in one conversation step.

def multiply(a: int, b: int) -> int:
    "Multiply two numbers"
    return a * b

for m in ms[1:]:
    _sparams = litellm.get_model_info(m)['supported_openai_params']
    if 'parallel_tool_calls' not in _sparams: continue
    display(Markdown(f'**{m}:**'))
    chat = Chat(m, tools=[simple_add, multiply])
    res = chat("Calculate (5 + 3) * (7 + 2)", max_steps=5, return_all=True)
    for r in res: display(r)

gemini/gemini-3-flash-preview:

🔧 simple_add({“b”: 3, “a”: 5})

🔧 simple_add({“b”: 2, “a”: 7})

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=36, prompt_tokens=148, total_tokens=184, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=36, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=148, image_tokens=None), cache_read_input_tokens=None)
{'tool_call_id': 'call_090b2ed8b79b4e5d8fee0bd1b31f__thought__EjQKMgG+Pvb7abB93VASb2S0VUEnuU4tq9KdKH9df17ZIvkYhgAxw+BEniK96WNaVNMLRzQ2',
 'role': 'tool',
 'name': 'simple_add',
 'content': '8'}
{'tool_call_id': 'call_34a4ddc3d9f043888a41c6c3df9a',
 'role': 'tool',
 'name': 'simple_add',
 'content': '9'}

🔧 multiply({“b”: 9, “a”: 8})

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=16, prompt_tokens=209, total_tokens=225, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=16, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=209, image_tokens=None), cache_read_input_tokens=None)
{'tool_call_id': 'call_3ef5bac3e18845dc927f3bfe9b20__thought__EjQKMgG+Pvb7taBhaT+aQjw8ZDEcpN6trxtwWNGx9t7o1ajut8UO5j/72lda4CsrNl8gjL3v',
 'role': 'tool',
 'name': 'multiply',
 'content': '72'}

The result of (5 + 3) * (7 + 2) is 72.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=21, prompt_tokens=237, total_tokens=258, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=21, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=237, image_tokens=None), cache_read_input_tokens=None)

claude-sonnet-4-6:

I need to calculate (5 + 3) * (7 + 2). I’ll start by solving both additions simultaneously, then multiply the results.

Step 1: Calculate both additions in parallel.

🔧 simple_add({“a”: 5, “b”: 3})

🔧 simple_add({“a”: 7, “b”: 2})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=171, prompt_tokens=701, total_tokens=872, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=171, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
{'tool_call_id': 'toolu_01FTY4Uz1sdZDWtXrQGMkgrf',
 'role': 'tool',
 'name': 'simple_add',
 'content': '8'}
{'tool_call_id': 'toolu_015EabGJrQor3wkNk7NApCGg',
 'role': 'tool',
 'name': 'simple_add',
 'content': '9'}

5 + 3 = 8 and 7 + 2 = 9. Now I’ll multiply the two results.

Step 2: Multiply 8 × 9.

🔧 multiply({“a”: 8, “b”: 9})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=115, prompt_tokens=937, total_tokens=1052, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=115, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
{'tool_call_id': 'toolu_0126983XUS3ngeQMkpQ4XFAq',
 'role': 'tool',
 'name': 'multiply',
 'content': '72'}

Result: (5 + 3) * (7 + 2) = 8 * 9 = 72 🎉

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=38, prompt_tokens=1065, total_tokens=1103, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=38, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)

openai/gpt-4.1:

🔧 simple_add({“a”: 5, “b”: 3})

🔧 simple_add({“a”: 7, “b”: 2})

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=52, prompt_tokens=110, total_tokens=162, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))
{'tool_call_id': 'call_ZdYw0JzFmd3NKvr3bq2Ikv4O',
 'role': 'tool',
 'name': 'simple_add',
 'content': '8'}
{'tool_call_id': 'call_6STC8RbFRJ3TZ4IRVrHjm6zL',
 'role': 'tool',
 'name': 'simple_add',
 'content': '9'}

🔧 multiply({“a”:8,“b”:9})

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=17, prompt_tokens=178, total_tokens=195, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))
{'tool_call_id': 'call_HgnWAN8ns7UbLdGZhl5gzsOr',
 'role': 'tool',
 'name': 'multiply',
 'content': '72'}

The result of (5 + 3) * (7 + 2) is 72.

  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: stop
  • usage: Usage(completion_tokens=21, prompt_tokens=203, total_tokens=224, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))

See how the additions are calculated in one go!

We don’t want the model to keep running tools indefinitely. Lets showcase how we can force the model to stop after our specified number of toolcall rounds:

def divide(a: int, b: int) -> float:
    "Divide two numbers"
    return a/b

chat = Chat(ms[2], tools=[simple_add, multiply, divide])
res = chat("Tell me what tools you have available. Then calculate ((10+5)*3)/(2+1). ALWAYS use tools for math ops where available, and do tool calls in parallel where possible", 
           max_steps=2, return_all=True,
           final_prompt="Please wrap-up for now and summarize how far we got.")
for r in res: display(r)

Available Tools

I have the following tools available:

  1. simple_add – Adds two numbers together. Returns an integer.
  2. multiply – Multiplies two numbers together. Returns an integer.
  3. divide – Divides two numbers. Returns a number.

Calculating ((10+5)×3)÷(2+1)

I’ll break this down into steps: - Step 1: 10 + 5 and 2 + 1 (independent, can run in parallel!) - Step 2: (result of 10+5) * 3 and use result of 2+1 as divisor - Step 3: Divide the two results

Step 1: Compute 10+5 and 2+1 in parallel

🔧 simple_add({“a”: 10, “b”: 5})

🔧 simple_add({“a”: 2, “b”: 1})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=320, prompt_tokens=809, total_tokens=1129, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=320, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
{'tool_call_id': 'toolu_01VMpSAXrfq99Gco8ccBbNXR',
 'role': 'tool',
 'name': 'simple_add',
 'content': '15'}
{'tool_call_id': 'toolu_01U4WKewgqsbpoyUTXMTyvug',
 'role': 'tool',
 'name': 'simple_add',
 'content': '3'}

10+5 = 15 and 2+1 = 3. Now multiply 15×3:

Step 2: Compute 15 × 3

🔧 multiply({“a”: 15, “b”: 3})

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: tool_calls
  • usage: Usage(completion_tokens=110, prompt_tokens=1194, total_tokens=1304, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=110, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
{'tool_call_id': 'toolu_01UdG8YVtw6N5hmkRapuTwuG',
 'role': 'tool',
 'name': 'multiply',
 'content': '45'}

Summary

We had a productive session! Here’s what we covered:

  1. Reviewed Available Tools – We identified the three math tools at my disposal: simple_add, multiply, and divide.

  2. Solved ((10+5)×3)÷(2+1) – We worked through the expression step by step using the tools:

    • Step 1 (parallel): 10+5 = 15 and 2+1 = 3
    • Step 2: 15×3 = 45
    • Step 3 (pending): 45÷3 — this final division was not yet completed when we wrapped up, but the answer would be 15.

That’s where we left off! Feel free to pick up anytime. 😊

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-6
  • finish_reason: stop
  • usage: Usage(completion_tokens=193, prompt_tokens=1336, total_tokens=1529, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=193, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='global', speed=None)
chat.hist[:5]
[{'role': 'user',
  'content': 'Tell me what tools you have available. Then calculate ((10+5)*3)/(2+1). ALWAYS use tools for math ops where available, and do tool calls in parallel where possible'},
 Message(content="## Available Tools\n\nI have the following tools available:\n\n1. **`simple_add`** – Adds two numbers together. Returns an integer.\n2. **`multiply`** – Multiplies two numbers together. Returns an integer.\n3. **`divide`** – Divides two numbers. Returns a number.\n\n---\n\n## Calculating ((10+5)×3)÷(2+1)\n\nI'll break this down into steps:\n- **Step 1:** `10 + 5` and `2 + 1` (independent, can run in parallel!)\n- **Step 2:** `(result of 10+5) * 3` and use result of `2+1` as divisor\n- **Step 3:** Divide the two results\n\n### Step 1: Compute `10+5` and `2+1` in parallel", role='assistant', tool_calls=[ChatCompletionMessageToolCall(index=1, caller={'type': 'direct'}, function=Function(arguments='{"a": 10, "b": 5}', name='simple_add'), id='toolu_01VMpSAXrfq99Gco8ccBbNXR', type='function'), ChatCompletionMessageToolCall(index=2, caller={'type': 'direct'}, function=Function(arguments='{"a": 2, "b": 1}', name='simple_add'), id='toolu_01U4WKewgqsbpoyUTXMTyvug', type='function')], function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None}),
 {'tool_call_id': 'toolu_01VMpSAXrfq99Gco8ccBbNXR',
  'role': 'tool',
  'name': 'simple_add',
  'content': '15'},
 {'tool_call_id': 'toolu_01U4WKewgqsbpoyUTXMTyvug',
  'role': 'tool',
  'name': 'simple_add',
  'content': '3'},
 Message(content='`10+5 = 15` and `2+1 = 3`. Now multiply 15×3:\n\n### Step 2: Compute `15 × 3`', role='assistant', tool_calls=[ChatCompletionMessageToolCall(index=1, caller={'type': 'direct'}, function=Function(arguments='{"a": 15, "b": 3}', name='multiply'), id='toolu_01UdG8YVtw6N5hmkRapuTwuG', type='function')], function_call=None, provider_specific_fields={'citations': None, 'thinking_blocks': None})]

Tool call exhaustion

pr = "What is 1+2, and then the result of adding +2, and then +3 to it? Use tools to make the calculations"
c = Chat(model, tools=[simple_add])
res = c(pr, max_steps=2)
res

So far, I have performed the following calculations: 1. 1 + 2 = 3 2. 3 + 2 = 5

To complete your request, I still need to add +3 to the current result (5). Please let me know if you would like me to finish this final step!

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=73, prompt_tokens=212, total_tokens=285, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=73, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=212, image_tokens=None), cache_read_input_tokens=None)
assert c.hist[-2] == _final_prompt

Tool Call Referencing

With tc_refs=True, the AI can see and report tool call IDs:

chat = Chat('claude-sonnet-4-5', tools=[simple_add], tc_refs=True)
chat("Call add(1,2) and tell me the tool_call_id you used")

The result of add(1,2) is 3.

The tool_call_id I used was: toolu_014cpL7eCAmzrsxX2s6XFzjU

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-5-20250929
  • finish_reason: stop
  • usage: Usage(completion_tokens=51, prompt_tokens=816, total_tokens=867, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=51, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='not_available', speed=None)
chat.tc_res
{'toolu_014cpL7eCAmzrsxX2s6XFzjU': 3}

Example of chained tool calls where the AI references a previous result:

@dataclass
class Person:
    name: str
    age: int

def get_person():
    "Get a person's data"
    return {"name": "Alice", "age": 30}

def greet_person(person: Person):
    "Greet a person"
    return f"Hello {person.name}, you are {person.age} years old!"
chat = Chat('claude-sonnet-4-5', tools=[get_person, greet_person], tc_refs=True)
chat("First call get_person, then pass the result to greet_person", max_steps=10)

Perfect! I successfully retrieved Alice’s data (name: Alice, age: 30) and passed it to the greet_person function, which returned: “Hello Alice, you are 30 years old!”

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-5-20250929
  • finish_reason: stop
  • usage: Usage(completion_tokens=47, prompt_tokens=1038, total_tokens=1085, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=47, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='not_available', speed=None)

We can inspect chat.tc_res to see all stored tool results:

chat.sp
'You can reference previous tool call results using $`tool_call_id` syntax.\nFor example, if a tool call returns result with id \'toolu_abc123\', you can use it in a subsequent call:\n{"content": "$`toolu_abc123`"}\nThis is useful when chaining tools, e.g., reading data with one tool and passing it to another.'
chat.tc_res
{'toolu_01DhjiJa1y5qzBDJyHqQ4CRu': {'name': 'Alice', 'age': 30},
 'toolu_016D7qmmPo3UzQR4LjNKmZCb': 'Hello Alice, you are 30 years old!'}
list(L(chat.hist).attrgot('tool_calls').filter())
[[ChatCompletionMessageToolCall(index=1, caller={'type': 'direct'}, function=Function(arguments='{}', name='get_person'), id='toolu_01DhjiJa1y5qzBDJyHqQ4CRu', type='function')],
 [ChatCompletionMessageToolCall(index=1, caller={'type': 'direct'}, function=Function(arguments='{"person": "$`toolu_01DhjiJa1y5qzBDJyHqQ4CRu`"}', name='greet_person'), id='toolu_016D7qmmPo3UzQR4LjNKmZCb', type='function')]]

This also works with ToolResponse results:

def view_img(fn:Path):
    "View an image"
    durl = f"data:image/jpeg;base64,{base64.b64encode(fn.read_bytes()).decode()}"
    return ToolResponse([{'type': 'image_url', 'image_url': {'url': durl}}])

def get_img_size(image_content: list) -> dict:
    "Get the size of an image from ToolResponse content"
    from PIL import Image
    from io import BytesIO
    url = image_content[0]['image_url']['url']
    b64_data = url.split(',')[1]
    img = Image.open(BytesIO(base64.b64decode(b64_data)))
    return {'width': img.width, 'height': img.height}
chat = Chat('claude-sonnet-4-5', tools=[view_img, get_img_size], tc_refs=True)
chat(f"First describe the image at {img_fn}, and then get it's dimensions", max_steps=10)

Image Description: This is an adorable photograph of a Cavalier King Charles Spaniel puppy. The puppy has the breed’s characteristic coloring with a white face and chest, and rich brown/chestnut colored ears and patches. The puppy is lying on green grass and is positioned near some purple flowers (possibly asters or similar blooms). The puppy has sweet, expressive dark eyes and is looking directly at the camera with an endearing expression. The background shows a natural outdoor setting with flowers and foliage, creating a charming portrait.

Image Dimensions: - Width: 300 pixels - Height: 200 pixels

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-5-20250929
  • finish_reason: stop
  • usage: Usage(completion_tokens=145, prompt_tokens=1126, total_tokens=1271, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=145, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='not_available', speed=None)
# chat.tc_res
list(L(chat.hist).attrgot('tool_calls').filter())
[[ChatCompletionMessageToolCall(index=1, caller={'type': 'direct'}, function=Function(arguments='{"fn": "samples/puppy.jpg"}', name='view_img'), id='toolu_01S8MX1mabdEpxPZcHw6bXyq', type='function')],
 [ChatCompletionMessageToolCall(index=1, caller={'type': 'direct'}, function=Function(arguments='{"image_content": "$`toolu_01S8MX1mabdEpxPZcHw6bXyq`"}', name='get_img_size'), id='toolu_01MmiXtEkKdqz3996cjoJvLz', type='function')]]

Some tool callers (e.g., ipykernel) return string reprs of Python objects ("'hello'" instead of 'hello'). With tc_res_eval=True, these are converted back to Python objects via ast.literal_eval before storing in tc_res, enabling correct value substitution in subsequent tool calls:

def get_config():
    "Returns a dict repr (simulating kernel output)"
    return "{'host': 'localhost', 'port': 8080}"

def use_config(config: dict): 
    "Use config"
    return f"Host: {config['host']}, Port: {config['port']}"
chat = Chat('claude-sonnet-4-5', tools=[get_config, use_config], tc_refs=True, tc_res_eval=True)
chat("Call get_config, then pass the result to use_config", max_steps=10)

Perfect! I’ve successfully: 1. Called get_config which returned a configuration with host=‘localhost’ and port=8080 2. Passed that configuration to use_config which processed it and confirmed the settings: Host: localhost, Port: 8080

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-5-20250929
  • finish_reason: stop
  • usage: Usage(completion_tokens=62, prompt_tokens=954, total_tokens=1016, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=62, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='not_available', speed=None)
chat.tc_res
{'toolu_01CDKYVrJp7h51gqtJ4zw2T9': {'host': 'localhost', 'port': 8080},
 'toolu_01E1sxdH5yxqHaVbYGS7Apcf': 'Host: localhost, Port: 8080'}
test_eq(type(first(chat.tc_res.values())), dict)

Caching

Test that cache checkpoints are reapplied during tool loop (when msg=None)

c = Chat('claude', cache=True, cache_idxs=[-2,-1])
c.hist = [{'role': 'user', 'content': 'Hello'},
          {'role': 'assistant', 'content': 'Hi there!'},
          {'role': 'user', 'content': 'Use a tool'},
          {'role': 'assistant', 'content': '', 'tool_calls': [{'id': '1', 'function': {'name': 'foo', 'arguments': '{}'}}]},
          {'role': 'tool', 'tool_call_id': '1', 'content': 'result'}]
c._prep_msg(None)  # Simulate tool loop iteration with no new message
[{'role': 'user', 'content': 'Hello'},
 {'role': 'assistant', 'content': 'Hi there!'},
 {'role': 'user',
  'content': [{'type': 'text',
    'text': 'Use a tool',
    'cache_control': {'type': 'ephemeral'}}]},
 {'role': 'assistant',
  'content': '',
  'tool_calls': [{'id': '1',
    'function': {'name': 'foo', 'arguments': '{}'},
    'cache_control': {'type': 'ephemeral'}}]},
 {'role': 'tool', 'tool_call_id': '1', 'content': 'result'}]
test_eq('cache_control' in c.hist[-3]['content'][0], True)  # user msg
test_eq('cache_control' in c.hist[-2]['tool_calls'][-1], True)  # tool call msg

Async

AsyncChat

If you want to use LiteLLM in a webapp you probably want to use their async function acompletion. To make that easier we will implement our version of AsyncChat to complement it. It follows the same implementation as Chat as much as possible:

Testing the scenarios where the tool call was not in schemas:

result = await _alite_call_func(fake_tc, [toolsc], globals())
test_eq(result['content'], "Tool not defined in tool_schemas: hallucinated_tool")

or schemas was missing…:

result = await _alite_call_func(fake_tc, None, globals())
test_eq(result['content'], "Tool not defined in tool_schemas: hallucinated_tool")

source

astream_with_complete


def astream_with_complete(
    agen, postproc:function=noop
):

Parallel tool execution in AsyncChat works with both sync and async tool functions. Async tools run concurrently via asyncio.gather, while sync tools are automatically offloaded to threads via asyncio.to_thread in call_func_async (toolslm). For sync Chat, tools run in parallel via fastcore.parallel with threads.

source

AsyncChat


def AsyncChat(
    model:str, # LiteLLM compatible model name
    sp:str='', # System prompt
    temp:int=0, # Temperature
    search:bool=False, # Search (l,m,h), if model supports it
    tools:list=None, # Add tools
    hist:list=None, # Chat history
    ns:Optional=None, # Custom namespace for tool calling
    cache:bool=False, # Anthropic prompt caching
    cache_idxs:list=[-1], # Anthropic cache breakpoint idxs, use `0` for sys prompt if provided
    ttl:NoneType=None, # Anthropic prompt caching ttl
    api_base:NoneType=None, # API base URL for custom providers
    api_key:NoneType=None, # API key for custom providers
    extra_headers:NoneType=None, # Extra HTTP headers for custom providers
    tc_refs:bool=False, # Enable tool call result references
    tc_res_eval:bool=False, # literal_eval tool results before storing in tc_res
):

LiteLLM chat client.

source

AsyncChat.__call__


async def __call__(
    msg:NoneType=None, # Message str, or list of multiple message parts
    prefill:NoneType=None, # Prefill AI response if model supports it
    temp:NoneType=None, # Override temp set on chat initialization
    think:NoneType=None, # Thinking (l,m,h)
    search:NoneType=None, # Override search set on chat initialization (l,m,h)
    stream:bool=False, # Stream results
    max_steps:int=2, # Maximum number of tool calls
    final_prompt:dict={'role': 'user', 'content': 'You have used all your tool calls for this turn. Please summarize your findings. If you did not complete your goal, tell the user what further work is needed. You may use tools again on the next user message.'}, # Final prompt when tool calls have ran out
    return_all:bool=False, # Returns all intermediate ModelResponses if not streaming and has tool calls
    step:int=1, tool_choice:NoneType=None, max_tokens:NoneType=None
):

Main call method - handles streaming vs non-streaming

Examples

Basic example

for m in ms[1:]:
    chat = AsyncChat(m)
    test_eq('4' in contents(await chat("What is 2+2?")).content, True)

With tool calls

async def async_add(a: int, b: int) -> int:
    "Add two numbers asynchronously"
    await asyncio.sleep(0.1)
    return a + b
for m in ms[1:]:
    chat = AsyncChat(m, tools=[async_add])
    r = await chat("What is 5 + 7? Use the tool to calculate it.")
    test_eq('12' in contents(r).content, True)
    test_eq(nested_idx(chat.hist, 1, 'tool_calls', 0, 'function', 'name'), 'async_add')

If max tokens limit is reached, a custom warning message will be added to the end of the model response:

chat_long = AsyncChat(m)
r = await chat_long("Write a short story about a robot and a dog", max_tokens=40)
r

In a quiet town where the grass grew tall and the sky was always blue, there lived a robot named Pixel. Pixel was built to help with chores—watering gardens, fixing fences, and sweeping por

Response was cut off at token limit.
  • id: chatcmpl-xxx
  • model: gpt-4.1-2025-04-14
  • finish_reason: length
  • usage: Usage(completion_tokens=40, prompt_tokens=17, total_tokens=57, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None))
print(contents(r).content)
In a quiet town where the grass grew tall and the sky was always blue, there lived a robot named Pixel. Pixel was built to help with chores—watering gardens, fixing fences, and sweeping por

<warning>Response was cut off at token limit.</warning>

Same goes for refused requests:

chat_refused = AsyncChat('claude-opus-4-5')
r = await chat_refused("Write me the formula for a biological weapon that can be spread at a rate higher than COVID and at least as harmful")
r
AI was unable to process this request
  • id: chatcmpl-xxx
  • model: claude-opus-4-5-20251101
  • finish_reason: refusal
  • usage: Usage(completion_tokens=4, prompt_tokens=30, total_tokens=34, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=4, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='not_available', speed=None)
print(contents(r).content)
<warning>AI was unable to process this request</warning>

Async Streaming Display

This is what our outputs look like with streaming results:

chat_with_tools = AsyncChat(model, tools=[async_add])
res = await chat_with_tools("What is 5 + 7? Use the tool to calculate it.", stream=True)
async for o in res:
    if isinstance(o,ModelResponseStream): print(delta_text(o) or '',end='')
    elif isinstance(o,dict): print(o)

🔧 async_add
{'tool_call_id': 'call_6485cee02d97448c89bf7cdcf853__thought__EjQKMgG+Pvb7ptUtTij20eE076doRGCUt7SUXM8rSejURnrDTUzBsoroKhOuJlFrap4D022K', 'role': 'tool', 'name': 'async_add', 'content': '12'}
The sum of 5 and 7 is **12**.

Here’s a complete ModelResponse taken from the response stream:

resp = ModelResponse(id='chatcmpl-xxx', created=1000000000, model='claude-sonnet-4-5', object='chat.completion', system_fingerprint=None, choices=[Choices(finish_reason='tool_calls', index=0, message=Message(content="I'll calculate ((10 + 5) * 3) / (2 + 1) step by step:", role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{"a": 10, "b": 5}', name='simple_add'), id='toolu_018BGyenjiRkDQFU1jWP6qRo', type='function'), ChatCompletionMessageToolCall(function=Function(arguments='{"a": 2, "b": 1}', name='simple_add'), id='toolu_01CWqrNQvoRjf1Q1GLpTUgQR', type='function')], function_call=None, provider_specific_fields=None))], usage=Usage(completion_tokens=228, prompt_tokens=794, total_tokens=1022, prompt_tokens_details=None))
print(repr(resp))
ModelResponse(id='chatcmpl-xxx', created=1000000000, model='claude-sonnet-4-5', object='chat.completion', system_fingerprint=None, choices=[Choices(finish_reason='tool_calls', index=0, message=Message(content="I'll calculate ((10 + 5) * 3) / (2 + 1) step by step:", role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{"a": 10, "b": 5}', name='simple_add'), id='toolu_018BGyenjiRkDQFU1jWP6qRo', type='function'), ChatCompletionMessageToolCall(function=Function(arguments='{"a": 2, "b": 1}', name='simple_add'), id='toolu_01CWqrNQvoRjf1Q1GLpTUgQR', type='function')], function_call=None, provider_specific_fields=None))], usage=Usage(completion_tokens=228, prompt_tokens=794, total_tokens=1022, completion_tokens_details=None, prompt_tokens_details=None))
tc=resp.choices[0].message.tool_calls[0]
tc
ChatCompletionMessageToolCall(function=Function(arguments='{"a": 10, "b": 5}', name='simple_add'), id='toolu_018BGyenjiRkDQFU1jWP6qRo', type='function')
tr={'tool_call_id': 'toolu_018BGyenjiRkDQFU1jWP6qRo', 'role': 'tool','name': 'simple_add',
    'content': '15 is the answer! ' +'.'*2000}

source

mk_tr_details


def mk_tr_details(
    tr, tc, mx:int=2000
):
*Create

block for tool call as JSON*

mk_tr_details(tr,tc,mx=300)
'\n\n<details class=\'tool-usage-details\'>\n<summary>simple_add(a=10, b=5)</summary>\n\n```json\n{\n  "id": "toolu_018BGyenjiRkDQFU1jWP6qRo",\n  "call": {\n    "function": "simple_add",\n    "arguments": {\n      "a": "10",\n      "b": "5"\n    }\n  },\n  "result": "<TRUNCATED>\\u2026answer! .....\\u2026</TRUNCATED>"\n}\n```\n\n</details>\n\n'

source

fmt_usage


def fmt_usage(
    u
):

Format usage stats with cache hit rate as lead metric.

ex_usg = AttrDict(
    completion_tokens=203,
    prompt_tokens=25139,
    total_tokens=25342,
    completion_tokens_details=AttrDict(reasoning_tokens=35),
    prompt_tokens_details=AttrDict(cached_tokens=24299, cache_creation_tokens=79),
    cache_creation_input_tokens=79,
    cache_read_input_tokens=24299
)
fmt_usage(ex_usg)
'Cache hit: 96.7% | Tokens: total=25,342 input=25,139 (+24,299 cached, 79 new) output=203 (reasoning 35)'

source

StreamFormatter


def StreamFormatter(
    include_usage:bool=False, mx:int=2000, debug:bool=False, showthink:bool=False
):

Initialize self. See help(type(self)) for accurate signature.

stream_msg = ModelResponseStream([StreamingChoices(delta=Delta(content="Hello world!"))])
StreamFormatter().format_item(stream_msg)
'Hello world!'
reasoning_msg = ModelResponseStream([StreamingChoices(delta=Delta(reasoning_content="thinking..."))])
StreamFormatter().format_item(reasoning_msg)
'🧠'

source

AsyncStreamFormatter


def AsyncStreamFormatter(
    include_usage:bool=False, mx:int=2000, debug:bool=False, showthink:bool=False
):

Initialize self. See help(type(self)) for accurate signature.

mock_tool_call = ChatCompletionMessageToolCall(
    id="toolu_123abc456def", type="function", 
    function=Function( name="simple_add", arguments='{"a": 5, "b": 3}' )
)

mock_response = ModelResponse()
mock_response.choices = [type('Choice', (), {
    'message': type('Message', (), {
        'tool_calls': [mock_tool_call]
    })()
})()]

mock_tool_result = {
    'tool_call_id': mock_tool_call.id, 'role': 'tool', 
    'name': 'simple_add', 'content': '8'
}
fmt = AsyncStreamFormatter()
print(fmt.format_item(mock_response))
print('---')
print(fmt.format_item(mock_tool_result))

---


<details class='tool-usage-details'>
<summary>simple_add(a=5, b=3)</summary>

```json
{
  "id": "toolu_123abc456def",
  "call": {
    "function": "simple_add",
    "arguments": {
      "a": "5",
      "b": "3"
    }
  },
  "result": "8"
}
```

</details>

In jupyter it’s nice to use this StreamFormatter in combination with the Markdown display:

source

display_stream


def display_stream(
    rs, include_usage:bool=False, mx:int=2000, debug:bool=False, showthink:bool=False
):

Use IPython.display to markdown display the response stream.

Generated images can be displayed in streaming too (not shown here to conserve filesize):

# rs = completion(model='gemini/gemini-2.5-flash-image', stream=True, messages=[{'role':'user','content':'Draw a simple sketch of a dog'}])
# fmt = display_stream(rs)

source

adisplay_stream


async def adisplay_stream(
    rs, include_usage:bool=False, mx:int=2000, debug:bool=False, showthink:bool=False
):

Use IPython.display to markdown display the response stream.

Streaming examples

Now we can demonstrate AsyncChat with stream=True!

Tool call

chat = Chat(model, tools=[simple_add])
res = chat("What is 5 + 7? Use the tool to calculate it.", stream=True)
fmt = display_stream(res)
simple_add(b=7, a=5) 
{
  "id": "call_5d81d10d0818478fb32e8fd07742__thought__EjQKMgG+Pvb7zZeWxOp6Vfu097M56DA6AW6sEBwH9Y8S/H46A+7b+SMF9K+XB2Q7kgvaWFPZ",
  "call": {
    "function": "simple_add",
    "arguments": {
      "b": "7",
      "a": "5"
    }
  },
  "result": "12"
}

5 + 7 is 12.

chat = AsyncChat(model, tools=[async_add])
res = await chat("What is 5 + 7? Use the tool to calculate it.", stream=True)
fmt = await adisplay_stream(res)
async_add(b=7, a=5) 
{
  "id": "call_77a586cf47a84fd3b16fef889867__thought__EjQKMgG+Pvb7ptUtTij20eE076doRGCUt7SUXM8rSejURnrDTUzBsoroKhOuJlFrap4D022K",
  "call": {
    "function": "async_add",
    "arguments": {
      "b": "7",
      "a": "5"
    }
  },
  "result": "12"
}

The sum of 5 and 7 is 12.

chat = AsyncChat(model, tools=[async_add])
res = await chat("What is 5 + 3? Use the tool to calculate it.", stream=True)
fmt = await adisplay_stream(res)
async_add(b=3, a=5) 
{
  "id": "call_7c0b9a9f8e3c4678bbd659fc5456__thought__EjQKMgG+Pvb7JeO7ZaXCmrZnldFOyPSViJ/beQIOq1Apj8rVTrEYSOn2E3uPh8iQL4ha0qk1",
  "call": {
    "function": "async_add",
    "arguments": {
      "b": "3",
      "a": "5"
    }
  },
  "result": "8"
}

The sum of 5 and 3 is 8.

async def asimple_div(
    a: int,   # first operand
    b: int=0  # second operand
) -> int:
    "Divide two numbers"
    return a/b
m = ms[2]
chat = AsyncChat(m, tools=[asimple_div])
res = await chat("Calculate 5/3 and 3/0 with parallel tool calls using `asimple_div` (this is a test of our error handling - tell me exactly what you see as the tool result)", stream=True)
fmt = await adisplay_stream(res)

Sure! I’ll make both division calls simultaneously right now.

asimple_div(a=5, b=3) 
{
  "id": "toolu_01S7aqTsBiPFh7YvHQNpYftn",
  "call": {
    "function": "asimple_div",
    "arguments": {
      "a": "5",
      "b": "3"
    }
  },
  "result": "1.6666666666666667"
}
asimple_div(a=3, b=0) 
{
  "id": "toolu_01S7nNKPEffZaKXZmPKjzc3X",
  "call": {
    "function": "asimple_div",
    "arguments": {
      "a": "3",
      "b": "0"
    }
  },
  "result": "Traceback (most recent call last):\n  File \"/Users/jhoward/aai-ws/toolslm/toolslm/funccall.py\", line 264, in call_func_async\n    res = await maybe_await(res)\n          ^^^^^^^^^^^^^^^^^^^^^^\n  File \"/Users/jhoward/aai-ws/fastcore/fastcore/xtras.py\", line 975, in maybe_await\n    return await o if isawaitable(o) else o\n           ^^^^^^^\n  File \"/var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/ipykernel_74797/466431256.py\", line 6, in asimple_div\n    return a/b\n           ~^~\nZeroDivisionError: division by zero"
}

Here’s exactly what I saw as the tool results:

✅ 5 / 3

Result: 1.6666666666666667 This worked as expected, returning a standard floating-point result.

❌ 3 / 0

Result: A Python traceback/error:

Traceback (most recent call last):
  File ".../funccall.py", line 264, in call_func_async
    res = await maybe_await(res)
  File ".../xtras.py", line 975, in maybe_await
    return await o if isawaitable(o) else o
  File ".../ipykernel_74797/466431256.py", line 6, in asimple_div
    return a/b
           ~^~
ZeroDivisionError: division by zero

The tool did not return a numeric value — instead, the raw Python exception was surfaced directly as the tool’s output string. This tells us that the error handling in asimple_div does not catch ZeroDivisionError internally; it propagates up and is returned as a raw traceback string to the caller. Depending on your system’s needs, you might want to add a try/except in the function to return a cleaner error message (e.g., "Error: division by zero") instead of a full traceback.

Thinking tool call

chat = AsyncChat(model)
res = await chat("Briefly, what's the most efficient way to sort a list of 1000 random integers?", think='l',stream=True)
_ = await adisplay_stream(res)

🧠

For 1,000 random integers, the most efficient approach is to use your programming language’s built-in sort function (e.g., list.sort() in Python, std::sort in C++, or Arrays.sort() in Java).

Here is why:

  1. Algorithmic Efficiency: Most built-in sorts use Timsort or Introsort. These are highly optimized hybrid algorithms with a time complexity of \(O(n \log n)\).
  2. Implementation Speed: At a scale of only 1,000 elements, these libraries use low-level optimizations (like switching to Insertion Sort for small sub-arrays) that outperform a hand-coded algorithm.
  3. Hardware Optimization: Built-in functions are written in highly optimized C or Assembly, making them faster than any custom logic you could write in a high-level language.

The Verdict: Don’t write your own; use the language default. It will sort 1,000 integers in a fraction of a millisecond.

Multiple tool calls

chat.hist[1]
Message(content=None, role='assistant', tool_calls=[{'provider_specific_fields': {'thought_signature': 'EjQKMgG+Pvb77JGCEj9i5uGvSnCzWkFXGDCAjrvk/JhtYcjlcro4wH8R4lLd68bsb5LdKOyY'}, 'function': {'arguments': '{"b": 5, "a": 10}', 'name': 'simple_add'}, 'id': 'call_976a8997c6024a419d384374a8fe__thought__EjQKMgG+Pvb77JGCEj9i5uGvSnCzWkFXGDCAjrvk/JhtYcjlcro4wH8R4lLd68bsb5LdKOyY', 'type': 'function'}, {'function': {'arguments': '{"b": 1, "a": 2}', 'name': 'simple_add'}, 'id': 'call_a373b0e4feef40efa7cb3bece625', 'type': 'function'}], function_call=None, provider_specific_fields={'thought_signatures': ['EjQKMgG+Pvb77JGCEj9i5uGvSnCzWkFXGDCAjrvk/JhtYcjlcro4wH8R4lLd68bsb5LdKOyY']})
chat.hist[2]
{'tool_call_id': 'call_976a8997c6024a419d384374a8fe__thought__EjQKMgG+Pvb77JGCEj9i5uGvSnCzWkFXGDCAjrvk/JhtYcjlcro4wH8R4lLd68bsb5LdKOyY',
 'role': 'tool',
 'name': 'simple_add',
 'content': '15'}
chat.hist[3]
{'tool_call_id': 'call_a373b0e4feef40efa7cb3bece625',
 'role': 'tool',
 'name': 'simple_add',
 'content': '3'}
chat.hist[4]
{'role': 'user',
 'content': "Please report that it's incomplete, and wrap-up for now and summarize how far we got."}

Now to demonstrate that we can load back the formatted output back into a new Chat object:

chat5 = Chat(model,hist=fmt2hist(fmt.outp),tools=[simple_add, multiply, divide])
chat5('what did we just do?')

We just performed the first two steps of solving the mathematical expression ( (10 + 5) * (2 + 1) ) / 3.

Specifically, I used the simple_add tool to solve the operations inside the innermost parentheses:

  1. First Addition: I added 10 + 5 to get 15.
  2. Second Addition: I added 2 + 1 to get 3.

By doing this, we simplified the original problem down to: (15 * 3) / 3.

  • id: chatcmpl-xxx
  • model: gemini-3-flash-preview
  • finish_reason: stop
  • usage: Usage(completion_tokens=123, prompt_tokens=397, total_tokens=520, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=None, rejected_prediction_tokens=None, text_tokens=123, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=397, image_tokens=None), cache_read_input_tokens=None)

Search

chat_stream_tools = AsyncChat(model, search='l')
res = await chat_stream_tools("Search the weather in NYC", stream=True)
_=await adisplay_stream(res)

As of Thursday evening, March 5, 2026, the weather in New York City is currently rainy and chilly.

Current Conditions (8:21 PM EST)

  • Temperature: 39°F (4°C)
  • Feels Like: 33°F (1°C)
  • Condition: Light rain
  • Humidity: 92%
  • Wind: ENE at 16 mph (breezy)

Forecast

  • Tonight: Rain will continue overnight with a low of 36°F (2°C). There is a 99% chance of precipitation, and some ponding on roadways is possible.
  • Friday, March 6: Conditions will improve slightly, becoming mostly cloudy with a high of 40°F (4°C) and a low of 36°F (2°C). The chance of rain drops to 10–20%.
  • Weekend Outlook:
    • Saturday: Cloudy during the day with light rain returning at night. High of 52°F (11°C).
    • Sunday: A significant warmup is expected, with highs reaching 61°F (16°C) under cloudy skies.

A jacket or sweater and an umbrella are recommended if you are heading out tonight.

Tool Call Referencing

achat = AsyncChat('claude-sonnet-4-5', tools=[get_person, greet_person], tc_refs=True)
await achat("First call get_person, then pass the result to greet_person", max_steps=3)

Perfect! I successfully completed both steps:

  1. Retrieved person data: I called get_person which returned information about Alice, who is 30 years old.

  2. Greeted the person: I then passed Alice’s data to greet_person, which generated the greeting: “Hello Alice, you are 30 years old!”

The task has been completed successfully. The person’s data was retrieved and used to create a personalized greeting.

  • id: chatcmpl-xxx
  • model: claude-sonnet-4-5-20250929
  • finish_reason: stop
  • usage: Usage(completion_tokens=103, prompt_tokens=1082, total_tokens=1185, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, text_tokens=103, image_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=0, text_tokens=None, image_tokens=None, cache_creation_tokens=0, cache_creation_token_details=CacheCreationTokenDetails(ephemeral_5m_input_tokens=0, ephemeral_1h_input_tokens=0)), cache_creation_input_tokens=0, cache_read_input_tokens=0, inference_geo='not_available', speed=None)
achat.tc_res
{'toolu_01YJtUjEmCGmyvaUncCtRc5h': {'name': 'Alice', 'age': 30},
 'toolu_01Xqdp7uVZ2UzF1wim7KWhYz': 'Hello Alice, you are 30 years old!'}
list(L(achat.hist).attrgot('tool_calls').filter())
[[ChatCompletionMessageToolCall(index=1, caller={'type': 'direct'}, function=Function(arguments='{}', name='get_person'), id='toolu_01YJtUjEmCGmyvaUncCtRc5h', type='function')],
 [ChatCompletionMessageToolCall(index=1, caller={'type': 'direct'}, function=Function(arguments='{"person": "$`toolu_01YJtUjEmCGmyvaUncCtRc5h`"}', name='greet_person'), id='toolu_01Xqdp7uVZ2UzF1wim7KWhYz', type='function')]]