> ## Documentation Index
> Fetch the complete documentation index at: https://docs-preview.modular.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Function calling and tool use

> Implement OpenAI-compatible function calling and tool use with the hosted inference API

Function calling is a feature available with some large language models (LLMs)
that allows them to call external program functions (or tools). This allows
the model to interact with external systems to retrieve new data for use as
input or execute other tasks. This is a foundational building block for
agentic AI applications, in which an LLM can chain together various functions
to achieve complex objectives.

Function calling is also called "tool use" because the manner in which you
tell the LLM what functions are available is with a `tools` parameter in the
request body. The inference API is OpenAI-compatible, so you can
use the OpenAI SDK without changes.

<Note>
  Function calling is enabled by default, but its availability
  is model-dependent and will produce valid output only if the model is
  pretrained to return tool-use responses. Function calling is supported on the
  `/v1/chat/completions` endpoint only; it does not apply to `/v1/responses`,
  which is used for image and video generation models. See [Supported
  models](#supported-models) below.
</Note>

## When to use function calling

You should use function calling when you want your LLM to:

* **Fetch data**: Such as fetch weather data, stock prices, or news updates
  from a database. The model will call a function to get information, and then
  incorporate that data into its final response.

* **Perform actions**: Such as modify application states, invoke workflows, or
  call upon other AI systems. The model will call another tool to perform an
  action, effectively handing off the request after it determines what the user
  wants.

## How function calling works

When you send an inference request to a model that supports function calling,
you can specify which functions are available to the model using the `tools`
body parameter.

The `tools` parameter provides information that allows the LLM to understand:

* What each function can do
* How to call each function (the arguments it accepts/requires)

For example, here's a request with the [chat completions
API](/api-reference/inference/create-chat-completion) that declares an
available function named `get_weather()`:

```python theme={null}
import os
from openai import OpenAI

def get_weather(city: str) -> str:
    print("Get weather:", city)

client = OpenAI(
    base_url="https://api.modular.com/v1",
    api_key=os.environ["MODULAR_API_KEY"],
)

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current temperature for a given location.",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "City and country e.g. Bogotá, Colombia"
                }
            },
            "required": [
                "location"
            ],
            "additionalProperties": False
        },
        "strict": True
    }
}]

messages = [
  {
    "role": "user",
    "content": "What's the weather like in San Francisco today?"
  }
]

completion = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=messages,
    tools=tools
)
```

Let's take a closer look at each parameter shown in the `tools` property:

* `type`: Currently this is always `function`
* `function`: Definition of the function
  * `name`: The function name used by the LLM to call it. Must be at most 64
    characters; the default character set is `[a-zA-Z0-9_-]`, though some
    model parsers widen it to allow dotted or namespaced names (for example,
    `my-tool.v2`)
  * `description`: A function description that helps the LLM understand when
    to use it
  * `parameters`: Definition of the function parameters
    * `type`: Defines this as an object containing parameters
    * `properties`: Lists all possible function arguments and their types
    * `required`: Specifies which function arguments are required

This format follows the [OpenAI function calling
specification](https://platform.openai.com/docs/guides/function-calling) to
specify functions as tools that a model can use.

You can also control whether (and how) the model is required to call a tool
with the `tool_choice` parameter:

* `none`: The model won't call any tool.
* `auto` (default): The model decides whether to call a tool or respond with
  a message.
* `required`: The model must call one or more tools.
* A specific-function object: Forces the model to call that named function,
  for example:

  ```json theme={null}
  {"type": "function", "function": {"name": "get_weather"}}
  ```

Using this information, the model will decide whether to call any functions
specified in `tools`. In this case, we expect the model to call
`get_weather()` and incorporate that information into its final response. So,
the initial `completion` response from above includes a `tool_calls`
parameter like this:

```python theme={null}
print(completion.choices[0].message.tool_calls)
```

```js theme={null}
[ChatCompletionMessageToolCall(
  id='call_a175692d9ff54554',
  function=Function(
    arguments='{
      "location": "San Francisco, USA"
    }',
    name='get_weather'
  ),
  type='function'
)]
```

From here, you must parse the `tool_calls` body and execute the function as
appropriate. For example:

```py theme={null}
import json

tool_call = completion.choices[0].message.tool_calls[0]
args = json.loads(tool_call.function.arguments)

result = get_weather(args["location"])
```

If the function is designed to **fetch data** for the model, you should call
the function and then call the model again with the function results appended
as a message using the `tool` role. Append the assistant message that
contained the `tool_calls`, then one message per result with the `tool` role
and the matching `tool_call_id`:

```json theme={null}
"messages": [
  { "role": "user", "content": "What's the weather in San Francisco?" },
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "call_abc123",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"location\": \"San Francisco\"}"
        }
      }
    ]
  },
  {
    "role": "tool",
    "tool_call_id": "call_abc123",
    "content": "{\"temperature\": 62, \"unit\": \"F\"}"
  }
]
```

Each `tool` message's `tool_call_id` must match the `id` of a tool call in the
preceding assistant message. A request with a non-JSON `tool_calls` argument,
an unmatched `tool_call_id`, or only a partial set of tool replies is rejected
with a 400 error.

If the function is designed to **perform an action**, then you don't need to
call the model again.

For detail about how to execute the function and feed the results back to the
model, see the [OpenAI docs about handling function
calls](https://platform.openai.com/docs/guides/function-calling?api-mode=chat\&example=get-weather#handling-function-calls).

The OpenAI function calling spec is compatible with multiple agent frameworks,
such as [AutoGen](https://github.com/microsoft/autogen),
[CrewAI](https://github.com/crewAIInc/crewAI), and more.

## Supported models

Function calling is model-dependent and will produce valid output only if the
model is pretrained to return tool-use responses. Streaming (`stream: true`)
with function calling also varies by model. If you see incomplete or
malformed tool-call output while streaming, set `stream: false` for that
request.

Function calling only applies to text-generation models called through
`/v1/chat/completions`. It isn't applicable to `/v1/responses`, since that
endpoint is used exclusively for image and video generation models, which
don't accept a `tools` parameter.

See the [supported models page](/models) for the current list of
models available on shared and dedicated endpoints, and which ones support
function calling.

## Quickstart

This walks through the same `get_weather` example from above as a single,
runnable script.

If you haven't already, [create an API
key](/administration/api-keys) in the Modular Console and export it:

```bash theme={null}
export MODULAR_API_KEY="your_api_key_here"
```

Then send a request with the `tools` parameter to a model that supports
function calling:

<Tabs>
  <Tab title="Python">
    Install the OpenAI SDK:

    ```bash theme={null}
    pip install openai
    ```

    Create a program to send a request specifying the available `get_weather()`
    function:

    ```python title="function-calling.py" theme={null}
    import os
    import json
    from openai import OpenAI

    def get_weather(city: str) -> str:
        print("Get weather:", city)

    client = OpenAI(
        base_url="https://api.modular.com/v1",
        api_key=os.environ["MODULAR_API_KEY"],
    )

    tools = [{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current temperature for a given location.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City and country e.g. Bogotá, Colombia"
                    }
                },
                "required": [
                    "location"
                ],
                "additionalProperties": False
            },
            "strict": True
        }
    }]

    messages = [
      {
        "role": "user",
        "content": "What's the weather like in San Francisco today?"
      }
    ]

    completion = client.chat.completions.create(
        model="google/gemma-4-31B-it",
        messages=messages,
        tools=tools
    )

    tool_call = completion.choices[0].message.tool_calls[0]
    args = json.loads(tool_call.function.arguments)

    result = get_weather(args["location"])
    ```

    Run it and the `get_weather()` function should print the argument received:

    ```sh theme={null}
    python function-calling.py
    ```

    ```output theme={null}
    Get weather: San Francisco, USA
    ```
  </Tab>

  <Tab title="curl">
    Use the following `curl` command to send a request specifying the available
    `get_weather()` function:

    ```bash theme={null}
    curl -X POST https://api.modular.com/v1/chat/completions \
        -H "Authorization: Bearer $MODULAR_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
            "model": "google/gemma-4-31B-it",
            "messages": [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": "What is the weather like in Boston today?"}
            ],
            "tools": [
                {
                    "type": "function",
                    "function": {
                        "name": "get_weather",
                        "description": "Get the current weather in a given location",
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "location": {
                                    "type": "string",
                                    "description": "The city and state, e.g. Los Angeles, CA"
                                }
                            },
                            "required": ["location"]
                        }
                    }
                }
            ],
            "tool_choice": "auto"
        }'
    ```

    You should receive a response similar to this:

    ```json theme={null}
    "tool_calls": [
      {
        "id": "call_ac73df14fe184349",
        "type": "function",
        "function": {
            "name": "get_weather",
            "arguments": "{\"location\": \"Boston, MA\"}"
        }
      }
    ]
    ```
  </Tab>
</Tabs>

For a more complete walkthrough of how to handle a `tool_calls` response and
send the function results back to the LLM as input, see the [OpenAI docs about
handling function
calls](https://platform.openai.com/docs/guides/function-calling?api-mode=chat\&example=get-weather#handling-function-calls).