Text generation

Generate text from a prompt or conversation with /v1/chat/completions, or analyze images and video by passing them alongside text in the same request. The endpoint is fully compatible with the OpenAI Chat Completions API, so existing OpenAI SDK code works unchanged. Check the supported models page to see which models are available via shared or dedicated endpoints, and which support text generation versus image/video analysis.

Already using the OpenAI SDK? Just change base_url to https://api.modular.com/v1 and set api_key to your API key. No other code changes required.

The `v1/chat/completions` endpoint

The v1/chat/completions endpoint supports both single-turn and multi-turn interactions. You provide a sequence of structured messages with roles (system, user, assistant), and the model generates a response. For example, within the v1/chat/completions request body, the "messages" array might look similar to the following:

"messages": [
  {
    "role": "system",
    "content": "You are a helpful assistant."
  },
  {
    "role": "user",
    "content": "Who won the world series in 2020?"
  }
]

Use a combination of roles to give the model the context it needs. A system message can define overall model response behavior, user messages represent instructions or prompts from the end-user, and assistant messages incorporate past model responses into the context. You can also include image or video inputs alongside text to work with multimodal models.

Generate text

Generate text from a prompt or conversation history using v1/chat/completions:

Python
curl

Send a chat completion request using the OpenAI Python SDK:

generate-text.py

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.modular.com/v1",
    api_key=os.environ["MODULAR_API_KEY"],
)

response = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"},
        {"role": "assistant", "content": "The LA Dodgers won in 2020."},
        {"role": "user", "content": "Where was it played?"}
    ]
)
print(response.choices[0].message.content)

The 2020 World Series was played at Globe Life Field in Arlington, Texas. It was a neutral site due to the COVID-19 pandemic.

Send the same request using curl:

curl -X POST https://api.modular.com/v1/chat/completions \
    -H "Authorization: Bearer $MODULAR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "google/gemma-4-31B-it",
        "messages": [
            {
            "role": "system",
            "content": "You are a helpful assistant."
            },
            {
            "role": "user",
            "content": "Hello, how are you?"
            }
        ],
        "max_tokens": 100
    }'

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "I'm doing well, thank you for asking. How can I assist you today?",
        "role": "assistant"
      }
    }
  ],
  "model": "google/gemma-4-31B-it",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 17,
    "prompt_tokens": null,
    "total_tokens": 17
  }
}

Stream responses in real time

Set stream=True to receive tokens as they are generated instead of waiting for the full response:

Python
curl

Set stream=True and iterate over the response chunks as they arrive:

stream-text.py

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.modular.com/v1",
    api_key=os.environ["MODULAR_API_KEY"],
)

stream = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[{"role": "user", "content": "Write a short poem about the sea."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

Pass "stream": true in the request body to receive server-sent events:

curl -X POST https://api.modular.com/v1/chat/completions \
    -H "Authorization: Bearer $MODULAR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "google/gemma-4-31B-it",
        "messages": [{"role": "user", "content": "Write a short poem about the sea."}],
        "stream": true
    }'

Each streamed chunk is a server-sent event with a data: prefix:

data: {"choices":[{"delta":{"content":"The"},"index":0}],"object":"chat.completion.chunk"}
data: {"choices":[{"delta":{"content":" sea"},"index":0}],"object":"chat.completion.chunk"}
...
data: [DONE]

Analyze images

Send images to the model alongside text prompts to get descriptions, answers to questions, or extracted information from visual content.

Pass image or video URLs

Within the v1/chat/completions request body, the "messages" array accepts inline image or video URLs.

Image input
Video input

Use image_url to pass an image:

"messages": [
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "What is in this image?"
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "https://example.com/path/to/image.jpg"
        }
      }
    ]
  }
]

Use video_url to pass a video:

"messages": [
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "What is happening in this video?"
      },
      {
        "type": "video_url",
        "video_url": {
          "url": "https://example.com/path/to/video.mp4"
        }
      }
    ]
  }
]

Both image_url and video_url also accept base64-encoded data URIs (such as data:image/jpeg;base64,... or data:video/mp4;base64,...).

Describe an image

Send an image URL alongside a text question to get a description or analysis of the image:

Python
curl

Include an image_url content block alongside your text prompt:

generate-image-description.py

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.modular.com/v1",
    api_key=os.environ["MODULAR_API_KEY"],
)

response = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
                    }
                }
            ]
        }
    ],
    max_tokens=300
)

print(response.choices[0].message.content)

Here's a breakdown of what's in the image:

*   **Peter Rabbit:** The main focus is a realistic-looking depiction of Peter
Rabbit, the character from Beatrix Potter's stories...

Pass the image URL and text prompt in the content array of the request body:

curl -X POST https://api.modular.com/v1/chat/completions \
-H "Authorization: Bearer $MODULAR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
  "model": "google/gemma-4-31B-it",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
          }
        }
      ]
    }
  ],
  "max_tokens": 300
}' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '\n' | sed 's/\\n/\n/g'

Here's a breakdown of what's in the image:

*   **Peter Rabbit:** The main focus is a realistic, anthropomorphic
(human-like) rabbit character...

Describe video content

Send a video URL alongside a text question to get a description or analysis of the video’s visual content.

Describe a video

Include a video_url content block in your request to analyze what happens in a video:

Python
curl

Pass a video URL and a text question to get a natural-language description of the video:

generate-video-description.py

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.modular.com/v1",
    api_key=os.environ["MODULAR_API_KEY"],
)

completion = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe what is happening in this video"
                },
                {
                    "type": "video_url",
                    "video_url": {
                        "url": "https://avtshare01.rz.tu-ilmenau.de/avt-vqdb-uhd-1/test_1/segments/bigbuck_bunny_8bit_15000kbps_1080p_60.0fps_h264.mp4"
                    }
                }
            ]
        }
    ],
    max_tokens=300
)

print(completion.choices[0].message.content)

The video is an animated short film featuring a large, fluffy rabbit in a
colorful meadow. The rabbit wanders through the environment, encountering
butterflies and small birds. The animation has a warm, lighthearted tone with
vibrant natural scenery...

Pass the video URL and text prompt in the content array of the request body:

curl -X POST https://api.modular.com/v1/chat/completions \
-H "Authorization: Bearer $MODULAR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
  "model": "google/gemma-4-31B-it",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Describe what is happening in this video"
        },
        {
          "type": "video_url",
          "video_url": {
            "url": "https://avtshare01.rz.tu-ilmenau.de/avt-vqdb-uhd-1/test_1/segments/bigbuck_bunny_8bit_15000kbps_1080p_60.0fps_h264.mp4"
          }
        }
      ]
    }
  ],
  "max_tokens": 300
}' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '\n' | sed 's/\\n/\n/g'

The video is an animated short film featuring a large, fluffy rabbit in a
colorful meadow. The rabbit wanders through the environment, encountering
butterflies and small birds. The animation has a warm, lighthearted tone with
vibrant natural scenery...

Inference

Deployments

Administration

The `v1/chat/completions` endpoint

Generate text

Stream responses in real time

Analyze images

Pass image or video URLs

Describe an image

Describe video content

Describe a video

​The v1/chat/completions endpoint

​Generate text

​Stream responses in real time

​Analyze images

​Pass image or video URLs

​Describe an image

​Describe video content

​Describe a video

The `v1/chat/completions` endpoint

Generate text

Stream responses in real time

Analyze images

Pass image or video URLs

Describe an image

Describe video content

Describe a video