api-inference

Chat Completion

Generate a response given a list of messages in a conversational context, supporting both conversational Language Models (LLMs) and conversational Vision-Language Models (VLMs). This is a subtask of text-generation and image-text-to-text.

Recommended models

Conversational Large Language Models (LLMs)

• google/gemma-2-2b-it: A text-generation model trained to follow instructions.
• meta-llama/Meta-Llama-3.1-8B-Instruct: Very powerful text generation model trained to follow instructions.
• microsoft/Phi-3-mini-4k-instruct: Small yet powerful text generation model.
• HuggingFaceH4/starchat2-15b-v0.1: Strong coding assistant model.
• mistralai/Mistral-Nemo-Instruct-2407: Very strong open-source large language model.

Conversational Vision-Language Models (VLMs)

• meta-llama/Llama-3.2-11B-Vision-Instruct: Powerful vision language model with great visual understanding and reasoning capabilities.
• microsoft/Phi-3.5-vision-instruct: Strong image-text-to-text model.

API Playground

For Chat Completion models, we provide an interactive UI Playground for easier testing:

• Quickly iterate on your prompts from the UI.
• Set and override system, assistant and user messages.
• Browse and select models currently available on the Inference API.
• Compare the output of two models side-by-side.
• Adjust requests parameters from the UI.
• Easily switch between UI view and code snippets.

Access the Inference UI Playground and start exploring: https://huggingface.co/playground

Using the API

The API supports:

• Using the chat completion API compatible with the OpenAI SDK.
• Using grammars, constraints, and tools.
• Streaming the output

Code snippet example for conversational LLMs

from huggingface_hub import InferenceClient

client = InferenceClient(api_key="hf_***")

for message in client.chat_completion(
	model="google/gemma-2-2b-it",
	messages=[{"role": "user", "content": "What is the capital of France?"}],
	max_tokens=500,
	stream=True,
):
    print(message.choices[0].delta.content, end="")

Code snippet example for conversational VLMs

from huggingface_hub import InferenceClient

client = InferenceClient(api_key="hf_***")

image_url = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"

for message in client.chat_completion(
	model="meta-llama/Llama-3.2-11B-Vision-Instruct",
	messages=[
		{
			"role": "user",
			"content": [
				{"type": "image_url", "image_url": {"url": image_url}},
				{"type": "text", "text": "Describe this image in one sentence."},
			],
		}
	],
	max_tokens=500,
	stream=True,
):
	print(message.choices[0].delta.content, end="")

API specification

Request

Some options can be configured by passing headers to the Inference API. Here are the available headers:

For more information about Inference API headers, check out the parameters guide.

Response

Output type depends on the stream input parameter. If stream is false (default), the response will be a JSON object with the following fields:

If stream is true, generated tokens are returned as a stream, using Server-Sent Events (SSE). For more information about streaming, check out this guide.