Inference
AI Gateways
AI Gateway Concepts
AI Gateway concepts
An AI Gateway stores a reusable inference configuration. Pass the gateway’s ID or slug in the request’s …
Models and Gateway Names
Models and gateway names
The model field of /v1/chat/completions selects what runs the request. It accepts three kinds of value:
Value …
Using an AI Gateway
Using an AI Gateway
Create a gateway in the dashboard or with the API, then call it through the OpenAI-compatible chat/completions endpoint …
Using AI Gateways via SDKs
Using AI Gateways via SDKs
The inference endpoint follows the OpenAI chat completions shape, so existing OpenAI-compatible SDKs work by …
Taking an AI Gateway to Production
Taking an AI Gateway to production
Start with a conservative configuration: clear system instructions, a model that supports the required …
Inference
Inference Concepts
Inference concepts
AIVAX exposes an OpenAI-compatible chat/completions API with additional AIVAX parameters. The additions are optional and …
Server-side Context
Server-side context
chat/completions is stateless: each request carries the conversation it needs, and AIVAX does not keep a message history …
Session Reuse
Session reuse
Session reuse lets a stateless chat/completions client keep the server-side context of a conversation between requests. Your …
Inference with AI Gateways
Inference with AI Gateways
An AI Gateway pre-configures an inference. When model is a gateway ID or slug, the request you send is only part …
Processing Pipeline
Processing pipeline
When a request uses an AI Gateway, AIVAX runs a fixed sequence of steps before, during, and after each model generation. …
Input and Multimodality
Input and multimodality
Send the conversation in messages, using the OpenAI roles system, user, assistant, and tool. With an AI Gateway, the …
Caching
Caching
Several caches affect an inference request. Prompt caching lowers the cost of input the provider has already processed. Other caches …
Tools
Tools
Tools let the model request actions and data. AIVAX supports two execution models, and one request can use both:
Client-side tools …
MCP and Protocol Functions in Inference
MCP and protocol functions in inference
MCP tools and protocol functions are server-side tools that call your systems. AIVAX lists or loads …
Structured Output
Structured output
Use structured responses when your application needs JSON output that matches a schema. AIVAX supports them through …
On-demand Built-in Tools
On-demand built-in tools
Use builtin_tools to enable AIVAX built-in tools for a direct request without creating a gateway:
JSON
{ …
Streaming
Streaming
Set stream: true to receive the response as server-sent events (SSE) while it is generated. Each event is a data: line with a JSON …
Custom Provider Request Body
Custom provider request body
Use extra_body to send provider-specific options. When a gateway uses your own provider API key and an …
Multimodal Pre-processing
Multimodal pre-processing
Use multimodal_resolver when the main model should receive a textual description of media instead of the original …
Context Truncation
Context truncation
Context truncation keeps long conversations within a maximum input size. An AI Gateway can limit the input with …
Moderation
Moderation
Moderation enforces safety policy on user requests before the main model answers them. Use it to block unsafe or out-of-policy …
Function Interpreter (Deprecated)
Function interpreter (deprecated)
Deprecated. The function interpreter is kept only for existing gateways. Do not use it in new gateways. …
Inference Limits
Inference limits
This page lists the limits that apply to chat/completions requests. Account-wide request rates and plan limits are …
Troubleshooting Inference
Troubleshooting inference
AIVAX uses two response styles:
OpenAI-compatible endpoints return an OpenAI-style error object. Account and …
Routing
Setup Provider Routing
Setup provider routing
Some integrated models are available through more than one provider. Provider routing lets AIVAX choose among those …
Inline Routing
Inline routing in the model name
An integrated model name accepts a routing suffix after #. Use it when your client can only change the …
Zero Data Retention Providers
Zero data retention providers only
Enable Zero data retention providers only in the dashboard account settings to route text inference with …
Special Providers
Special providers
Some integrated models are also offered by special providers, which serve them at discounted prices under conditions that …
Structured Responses
Structured Responses Concepts
Structured responses concepts
AIVAX can produce structured JSON through two paths:
response_schema: AIVAX validates the final model output …
JSON Healing
JSON Healing
JSON Healing is how AIVAX makes any model return JSON that matches a schema, including models without native structured output. …
Structured Responses with Reasoning and Tools
Reasoning and tools
A structured request can use a reasoning model and tools. The model reasons and calls tools as in any other request, and …
JSON-Only Responses
JSON-only responses
Set json_only: true when the client should receive only the generated JSON:
JSON
{ "model": …
Response rendering
Chat Presentation Guidance
Chat presentation guidance
A streamed assistant turn can contain reasoning, server-side tool activity, partial answers, and the final …
Raw Multi-turn Example
Raw multi-turn example
The example below shows the shape of a streamed response when server-side reasoning is visible to the client, …
Tool Explanations
Tool explanations
Use tool explanations when your client shows users why a server-side tool is running. The setting can also be stored on an …
Chat Clients
Chat Clients Overview
Chat Clients Overview
A chat client publishes an AI Gateway to end users. It gives them a place to talk to the assistant—a hosted web chat, …
Managing Chat Client Sessions
Managing Chat Client Sessions
A session is one conversation between a chat client and an end user. It stores the message history, the …
Web Chat Client
Web Chat Client
The web client is the browser-facing side of a chat client. You can use it in two ways:
Hosted web chat: send the user to …
Chat Client Integrations
Chat Client Integrations
Integrations connect a chat client to messaging apps, so users can talk to the assistant where they already are. …
Subscription Models
Subscription Models Overview
Subscription models
Subscription models are text models included in the Pro, Max, and Ultimate plans. Their names start with …
Harness Integration
Harness integration
Coding agents and other harnesses reach subscription models through the same [OI]-compatible Chat Completions API used …
Inference
Agentic Tests
Agentic Tests
Agentic Tests evaluate how an AI Gateway behaves across a complete, goal-oriented conversation instead of grading one isolated …
Voice and Realtime
Voice and Realtime
Voice Session is AIVAX’s low-latency, stateful voice API. Use it for a live spoken conversation. Your client opens …
Gateway Workers
Gateway Workers
Gateway Workers are HTTP hooks that let your service control an AI Gateway while a request is running. Use one when a rule …
Skills
Skills
Skills (also known as abilities) are instructions that your agent loads on demand to perform specific tasks better. Use them for …
Batch Processing
Batch Processing
Batch runs the same AI workflow over many independent items. It turns a list of inputs into a background queue with fixed …
