Inference

Inference

Inference Concepts Inference concepts AIVAX exposes an OpenAI-compatible chat/completions API with additional AIVAX parameters. The additions are optional and … Server-side Context Server-side context chat/completions is stateless: each request carries the conversation it needs, and AIVAX does not keep a message history … Session Reuse Session reuse Session reuse lets a stateless chat/completions client keep the server-side context of a conversation between requests. Your … Inference with AI Gateways Inference with AI Gateways An AI Gateway pre-configures an inference. When model is a gateway ID or slug, the request you send is only part … Processing Pipeline Processing pipeline When a request uses an AI Gateway, AIVAX runs a fixed sequence of steps before, during, and after each model generation. … Input and Multimodality Input and multimodality Send the conversation in messages, using the OpenAI roles system, user, assistant, and tool. With an AI Gateway, the … Caching Caching Several caches affect an inference request. Prompt caching lowers the cost of input the provider has already processed. Other caches … Tools Tools Tools let the model request actions and data. AIVAX supports two execution models, and one request can use both: Client-side tools … MCP and Protocol Functions in Inference MCP and protocol functions in inference MCP tools and protocol functions are server-side tools that call your systems. AIVAX lists or loads … Structured Output Structured output Use structured responses when your application needs JSON output that matches a schema. AIVAX supports them through … On-demand Built-in Tools On-demand built-in tools Use builtin_tools to enable AIVAX built-in tools for a direct request without creating a gateway: JSON { … Streaming Streaming Set stream: true to receive the response as server-sent events (SSE) while it is generated. Each event is a data: line with a JSON … Custom Provider Request Body Custom provider request body Use extra_body to send provider-specific options. When a gateway uses your own provider API key and an … Multimodal Pre-processing Multimodal pre-processing Use multimodal_resolver when the main model should receive a textual description of media instead of the original … Context Truncation Context truncation Context truncation keeps long conversations within a maximum input size. An AI Gateway can limit the input with … Moderation Moderation Moderation enforces safety policy on user requests before the main model answers them. Use it to block unsafe or out-of-policy … Function Interpreter (Deprecated) Function interpreter (deprecated) Deprecated. The function interpreter is kept only for existing gateways. Do not use it in new gateways. … Inference Limits Inference limits This page lists the limits that apply to chat/completions requests. Account-wide request rates and plan limits are … Troubleshooting Inference Troubleshooting inference AIVAX uses two response styles: OpenAI-compatible endpoints return an OpenAI-style error object. Account and …