Documents

A document is the unit that AIVAX indexes and retrieves inside a collection. Each document stores text, an optional reference, optional tags, optional metadata, and the vectors generated by the indexing job. Search quality depends heavily on how clear, focused, and self-contained each document is.

Anatomy of a document #

Only the contents are embedded. Metadata and tags are returned with results and can be used in advanced filters, but they do not influence semantic matching. If users may search for “refund policy”, those words must appear in the contents, not only in a tag.

Use references when several documents represent parts of the same source, such as sections of a contract, clauses of a policy, or chunks of a PDF. When a search enables reference expansion, a matched document can bring the other documents that share its reference.

Write good documents #

A document should represent one limited piece of knowledge. As a practical target, keep most documents between 20 and 700 words. This is not an API rule, but it usually gives the embedding model enough context without mixing unrelated topics. Split longer content at logical boundaries such as sections, clauses, FAQ entries, or product descriptions.

Do:

• Give each document a clear name and focused text.
• Put the subject near the start of the document.
• Use natural language similar to the way users ask questions.
• Repeat important identifiers, product names, policy names, acronyms, and terms when they matter.
• Include enough context for the text to be understood on its own.

Avoid:

• Empty, tiny, or title-only documents.
• Entire PDFs, chapters, logs, or manuals as one document.
• Multiple unrelated subjects in one document.
• Text that depends on surrounding pages to make sense.
• Mixed languages inside the same document, unless users are expected to search that way.
• Raw JSON, code, tables, or logs without a short natural-language explanation.
• Pronouns such as “it” or “the product” when the document does not identify the subject.

For example, prefer:

The color of the 2015 Honda Civic registered in fleet record CAR-123 is yellow.

over:

The car is yellow.

The first version can be retrieved and understood without external context.

Chunking larger sources #

When importing PDFs, spreadsheets, web pages, or manuals, inspect the resulting chunks before relying on the collection. Remove repeated headers, footers, navigation menus, broken tables, boilerplate, and irrelevant disclaimers when possible. A good chunk includes a title or source heading, the immediate section context, and the complete rule, answer, or explanation. A poor chunk contains half of a table row, a sentence that depends on the previous page, or several policies mixed together. Text Segmentation can split source text into cohesive segments before import.

Document size #

Very small documents, below about 10 tokens, are accepted but may not be retrieved reliably. Documents above about 1,562 tokens are accepted and indexed, but they record a warning in the account logs because a single vector represents their whole text less precisely. Text far beyond that size is truncated for embedding. Split large documents instead of relying on truncation.

Import documents #

AIVAX supports several ways to add documents. All of them queue new or changed documents for indexing; a document becomes searchable after its state changes from Queued to Indexed.

Updates and reindexing #

Documents are matched by name. When a document is created, it is queued for indexing. When its contents change, it is queued again and its vectors are regenerated. Unchanged documents are skipped.

The single-document update can change the reference, tags, or metadata without reindexing, and fields omitted from it keep their current values. A batch import updates metadata alone without reindexing, but updates the reference and tags only together with changed contents; a line that changes only the reference or tags is skipped.

Batch import #

Use the Index Documents (JSONL WebSocket) endpoint for batch imports. It replaces the multipart Index Documents (JSONL) upload, which remains available only for existing integrations. The WebSocket import streams documents in blocks, reports progress as it goes, accepts 10× the per-request document limit of the multipart upload, and lets you resume from the last committed line after a disconnection.

Each JSONL line represents one document:

The import works as follows:

1. Open a WebSocket to the endpoint. Browsers can authenticate with the api-key query parameter. Add insert-mode=sync to delete collection documents whose names were not sent.
2. Send UTF-8 text messages with complete JSONL lines separated by \n. A message can contain one or many lines. Each message must arrive within 30 seconds of the previous one.
3. AIVAX commits the lines in blocks. For each block it checks the account balance, storage quota, and collection access, saves the documents, queues them for indexing, and sends a progress message with the number of committed lines.
4. Send a line containing only END. AIVAX saves the remaining lines, applies insert-mode=sync when requested, and sends a completed message before closing the connection.

If the connection closes, times out, or fails before END, uncommitted lines are discarded, while committed blocks remain queued. The error message reports the committed line count, so you can resend only the remaining lines. Only one import runs per collection at a time; a second import waits up to 30 seconds and then fails with 409 Conflict.

When a line changes the text of an existing document, its reference and tags are replaced by the values sent; an omitted __meta keeps the existing metadata.

Import from files with Media Injector #

Media Injector turns a source file into focused, self-contained documents. It examines the source, identifies materially useful knowledge, writes concise factual documents in the source’s predominant language, and queues them for indexing. Use direct import instead when you need exact source wording, deterministic boundaries, stable document names, or application-controlled metadata.

Media Injector is useful for PDFs that contain several independent topics, images or scanned pages whose visible content should become searchable, and audio or video whose facts should be available through RAG. It does not preserve the source as one searchable document. Review the generated documents when wording, coverage, legal fidelity, or sensitive-data handling is important.

In the Console:

1. Open the target collection and choose Import from files.
2. Select one or more source files.
3. Optionally provide processing context. The same context is applied to every selected file.
4. Confirm the import. The files are uploaded sequentially, and a separate job is created for each file after all of its chunks have reached AIVAX.
5. Follow the jobs under Batch > Media Processing.
6. After each job completes, review the generated documents and wait for indexing before testing Search.

A job can be queued, processing, completed, failed, or cancelled. Failed or cancelled jobs can be retried while their uploaded data is still available. If no job appears, the upload did not reach its completion step; retry the upload and check the error shown. File names are preserved during upload, including accented letters and non-Latin scripts.

Use the original, accurate file extension, because it identifies the media type. If an audio or video file fails, convert it to a common format and retry.

Processing context is an optional instruction that helps Media Injector decide which facts are most valuable. It is not treated as a factual source and cannot add facts that are absent from the file. Describe the source’s identity and purpose, the audience, the topics that matter, ambiguous terminology, and content to deprioritize:

This is the 2026 support policy for Acme Cloud customers in Brazil. Prioritize eligibility rules, deadlines, plan differences, exceptions, and the steps a support agent must communicate. Ignore repeated page headers and signature blocks.

Do not ask it to infer conclusions, supply missing information, or calculate values the source does not report.

Generated documents prioritize source identity, scope, principal facts, relationships, and exceptions; combine closely related facts; ignore decorative text and pagination; and preserve the source’s language, terminology, dates, and number formats. They are tagged as automatically produced content and indexed like other documents. Inspect documents generated from tables, scans, and repeated layouts before relying on them in production.

Manage documents #

You can manage documents in the Console, on the collection’s Browse documents tab, or through the API. Management capabilities include:

• Browse and filter by tag, reference, content, name, or ID, by indexing state (queued or indexed), and sorted by creation, update, or indexing date.
• Inspect a document’s contents, reference, tags, metadata, timestamps, and approximate token count. The Console editor also warns when a document has too many or too few words.
• Edit contents, reference, tags, and metadata. Changing the contents reindexes the document.
• Delete individual documents, or remove documents that were not sent in a synchronized batch import with insert-mode=sync.
• Export all documents of a collection as JSONL.
• Remove duplicates with the collection de-duplicator.

See the Documents and Collections groups of the API reference for the endpoints, including Create or Update Document, Browse Documents, Get Document, Delete Document, and Export Collection.

Troubleshoot retrieval #

When search results are poor, check the basics first:

• Confirm the documents are indexed.
• Test the collection with Search or the Console playground before testing through an AI Gateway.
• Try a complete question instead of isolated keywords, and compare the query language with the document language.
• Review whether the answer is split across too many small documents or buried inside a very large one.
• Check search history and query coverage in RAG Observability.

Pricing and limits #

Document indexing is billed as collection text embedding; see Collections. Media Injector is billed for input, cached input, output, and media usage of the processing job, in addition to the normal embedding cost of the generated documents. Rates depend on media type and input-token volume.

An em dash (—) means the plan does not impose a limit. If an import exceeds the per-import limit, split it into multiple imports. If the account reaches the daily insertion limit, wait for the window to reset or upgrade the plan.