APIs

Chat Completions

The classic /v1/chat/completions endpoint — drop-in compatible with the OpenAI SDK.

Basic usage

Send a list of messages and get a single completion back.

example.ts

import { client } from "./client";

const completion = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [
    { role: "system", content: "You are a concise assistant." },
    { role: "user", content: "Summarize what AI Gateway does in one sentence." },
  ],
});

console.log(completion.choices[0].message.content);

Streaming

Set stream: true to receive Server-Sent Events — data: chunks terminated by data: [DONE]. The SDK hands them back as an async iterable of deltas.

example.ts

import { client } from "./client";

const stream = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Stream a short haiku about proxies." }],
  stream: true,
});

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta) process.stdout.write(delta);
}

Streaming is rejected on keys with active PII masking (400) — masking can't be applied mid-stream.

Multi-turn conversation

Append the assistant turn to the message list and send it back to continue the conversation.

example.ts

import { client } from "./client";

const history = [
  { role: "system", content: "You are a concise assistant." },
  { role: "user", content: "What is AI Gateway?" },
];

const first = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: history,
});
history.push(first.choices[0].message);

history.push({ role: "user", content: "And what is the proxy port?" });

const second = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: history,
});

console.log(second.choices[0].message.content);

When the API key has force_user enabled, a request without a user (or the X-User header) is rejected with 400. The identifier also selects which per-user budget or request cap applies. Full behavior in End-user identifier.

Prompt Vault

Reference a stored prompt via the prompt extra-body field instead of hard-coding the system message; messages becomes optional when the prompt provides the conversation.

example.ts

import { client } from "./client";

const completion = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "App keeps crashing on launch" }],
  // Caronte extension — the gateway resolves and strips it
  prompt: { id: "support/triage", version: "production" },
} as never);

tip · Merge semantics

With no client messages the compiled prompt becomes the conversation; with them, only the prompt's system part is prepended as a system message (vault-first) and its user/assistant turns are dropped. Full cross-endpoint rules in Prompt Vault → Merge.

Basic usage

Streaming

Multi-turn conversation

Identifying the end user

Prompt Vault