Getting started
llms.md
A single Markdown file that consolidates all the pages in these docs. Copy it into an LLM/agent context (or download it and attach it as a file) so the assistant knows how to call AI Gateway without you having to paste every page individually. The same content is served raw at /llms.md.
Grab the file
Copy to your clipboard or download as llms.md. The raw URL is plain text/markdown, so curl works too.
1,869 lines · 86,209 characters
Preview
Full content rendered as plain text — exactly what gets copied or downloaded.
# Caronte — LLM-friendly documentation
Caronte is an OpenAI-compatible LLM gateway. Point any OpenAI SDK at the Caronte base URL with a Caronte API key (`sk_...`), and you get OpenAI's `/v1/chat/completions`, `/v1/responses`, and related endpoints — plus budgets, request caps, PII masking, tracing, hosted server tools (Code Interpreter), and the Prompt Vault.
This file consolidates every page of the public Caronte developer documentation in plain Markdown so it can be pasted into an LLM/agent context. The canonical HTML version lives under `/docs`.
## Prerequisites
- A Caronte API key (prefix `sk_`, only shown once at creation time).
- The base URL of your Caronte proxy, ending with `/v1` (e.g. `http://localhost:5002/v1` in local dev, or `https://<your-caronte>/v1` in production).
- Each key belongs to a project and carries a list of allowed models.
Quick sanity check:
```bash
curl https://<your-caronte>/v1/chat/completions \
-H "Authorization: Bearer $CARONTE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o-mini",
"messages": [{ "role": "user", "content": "ping" }]
}'
```
---
## Client configuration
Override two settings on the official OpenAI SDK: `baseURL` and `apiKey`. Every other example in this file reuses the client built here.
### Synchronous client
```typescript
// client.ts
import OpenAI from "openai";
export const client = new OpenAI({
apiKey: process.env.CARONTE_API_KEY!,
baseURL: process.env.CARONTE_BASE_URL ?? "https://<your-caronte>/v1",
});
```
```python
# client.py
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["CARONTE_API_KEY"],
base_url=os.getenv("CARONTE_BASE_URL", "https://<your-caronte>/v1"),
)
```
### Async client
Same two settings, async variant — useful for agents, web servers, or anything streaming.
```typescript
// client.ts
import OpenAI from "openai";
// The OpenAI JS SDK is async by default; the same instance works for
// both blocking and streaming calls.
export const client = new OpenAI({
apiKey: process.env.CARONTE_API_KEY!,
baseURL: process.env.CARONTE_BASE_URL ?? "https://<your-caronte>/v1",
});
```
```python
# client.py
import os
from openai import AsyncOpenAI
client = AsyncOpenAI(
api_key=os.environ["CARONTE_API_KEY"],
base_url=os.getenv("CARONTE_BASE_URL", "https://<your-caronte>/v1"),
)
```
### Environment variables
If you'd rather not pass the settings explicitly, the OpenAI SDK also reads `OPENAI_API_KEY` and `OPENAI_BASE_URL` from the environment, so you can drive existing code through Caronte by just exporting them.
```bash
OPENAI_API_KEY=sk_xxxxxxxxxxxx
OPENAI_BASE_URL=https://<your-caronte>/v1
```
### Custom headers
Caronte reads a few of its own request headers on top of the standard OpenAI ones. Set them once on the client via `defaultHeaders` (every call) or per request via the SDK's request options. The most common one is `X-No-Log`, which keeps the request/response bodies out of storage (see [Monitoring & traces](#monitoring--traces)).
```typescript
import OpenAI from "openai";
// Applied to every request from this client.
export const client = new OpenAI({
apiKey: process.env.CARONTE_API_KEY!,
baseURL: process.env.CARONTE_BASE_URL ?? "https://<your-caronte>/v1",
defaultHeaders: { "X-No-Log": "1" },
});
// …or just for a single call (overrides the client default):
await client.chat.completions.create(
{ model: "gpt-4o-mini", messages: [{ role: "user", content: "Hi" }] },
{ headers: { "X-No-Log": "1" } },
);
```
```python
import os
from openai import OpenAI
# Applied to every request from this client.
client = OpenAI(
api_key=os.environ["CARONTE_API_KEY"],
base_url=os.getenv("CARONTE_BASE_URL", "https://<your-caronte>/v1"),
default_headers={"X-No-Log": "1"},
)
# …or just for a single call (overrides the client default):
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hi"}],
extra_headers={"X-No-Log": "1"},
)
```
#### Trace metadata headers
The trace fields you'd normally put in the body `metadata` object (see [Monitoring & traces](#monitoring--traces)) can also be sent as request headers. This is the recommended way to attach them on the multipart endpoints (`/v1/audio/transcriptions`, `/v1/images/edits`), where nesting a `metadata` object in the form is awkward, but the headers work on **every** endpoint. When both are present, the **header wins** over the body `metadata` field.
| Header | Body `metadata` equivalent |
| --- | --- |
| `X-Trace-Id` | `trace_id` |
| `X-Trace-Name` | `trace_name` |
| `X-Generation-Name` | `generation_name` |
| `X-Session-Id` | `session_id` |
| `X-Session-Name` | `session_name` |
```typescript
// Works the same on chat, responses, embeddings, images and audio.
await client.audio.transcriptions.create(
{ model: "whisper-1", file: fs.createReadStream("call.mp3") },
{ headers: { "X-Trace-Id": traceId, "X-Generation-Name": "call-transcript" } },
);
```
#### End-user identifier header
`X-User` sets the request's **end-user identifier** — the header-based alternative to the body `user` field (OpenAI endpoints) / `metadata.user_id` (Messages API). Same rules as the trace headers: it wins over the body and works on every endpoint, including the multipart ones. It drives `force_user` enforcement and per-user budgets / request caps — see [End-user identifier](#end-user-identifier).
```typescript
await client.chat.completions.create(
{ model: "gpt-4o-mini", messages: [{ role: "user", content: "Hi" }] },
{ headers: { "X-User": "user_42" } },
);
```
### Request size limits
Each endpoint caps the total request body. The JSON APIs allow larger bodies so inline base64 images fit; the multipart upload APIs cap higher for file payloads. A body above the cap is rejected with `413` / `invalid_request_error` (code `request_too_large`) before it reaches the provider.
| Endpoint | Max request body | Notes |
| --- | --- | --- |
| `/v1/chat/completions` | 32 MB | Inline base64 images count toward this total. |
| `/v1/responses` | 32 MB | Inline base64 images count toward this total. |
| `/v1/embeddings` | 32 MB | |
| `/v1/code-execution/runs` | 32 MB | Submitted code plus any inline tool results. |
| `/v1/images/generations`, `/v1/images/edits` | 50 MB | Multipart uploads (source image + optional mask). |
| `/v1/audio/transcriptions` | 50 MB | A single audio file must not exceed 25 MB. |
| `/v1/files` | 50 MB | Per uploaded file. |
These are Caronte's ceilings; the upstream provider may enforce stricter per-item limits (e.g. some cap a single image at ~5 MB). For inputs larger than the request cap, upload via `/v1/files` instead of inlining base64, or split the input.
---
## MCP server
Beyond the LLM proxy, Caronte exposes its **management tools** over the [Model Context Protocol](https://modelcontextprotocol.io): connect an MCP client (Claude Code, Claude, GitHub Copilot, Cursor, …) and it can inspect models, projects, teams, API keys, budgets and PII configs, and work with the Prompt Vault — acting as your user, with your permissions. Requires the `mcp` feature on your organization.
- **Endpoint:** `https://<your-org-subdomain>/mcp` — your organization's dashboard subdomain (NOT the `api.` proxy host used for `/v1/*`), path `/mcp`, Streamable HTTP.
- **Auth is OAuth 2.1, not an API key.** On first connect the client opens the dashboard sign-in + consent in your browser and registers itself automatically (Dynamic Client Registration). Tokens expire after ~1h and refresh over OAuth; disconnect the client to revoke.
- **Tools mirror your permissions**, exactly like the in-app assistant: a member sees read/list tools (scoped to the projects they are on) plus prompt authoring; create/update/delete tools for projects, teams, API keys, budgets and PII configs appear only for roles that hold those permissions.
Claude Code:
```bash
claude mcp add --transport http ai-gateway https://<your-org-subdomain>/mcp
# then run /mcp inside Claude Code and pick "ai-gateway" to finish the OAuth sign-in
```
Most other clients (GitHub Copilot in VS Code, Cursor, Windsurf, Zed, …) accept a remote server URL directly:
```json
{
"mcpServers": {
"ai-gateway": { "url": "https://<your-org-subdomain>/mcp" }
}
}
```
For clients that only support local `stdio` servers, bridge with `mcp-remote`: `npx -y mcp-remote https://<your-org-subdomain>/mcp`.
To verify, ask the agent to run `list_models` or "list my projects" — the tools it sees are exactly what your role allows. Common errors: `401` (token expired — re-authenticate from the client), `403 MCP not enabled` (feature off for the org), or sign-in loops (sign in on *this organization's* subdomain — sessions and tokens are tenant-bound).
This is the management/tools surface; for the LLM API (chat completions, responses, function calling) start from [Client configuration](#client-configuration).
---
## MCP Gateway
The mirror image of the section above. There, Caronte **is** the MCP server and exposes its own management tools. Here Caronte is an MCP **proxy**: your organization registers third-party MCP servers (GitHub, Jira, an internal service) once, and your agents reach them through Caronte — no per-agent credentials for the upstream, and every call authorized and logged. Requires the `mcp_gateway` feature on your organization.
Two credentials, two endpoints for the same server:
- **With an API key:** `https://<your-caronte>/mcp/<slug>` and `Authorization: Bearer sk_…` — the same proxy host as `/v1/*`. The `<slug>` is shown next to each server in the dashboard under **MCP Gateway**.
- **With your own account (OAuth):** `https://<your-org>.<your-domain>/mcp/<slug>` — your dashboard's host, and no header at all. Your client opens a browser once, you sign in and approve, and it holds a token from then on. The host matters: an MCP client discovers where to sign in by asking the host it just called, and only your organization's host can answer for it.
- **The org comes from the credential** either way, so `(slug, caller)` resolves to exactly one upstream server — a tool name can never be ambiguous.
- **Transport:** Streamable HTTP over `POST`. Stateless — no session id is issued, so any replica can serve any request. `GET` (the server-initiated SSE half) is refused with `405`: the gateway never pushes.
- **You only see what you were granted.** `tools/list` returns the tools granted to *you* — bound to your key, or granted to you and your teams. A tool that exists upstream but was not granted is indistinguishable from one that does not exist. A grant on you personally **replaces** what your teams grant on that server, so it can be narrower than theirs.
```bash
curl https://<your-caronte>/mcp/github \
-H "Authorization: Bearer $CARONTE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
```
Claude Code, and most clients that accept a remote server URL with a header:
```bash
claude mcp add --transport http github https://<your-caronte>/mcp/github \
--header "Authorization: Bearer $CARONTE_API_KEY"
```
Or with your own account — no key to distribute, and the calls are logged under your name:
```bash
claude mcp add --transport http github https://<your-org>.<your-domain>/mcp/github
```
### What the gateway does to a call
Every `tools/call` is admitted before it is forwarded, and the two refusal shapes are different on purpose:
- A tool your key was **not granted** comes back as a JSON-RPC **error**: the call never happened, so there is no tool run to report on.
- A **guardrail block** comes back as a normal tool result with `isError: true` and a readable reason, so the agent can adapt — pick another tool, drop a field — instead of failing the turn.
What the gateway decides is **which** tools you may call — the grants an administrator made, to your key or to you and your teams. It does not validate your arguments against the tool's schema: that contract belongs to the upstream server, which enforces it itself. `tools/list` relays each tool's `inputSchema` exactly as the upstream advertises it, so build your calls against that.
**Guardrails** assigned to the **server** apply here — on the gateway a rule is attached to the upstream server, so it holds for every caller that reaches it, key or person. They are evaluated twice per call — on the tool arguments before they are forwarded, and on the result that comes back — and a block comes back as a tool result with a generic reason (`This tool call was blocked by your organization's guardrail policy.`), never naming the rule. A block on the RESULT means the tool did run upstream and its answer was withheld from you: treat it as "the side effect may have happened", not as "nothing happened". Rules assigned only to API keys govern LLM calls and never MCP ones; the two assignments are independent, and a rule can carry both.
**PII masking** applies here too, configured on the **server** rather than on your credential: if the server has [PII masking](#pii-masking) configured, the **tool arguments** are redacted before they reach it and the original values are restored in the result you receive — for every caller, since the policy is a statement about what that recipient may be sent. (Your key's own masking config governs its `/v1/*` calls.) The same `X-Trace-Id` keys both surfaces, so a value the model saw as `<EMAIL_1>` is that same placeholder when it reaches a tool. Only the arguments are masked (never the tool name), and if the masker is unavailable the call is **refused** rather than forwarded in the clear.
> ⚠️ Masking is **lossy for tools that look a value up**. `crm_lookup(email: "<EMAIL_1>")` finds nothing, and restoring the placeholder afterwards cannot repair a wrong answer — the tool just reads as "no results". If your agents depend on lookup tools, do not enable PII masking on the server that hosts them.
### Correlating tool calls with your LLM traffic
Send the **same `X-Trace-Id` on both** your `/v1/*` call and your `/mcp/<slug>` calls, and the dashboard shows the model turn and the external actions it led to in one place — the trace page grows an "MCP tool calls" section, and the MCP Gateway → Calls tab links each row back to its trace.
```bash
TRACE=$(uuidgen)
curl https://<your-caronte>/v1/chat/completions \
-H "Authorization: Bearer $CARONTE_API_KEY" \
-H "X-Trace-Id: $TRACE" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"open an issue"}]}'
curl https://<your-caronte>/mcp/github \
-H "Authorization: Bearer $CARONTE_API_KEY" \
-H "X-Trace-Id: $TRACE" \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"create_issue","arguments":{}}}'
```
This is **opt-in and cannot be inferred**: the two are separate HTTP requests and the MCP transport is stateless, so without the header the gateway has no way to know which turn a tool call belongs to. Calls without it are still fully logged — they just show no trace.
### What is recorded
Every `tools/call` writes a row: tool, outcome, per-phase timings, and the reason when it was refused. The **arguments and the result** are stored too, as a payload you can open from the call's drill-down — refusals included, so you can see what the agent got back instead. Two things worth knowing: the stored result is the one you actually received (a very large result is capped on the way through, and the text says so where it was cut), and `X-No-Log: true` skips the payload while still recording the call itself. Payloads are served for 30 days; older ones can be retrieved on explicit request.
Common errors: `404` with `-32601` (unknown slug, server disabled, key not bound, or the org lacks the feature — all answer identically, so a probing client learns nothing), `401` (bad or revoked API key), `405` on `GET`.
---
## Developer tools (Claude Code)
Route Claude Code through Caronte so your coding-agent sessions are recorded alongside the rest of your organization's traffic. In this mode **your own claude.ai login authorizes each call, not a Caronte API key**: Caronte relays the request untouched and stores only the trace. Requires the `developer_tools` feature on your organization, switched on by an administrator.
- **A developer token identifies, it does not authorize.** Mint one in the dashboard under *Developer tools* (prefix `cdt_`, shown exactly once — only a SHA256 is stored). On its own it grants access to no model, key or endpoint; it answers one question, "whose session is this?".
- **The base URL is the Anthropic-compatible entry point**, `https://<your-caronte>/anthropic` — not the `/v1` OpenAI-compatible path used everywhere else here.
- **The same base URL serves two modes and `X-Caronte-Developer-Token` selects between them.** With the header you get this pass-through. Without it, the very same URL is the [Messages API](#messages-api-anthropic-compatible) authorized by a Caronte `sk_` key, with the full gateway behind it — routing, prompt compilation, PII masking, guardrails, budgets and caps, and cost attributed to the organization. They are alternatives, not a combination.
- **Never send a Caronte `sk_` key in `Authorization` on this path.** In pass-through mode that header is forwarded to Anthropic **untouched** — Caronte checks only that it is present and bearer-shaped, then lets the provider be the authority. A Caronte key placed there is transmitted to a third party and rejected. The only header stripped before forwarding is `X-Caronte-Developer-Token` itself.
- **`ANTHROPIC_API_KEY` and `ANTHROPIC_AUTH_TOKEN` must stay unset.** Defining either makes Claude Code send that credential instead of your claude.ai login, and the gateway has no way to make that work — the call fails rather than falling back.
Durable setup — Claude Code applies `env` to every session it starts, so it survives new terminals and applies across projects:
```json
// ~/.claude/settings.json
{
"env": {
"ANTHROPIC_BASE_URL": "https://<your-caronte>/anthropic",
"ANTHROPIC_CUSTOM_HEADERS": "X-Caronte-Developer-Token: cdt_your_token_here"
}
}
```
`_CLAUDE_CODE_ASSUME_FIRST_PARTY_BASE_URL` is the exception: it only takes effect as a global environment variable, so it goes in your shell profile (`~/.zshrc`, `~/.bashrc`) rather than in `settings.json`.
```bash
export _CLAUDE_CODE_ASSUME_FIRST_PARTY_BASE_URL="1"
```
Or, to try the whole configuration in a single shell first:
```bash
export ANTHROPIC_BASE_URL="https://<your-caronte>/anthropic"
export ANTHROPIC_CUSTOM_HEADERS="X-Caronte-Developer-Token: cdt_your_token_here"
export _CLAUDE_CODE_ASSUME_FIRST_PARTY_BASE_URL="1"
```
Prefer the file in your home directory over a project's `.claude/settings.json`, which is normally committed: the token identifies you personally, so anyone running a copy of it would file sessions under your name. If you need it per-project, `.claude/settings.local.json` is gitignored.
### Why `_CLAUDE_CODE_ASSUME_FIRST_PARTY_BASE_URL` is in that snippet
Claude Code gates two of its own capabilities on the base URL being literally `api.anthropic.com`: behind any other host it cannot tell which backend will serve the request, and a transparent pass-through is indistinguishable from a rewriting one at that point. It therefore falls back to conservative defaults — the context window drops to **200k** even on models whose 1M window is native, and MCP tool schemas stop being fetched on demand and are written into the system prompt instead. Neither degradation raises an error, which is why the variable ships in the configuration rather than being left for you to find.
Confirm with `/context`: the header reads `1m` rather than `200k`, and the MCP tools line is labelled `(loaded on-demand)` with the schemas costing zero tokens. On a session carrying 144 MCP tools the prompt-and-tools payload at session start fell from roughly 91k tokens to 52k, against a window five times larger.
- **It must be a global environment variable.** Setting it in `settings.json` has no effect; export it from your shell profile so every session inherits it.
- **Undocumented, and it may stop working.** The leading underscore marks it internal to Claude Code; it is absent from the public settings reference. If a release stops honouring it nothing breaks — an env var the agent no longer reads is ignored, and you are back to the conservative defaults.
- **It changes capability detection only.** Authorization, metering and rate limits are unaffected — the call still reaches Anthropic with your own credential and still counts against your own subscription. The variable only tells the client to trust that the endpoint is first-party, which for this pass-through it is.
- **Nothing to change on the gateway.** The decision is made in the client before the request is sent, so the capability values are already absent when Caronte sees the request; the gateway forwards `anthropic-beta` verbatim and has nothing to restore.
### Reading the sessions
Run `claude` as usual. Sessions appear under *Developer tools* in the dashboard, one row per invocation rather than per turn — a Claude Code session is hundreds of turns — aggregating turn count, wall-clock duration, models, token totals and failed turns, and opening into the same trace timeline as the monitoring pages. **Last session** on the token list is the quickest confirmation that the configuration took.
No cost is reported: the calls are billed to your personal subscription, so every cost column on these traces is a structural zero and none is shown.
Retention is capped at **30 days**, rows and payloads alike, and is not org-configurable — these are development diagnostics, not billing or audit records. An organization with a shorter retention policy removes them sooner.
### Token lifecycle
| Action | Effect |
| --- | --- |
| Regenerate | Fresh secret on the same row; the name and every recorded session are kept. The old secret dies immediately, so that machine gets `401`s until it is reconfigured. Own tokens only — no admin override, because the plaintext would let someone else file sessions under your name. |
| Revoke | Stops working at the gateway. Traces already recorded are kept and the row stays visible, so the history remains auditable. Administrators can revoke anyone's. |
| Delete | The row disappears entirely (the audit log keeps the record). Recorded sessions are NOT removed — they stay, attributed to you, no longer linked to any token. |
Common failures: `401` after a working setup usually means the token was revoked or regenerated, or an administrator switched the capability off — the org toggle is a kill switch that rejects already-minted tokens. Calls that authorize as something unexpected mean an `ANTHROPIC_API_KEY` is still set in the environment.
Only Claude Code is supported today. Codex appears in the tool picker but is not wired up: the identity mechanism carries over, its own authentication against a third-party base URL does not.
---
## Chat Completions
The classic `/v1/chat/completions` endpoint — drop-in compatible with the OpenAI SDK. It accepts the standard OpenAI `user` field (or the `X-User` header) to attribute the generation and scope per-user budgets — see [End-user identifier](#end-user-identifier).
### Basic usage
```typescript
import { client } from "./client";
const completion = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "system", content: "You are a concise assistant." },
{ role: "user", content: "Summarize what Caronte does in one sentence." },
],
});
console.log(completion.choices[0].message.content);
```
```python
from client import client
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "Summarize what Caronte does in one sentence."},
],
)
print(completion.choices[0].message.content)
```
### Multi-turn conversation
Append the assistant turn to the message list and send it back to continue the conversation.
```typescript
import { client } from "./client";
const history = [
{ role: "system", content: "You are a concise assistant." },
{ role: "user", content: "What is Caronte?" },
];
const first = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: history,
});
history.push(first.choices[0].message);
history.push({ role: "user", content: "And what is the proxy port?" });
const second = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: history,
});
console.log(second.choices[0].message.content);
```
```python
from client import client
history = [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is Caronte?"},
]
first = client.chat.completions.create(model="gpt-4o-mini", messages=history)
history.append(first.choices[0].message.model_dump())
history.append({"role": "user", "content": "And what is the proxy port?"})
second = client.chat.completions.create(model="gpt-4o-mini", messages=history)
print(second.choices[0].message.content)
```
---
## Responses API
The newer `/v1/responses` endpoint is fully supported. It is the only endpoint that accepts Caronte-specific features: hosted Code Interpreter and Prompt Vault references. Like Chat Completions it accepts the standard `user` field (or the `X-User` header) — see [End-user identifier](#end-user-identifier).
### Basic usage
Pass `instructions` (system message) and `input` (user turn). Read `output_text` for the answer.
```typescript
import { client } from "./client";
const response = await client.responses.create({
model: "gpt-4o-mini",
instructions: "You are a concise assistant.",
input: "Summarize what Caronte does in one sentence.",
});
console.log(response.output_text);
```
```python
from client import client
response = client.responses.create(
model="gpt-4o-mini",
instructions="You are a concise assistant.",
input="Summarize what Caronte does in one sentence.",
)
print(response.output_text)
```
### Structured input
`input` also accepts a list of items — messages, function_call_outputs, file inputs, etc.
```typescript
import { client } from "./client";
const response = await client.responses.create({
model: "gpt-4o-mini",
input: [
{ role: "system", content: "You answer in JSON." },
{
role: "user",
content: [
{ type: "input_text", text: "Extract name and city from: 'Anna, Milan'" },
],
},
],
});
console.log(response.output_text);
```
```python
from client import client
response = client.responses.create(
model="gpt-4o-mini",
input=[
{"role": "system", "content": "You answer in JSON."},
{
"role": "user",
"content": [
{"type": "input_text", "text": "Extract name and city from: 'Anna, Milan'"},
],
},
],
)
print(response.output_text)
```
### Stateless conversations (`store`, `previous_response_id`)
Caronte serves the Responses API **stateless-only**: every call is forced to `store: false` upstream, and when the request speaks reasoning (a `reasoning` param, or reasoning items replayed in `input`) the gateway also asks for the reasoning state **inline** as encrypted content (`include: ["reasoning.encrypted_content"]`, added automatically). Deployments that don't support returning encrypted reasoning simply omit reasoning items from the output — the replay contract below is identical either way. This is what lets the gateway load-balance a model name across multiple provider deployments — stored item ids are bound to the specific OpenAI/Azure resource that created them, so nothing may depend on server-side storage.
What this means for your client:
- **`previous_response_id` is rejected** with a `400` — there is no stored previous response to reference. Multi-turn conversations replay the full history in `input` instead: append the previous response's `output` items (including the `reasoning` items, unmodified — their `encrypted_content` carries the model's reasoning state) followed by your new user turn or `function_call_output` items. The OpenAI Agents SDK and similar frameworks already do exactly this.
- **Omitting `store` (or setting `store: true`) adds a `Warning` response header** (`299 caronte "store is forced to false…"`) so the behavior change is visible; the request still succeeds with `store: false` applied. Pass `store: false` explicitly to acknowledge the contract and silence the warning.
- `response.id` values cannot be retrieved later (`GET /v1/responses/{id}` is not served) — persist what you need client-side.
Prompt caching is unaffected: it keys on the request prefix, not on storage.
---
## Messages API (Anthropic-compatible)
The Anthropic-style Messages endpoint lets Anthropic SDK clients — and Claude Code — talk to Caronte. It speaks the Anthropic wire format for requests, responses, streaming events and errors, and it is provider-agnostic: the same request runs against Claude, GPT, Gemini or Bedrock models, translated transparently.
- **Endpoints:** `POST /anthropic/v1/messages`, `POST /anthropic/v1/messages/count_tokens`.
- **Base URL:** the Messages dialect lives under the `/anthropic` prefix (NOT `/v1`). Set the SDK base URL to `https://<your-caronte>/anthropic`; the SDK appends `/v1/messages`.
- **Auth:** your Caronte key (`sk_...`), accepted as `x-api-key` (Anthropic SDK default) or `Authorization: Bearer`. The `anthropic-version` / `anthropic-beta` headers are accepted but not required or validated.
- **This same URL has a second mode.** Send `X-Caronte-Developer-Token` and your own provider credential instead of a Caronte key, and the request becomes an untouched pass-through traced under your developer identity — see [Developer tools (Claude Code)](#developer-tools-claude-code). Everything described below applies to the Caronte-key mode only.
### Point the Anthropic SDK at the gateway
```typescript
// client.ts
import Anthropic from "@anthropic-ai/sdk";
export const anthropic = new Anthropic({
apiKey: process.env.CARONTE_API_KEY!, // sent as the x-api-key header
baseURL: process.env.CARONTE_ANTHROPIC_BASE_URL ?? "https://<your-caronte>/anthropic",
});
```
```python
# client.py
import os
from anthropic import Anthropic
anthropic = Anthropic(
api_key=os.environ["CARONTE_API_KEY"],
base_url=os.getenv("CARONTE_ANTHROPIC_BASE_URL", "https://<your-caronte>/anthropic"),
)
```
For Claude Code (or any Anthropic client), export two env vars:
```bash
export ANTHROPIC_BASE_URL="https://<your-caronte>/anthropic"
export ANTHROPIC_API_KEY="$CARONTE_API_KEY"
```
### Basic usage
`max_tokens` is required. The assistant reply is an array of content blocks.
```bash
curl https://<your-caronte>/anthropic/v1/messages \
-H "x-api-key: $CARONTE_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-5",
"max_tokens": 256,
"messages": [{ "role": "user", "content": "ping" }]
}'
```
Dispatch is provider-agnostic: pass a Claude model for full Anthropic-native fidelity, or any other model your key can call (`gpt-4o-mini`, a Gemini or Bedrock model) and the gateway translates the Anthropic request to that provider and back.
### Features
- **Streaming** (`stream: true`): named Anthropic SSE events (`message_start`, `content_block_delta`, `message_delta`, `message_stop`) — no `[DONE]` terminator.
- **Tool use**: declare tools with an `input_schema`; `tool_use` ⇄ `tool_result` blocks map to native function calling on non-Claude models. Server-tool variants (e.g. `web_search`) only work on the native Anthropic path.
- **System prompts & images**: top-level `system` (string or text blocks); `base64`/`url` image blocks.
- **Extended thinking**: the `thinking` parameter — see [Extended thinking](#extended-thinking).
- **Code execution / programmatic tool calling**: declare `{ "type": "code_execution_20260120", "name": "code_execution" }` (+ custom tools with `allowed_callers`). Requires the `code_executor` org feature; shares the runtime/containers documented under Code Interpreter.
- **count_tokens**: `POST /anthropic/v1/messages/count_tokens` returns `{ input_tokens }`; not a generation (no cost/guardrails/trace), `max_tokens` not required, PII masking still applies.
- **Gateway features**: budgets/request caps (429 over limit), guardrails, tracing headers (`X-Trace-*`, `X-User`, `X-No-Log`), and PII masking on non-streaming (streaming with active masking → 400 `pii_masking_streaming_unsupported`).
- **Prompt Vault**: pass a stored-prompt reference via `extra_body`'s `prompt` field — see [Prompt Vault](#prompt-vault). With no `messages` the prompt becomes the conversation (a text prompt becomes one `user` turn); with your own `messages`, only the system part is merged vault-first into the top-level `system` field (a system-only prompt then can't replace `messages`).
- **Errors**: Anthropic envelope `{ "type": "error", "error": { "type", "message" } }`. Request body capped at 32 MB.
---
## Extended thinking
Enable reasoning with `thinking`; the response carries `thinking` content blocks (before the `text` blocks) and reports reasoning tokens in `usage.output_tokens_details`.
```typescript
const message = await anthropic.messages.create({
model: "claude-sonnet-5",
max_tokens: 4096,
thinking: { type: "enabled", budget_tokens: 8192 },
messages: [{ role: "user", content: "Prove there are infinitely many primes." }],
});
for (const block of message.content) {
if (block.type === "thinking") console.log("[reasoning]", block.thinking);
if (block.type === "text") console.log(block.text);
}
```
Newer Claude models also accept `thinking: { type: "adaptive" }` + `output_config: { effort: "low" | "medium" | "high" }` instead of a fixed budget.
The endpoint hands your request to the resolved provider, which translates the reasoning request in and the reasoning output back. Anthropic's two shapes are normalized to one effort dial: `adaptive` uses `output_config.effort` (default `medium`); `enabled` buckets `budget_tokens` — `< 4096 → low`, `4096–16383 → medium`, `≥ 16384 → high`; `disabled`/absent → off.
| Provider | Request: `thinking` → upstream | Response: reasoning → `thinking` blocks |
|----------|-------------------------------|------------------------------------------|
| Anthropic (Claude, direct) | Verbatim passthrough (thinking, budget_tokens, output_config, signature, redacted_thinking). | Native thinking / redacted_thinking blocks with real signatures. |
| OpenAI · Azure OpenAI | effort → `reasoning.effort` on the Responses API (effort only, no summary requested). | Reasoning item `summary_text` → thinking block (empty signature). |
| Gemini (API & Vertex) | effort → `thinkingConfig` (`thinkingLevel` on Gemini 3, else numeric `thinkingBudget` 1024/8192/24576). | Gemini thought parts → thinking blocks (empty signature). |
| AWS Bedrock (incl. Claude-on-Bedrock) | effort → Converse thinking: adaptive + `output_config.effort` on recent Claude (Opus 4.6/4.7/4.8, Sonnet 4.6/5, Fable 5), else fixed `budget_tokens`. | NOT surfaced back as thinking blocks on this path. |
Caveats:
- **Signatures are Anthropic-only** — every translated path synthesizes an empty `signature`.
- **budget_tokens is lossy off Anthropic** — bucketed to low/medium/high, then re-expanded to the provider's own budget (`5000 → medium → 8192`).
- **History thinking is dropped** — `thinking`/`redacted_thinking` replayed in assistant history are stripped on translated paths; only the native Anthropic path keeps them.
- **OpenAI reasoning may have no text** — effort is requested but not a summary, so you may get the token count without a `thinking` block.
- **Bedrock reasoning output is not surfaced** — the budget is applied upstream (affecting quality/cost) but Converse reasoning is not translated back, and `thinking_tokens` reports 0.
- **Gemini rejects `xhigh`/`max`** effort (400); Anthropic and Bedrock accept them.
For full thinking fidelity (real signatures, redacted_thinking, multi-turn continuity, prompt caching), route to a model on the native `anthropic` provider.
---
## Streaming
Streaming uses the same SSE wire format as OpenAI: `data:` chunks terminated by `data: [DONE]`. The SDK abstracts this away, so you iterate over an async iterable.
> **PII masking + streaming.** If your API key has PII masking enabled, streaming requests (`stream: true`) are rejected with HTTP `400` (`code: "pii_masking_streaming_unsupported"`) — masking cannot be applied mid-stream, so the gateway fails closed rather than sending unmasked data to the provider. Use non-streaming calls on masked keys, or disable masking on the key. See [PII masking](#pii-masking).
### Chat Completions (stream)
```typescript
import { client } from "./client";
const stream = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Stream a short haiku about proxies." }],
stream: true,
});
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content;
if (delta) process.stdout.write(delta);
}
```
```python
from client import client
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Stream a short haiku about proxies."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
```
### Responses API (stream)
The Responses API streams a richer event sequence (`response.output_text.delta`, `response.completed`, etc.). The SDK exposes them as typed events.
```typescript
import { client } from "./client";
const stream = await client.responses.create({
model: "gpt-4o-mini",
input: "Stream a short haiku about proxies.",
stream: true,
});
for await (const event of stream) {
if (event.type === "response.output_text.delta") {
process.stdout.write(event.delta);
}
}
```
```python
from client import client
with client.responses.stream(
model="gpt-4o-mini",
input="Stream a short haiku about proxies.",
) as stream:
for event in stream:
if event.type == "response.output_text.delta":
print(event.delta, end="", flush=True)
```
---
## Image generation
Caronte exposes the OpenAI `/v1/images/*` family verbatim. There are three endpoints:
| Endpoint | Method | Body | Purpose |
|----------|--------|------|---------|
| `/v1/images/generations` | POST | JSON | Generate one or more images from a text prompt. |
| `/v1/images/edits` | POST | multipart/form-data | Edit a base image (optionally guided by a mask) from a text prompt. |
Each request is logged as a generation under the trace just like text endpoints, so spend, latency and token/image-usage land in the dashboard automatically. The same `metadata` fields described in [Monitoring & traces](#monitoring--traces) (`trace_id`, `trace_name`, `generation_name`, `session_id`, `session_name`) are honored on `/v1/images/generations`.
### Generate an image
JSON body, drop-in compatible with the OpenAI SDK.
```typescript
import { client } from "./client";
const result = await client.images.generate({
model: "dall-e-3",
prompt: "A flat-design illustration of Charon ferrying a small AI robot across a misty river.",
n: 1,
size: "1024x1024",
quality: "hd",
style: "vivid",
response_format: "b64_json",
});
console.log(result.data[0].b64_json?.slice(0, 32), "…");
```
```python
from client import client
result = client.images.generate(
model="dall-e-3",
prompt="A flat-design illustration of Charon ferrying a small AI robot across a misty river.",
n=1,
size="1024x1024",
quality="hd",
style="vivid",
response_format="b64_json",
)
print(result.data[0].b64_json[:32], "…")
```
Plain HTTP equivalent:
```bash
curl https://<your-caronte>/v1/images/generations \
-H "Authorization: Bearer $CARONTE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "dall-e-3",
"prompt": "A flat-design illustration of Charon ferrying a small AI robot across a misty river.",
"n": 1,
"size": "1024x1024",
"quality": "hd",
"style": "vivid",
"response_format": "b64_json"
}'
```
Request fields:
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `model` | string | yes | Image model id (e.g. `dall-e-3`, `dall-e-2`, `gpt-image-1`). Must be allowed on the API key. |
| `prompt` | string | yes | Text description of the image. |
| `n` | integer | | 1–10. Defaults to 1. `dall-e-3` only accepts 1. |
| `size` | string | | `WIDTHxHEIGHT` or `auto`. Accepted values depend on the model — see [Sizes & aspect ratios](#sizes--aspect-ratios). |
| `quality` | string | | `standard` / `hd` for `dall-e-3`; `high` / `medium` / `low` / `auto` for GPT Image models. |
| `style` | `"vivid" \| "natural"` | | `dall-e-3` only. |
| `background` | `"transparent" \| "opaque" \| "auto"` | | GPT Image models (`gpt-image-*`) only. |
| `output_format` | `"png" \| "jpeg" \| "webp"` | | GPT Image models only. |
| `output_compression` | integer | | 0–100, GPT Image models with `webp`/`jpeg`. |
| `moderation` | `"low" \| "auto"` | | GPT Image models only. |
| `response_format` | `"url" \| "b64_json"` | | Defaults to `url`. Use `b64_json` when the URL host isn't reachable from the client. |
| `seed` | integer \| null | | Generation seed — Gemini only, not forwarded to OpenAI. |
| `user` | string | | Stable end-user identifier. Required when the API key has `force_user: true`. |
| `metadata` | object | | Trace metadata (`trace_id`, `trace_name`, `generation_name`, `session_id`, `session_name`). |
Response shape:
```json
{
"created": 1716651200,
"data": [
{
"url": "https://...",
"b64_json": null,
"revised_prompt": "A flat-design illustration… (model-rewritten prompt)"
}
],
"usage": {
"input_tokens": 42,
"output_tokens": 0,
"total_tokens": 42
}
}
```
### Edit an image
Multipart form-data — send the image (and optionally a mask) as a file part. White pixels in the mask mark the region the model is allowed to repaint.
```typescript
import fs from "node:fs";
import { client } from "./client";
const result = await client.images.edit({
model: "gpt-image-1",
image: fs.createReadStream("./room.png") as never,
mask: fs.createReadStream("./room-mask.png") as never,
prompt: "Replace the sofa with a modern dark-green velvet one.",
size: "1024x1024",
response_format: "b64_json",
});
console.log(result.data[0].b64_json?.slice(0, 32), "…");
```
```python
from client import client
with open("room.png", "rb") as image, open("room-mask.png", "rb") as mask:
result = client.images.edit(
model="gpt-image-1",
image=image,
mask=mask,
prompt="Replace the sofa with a modern dark-green velvet one.",
size="1024x1024",
response_format="b64_json",
)
print(result.data[0].b64_json[:32], "…")
```
```bash
curl https://<your-caronte>/v1/images/edits \
-H "Authorization: Bearer $CARONTE_API_KEY" \
-F model="gpt-image-1" \
-F prompt="Replace the sofa with a modern dark-green velvet one." \
-F size="1024x1024" \
-F response_format="b64_json" \
-F image=@room.png \
-F mask=@room-mask.png
```
Form fields (in addition to `image`, `mask`, `prompt`, `model`):
| Field | Type | Description |
|-------|------|-------------|
| `n` | integer | 1–10. |
| `size` | string | `WIDTHxHEIGHT` or `auto`. Accepted values depend on the model — see [Sizes & aspect ratios](#sizes--aspect-ratios). |
| `quality` | string | `high` / `medium` / `low` / `standard` / `auto` — GPT Image models (`gpt-image-*`) only (except `standard`, which is the `dall-e-2` value). |
| `background` | `"transparent" \| "opaque" \| "auto"` | GPT Image models only. |
| `input_fidelity` | `"high" \| "low"` | GPT Image models only. |
| `output_format` | `"png" \| "jpeg" \| "webp"` | GPT Image models only. |
| `output_compression` | integer | 0–100, only for GPT Image models with `webp`/`jpeg`. |
| `response_format` | `"url" \| "b64_json"` | `url` for `dall-e-2`, `b64_json` for `gpt-image-1`. |
| `user` | string | Stable end-user identifier. |
On OpenAI models every field of both endpoints is forwarded verbatim, except `metadata` (Caronte tracing) and `seed` (Gemini only): parameters OpenAI adds work without a gateway update, and an invalid value comes back as OpenAI's own 400. `stream` is not supported and is rejected with a 400.
### Sizes & aspect ratios
`size` is the only field that controls the output format, and both endpoints accept the same values. How it is applied depends on the provider behind the model.
**OpenAI models** — `size` is forwarded verbatim and OpenAI validates it (e.g. `dall-e-3`: `1024x1024`, `1792x1024`, `1024x1792`; `gpt-image-1`: `1024x1024`, `1536x1024`, `1024x1536`, `auto`). A value the model does not support comes back as OpenAI's own 400.
**Gemini models** — Gemini takes an aspect ratio and a resolution tier, not a pixel size. The gateway converts the proportions of `WIDTHxHEIGHT` to the nearest of the ratios below, so any size with the right proportions works — the OpenAI sizes too (`1536x1024` → 3:2, `1792x1024` → 16:9).
| Aspect ratio | Example size (1K) | Availability |
|--------------|-------------------|--------------|
| 1:1 | `1024x1024` | All Gemini image models |
| 2:3 / 3:2 | `848x1264` / `1264x848` | All Gemini image models |
| 3:4 / 4:3 | `896x1200` / `1200x896` | All Gemini image models |
| 4:5 / 5:4 | `928x1152` / `1152x928` | All Gemini image models |
| 9:16 / 16:9 | `768x1376` / `1376x768` | All Gemini image models |
| 21:9 | `1584x672` | All Gemini image models |
| 1:4 / 4:1 | `512x2048` / `2048x512` | Gemini 3.1 Flash Image family only |
| 1:8 / 8:1 | `384x3072` / `3072x384` | Gemini 3.1 Flash Image family only |
- The gateway does not check which ratios a model supports: a model that lacks one (e.g. `1:4` on Gemini 2.5 Flash Image or Gemini 3 Pro Image) answers with a 400.
- The resolution tier follows the pixel count of `size`: under 2 megapixels is 1K (the model default), 2–8 megapixels is 2K, above is 4K — e.g. `2048x2048` or `2752x1536` → 2K, `4096x4096` → 4K. Higher tiers cost more output tokens. Models limited to 1K (e.g. Gemini 3.1 Flash-Lite Image) reject 2K and 4K with a 400; the gateway appends the `size` and the tier it was sent as to the error message.
- With `auto` or no `size`, generations use the model's default ratio and edits follow the input image.
- A `size` that is not `WIDTHxHEIGHT` or `auto` (e.g. `16:9`) is rejected with a 400 on `param: "size"`.
- Gemini-native fields such as `aspect_ratio` or `image_size` in the body are ignored.
### Notes
- The API key must have access to the image model — same allow-list rules as text models.
- Budgets and request caps apply to image endpoints exactly like to chat endpoints. PII masking does **not** cover image endpoints — the `prompt` is forwarded as-is even when a key has masking enabled (masking is limited to non-streaming chat/responses). See [PII masking](#pii-masking).
- For very large `b64_json` payloads prefer `response_format: "url"` to avoid base64-bloating client memory.
- Uploaded image files (and masks) must be under 50 MiB — the same hard cap that applies to `/v1/files` uploads.
---
## Function calling
Define your own tools and let the model decide when to call them. The proxy forwards the OpenAI tool schema verbatim — nothing Caronte-specific here.
### Declare a tool
```typescript
import { client } from "./client";
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "What's the weather in Milan?" }],
tools: [
{
type: "function",
function: {
name: "get_weather",
description: "Get the current weather for a city.",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
],
});
const toolCall = response.choices[0].message.tool_calls?.[0];
console.log(toolCall?.function.name, toolCall?.function.arguments);
```
```python
from client import client
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "What's the weather in Milan?"}],
tools=[
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}
],
)
tool_call = response.choices[0].message.tool_calls[0]
print(tool_call.function.name, tool_call.function.arguments)
```
### Return the tool result
Append the assistant message that emitted the `tool_call` plus a `tool` message carrying the result, then re-issue the request.
```typescript
import { client } from "./client";
const result = JSON.stringify({ city: "Milan", temp_c: 21 });
const followup = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "user", content: "What's the weather in Milan?" },
/* assistant tool_call message from previous response */
{
role: "tool",
tool_call_id: "<call_id from previous response>",
content: result,
},
],
});
console.log(followup.choices[0].message.content);
```
```python
import json
from client import client
result = json.dumps({"city": "Milan", "temp_c": 21})
followup = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": "What's the weather in Milan?"},
# assistant tool_call message from previous response
{
"role": "tool",
"tool_call_id": "<call_id from previous response>",
"content": result,
},
],
)
print(followup.choices[0].message.content)
```
---
## Monitoring & traces
Every request that hits Caronte is logged as a *generation* under a *trace*, which can be grouped into a *session*. You don't need to do anything to enable this, but setting a few optional fields in `metadata` makes the dashboard much more useful (related turns grouped together, human-readable names in the traces list, end-to-end conversation views).
### The trace model
Caronte uses a three-level hierarchy: `session → trace → generation`. Each level has an id (for grouping) and an optional name (for display).
- **Generation** — a single API call (one chat completion, one response, one embedding). This is what you see as a row in the traces list.
- **Trace** — a group of generations that belong to the same logical request (e.g. multiple turns of one conversation, or a tool-use loop).
- **Session** — a group of traces that belong together at a higher level (e.g. one user's back-and-forth across several conversations).
### Metadata fields
Set any of these as string properties of `metadata` on the request body. They live inside the standard OpenAI `metadata` object — they are NOT top-level request parameters.
| Field | Type | Description |
|-------|------|-------------|
| `trace_id` | string | Groups multiple generations under the same trace. If absent, Caronte generates a UUID per generation. Also used as the PII masking session key, so reusing it keeps the same redacted ↔ original bindings across turns. |
| `trace_name` | string | Display name for the trace. Defaults to the API key's name. |
| `generation_name` | string | Display name for this generation. Defaults to a per-endpoint label. |
| `session_id` | string | Groups multiple traces under the same session. Defaults to the resolved `trace_id`. |
| `session_name` | string | Display name for the session. Defaults to `trace_name`. |
All five fields must be `string` when present — passing a number, object, or array returns a 400. UUIDs are not required, any non-empty string works.
All five can alternatively be sent as the `X-Trace-Id` / `X-Trace-Name` / `X-Generation-Name` / `X-Session-Id` / `X-Session-Name` request headers (see [Custom headers](#custom-headers)) — handy on the multipart audio/image endpoints. The header takes precedence over the same field in the body `metadata`.
### Single-turn (no grouping)
```typescript
import { client } from "./client";
await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Hello" }],
metadata: {
trace_name: "support-bot",
generation_name: "greeting",
},
});
```
```python
from client import client
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hello"}],
metadata={
"trace_name": "support-bot",
"generation_name": "greeting",
},
)
```
### Multi-turn conversation (one trace)
Generate a `trace_id` at the start of the conversation and reuse it on every turn — all generations land on the same trace.
```typescript
import { randomUUID } from "node:crypto";
import { client } from "./client";
const traceId = randomUUID();
const meta = {
trace_id: traceId,
trace_name: "support ticket #4291",
};
await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "My app keeps crashing." }],
metadata: { ...meta, generation_name: "classify" },
});
// …later, same conversation
await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [/* full history */],
metadata: { ...meta, generation_name: "draft-reply" },
});
```
```python
from uuid import uuid4
from client import client
trace_id = str(uuid4())
meta = {
"trace_id": trace_id,
"trace_name": "support ticket #4291",
}
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "My app keeps crashing."}],
metadata={**meta, "generation_name": "classify"},
)
# …later, same conversation
client.chat.completions.create(
model="gpt-4o-mini",
messages=[...], # full history
metadata={**meta, "generation_name": "draft-reply"},
)
```
### Sessions (cross-conversation grouping)
Use `session_id` to group several traces that share a higher-level context — typically the same end user across multiple conversations.
```typescript
import { randomUUID } from "node:crypto";
import { client } from "./client";
const sessionId = "user_42";
const sessionName = "Anna Bianchi";
// First conversation
await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "How do I reset my password?" }],
metadata: {
session_id: sessionId,
session_name: sessionName,
trace_id: randomUUID(),
trace_name: "password-reset",
},
});
// Different conversation, same user
await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Cancel my subscription." }],
metadata: {
session_id: sessionId,
session_name: sessionName,
trace_id: randomUUID(),
trace_name: "cancel-subscription",
},
});
```
```python
from uuid import uuid4
from client import client
session_id = "user_42"
session_name = "Anna Bianchi"
# First conversation
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "How do I reset my password?"}],
metadata={
"session_id": session_id,
"session_name": session_name,
"trace_id": str(uuid4()),
"trace_name": "password-reset",
},
)
# Different conversation, same user
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Cancel my subscription."}],
metadata={
"session_id": session_id,
"session_name": session_name,
"trace_id": str(uuid4()),
"trace_name": "cancel-subscription",
},
)
```
### Default behavior
| Field | Fallback |
|-------|----------|
| `trace_id` | Random UUID generated per request. |
| `trace_name` | API key's name. |
| `generation_name` | Per-endpoint label — `Chat Completion`, `Responses`, `Embedding`, … |
| `session_id` | Resolved `trace_id`. |
| `session_name` | Resolved `trace_name`. |
### Disabling payload storage (X-No-Log)
By default Caronte stores the request and response bodies of every generation so you can inspect inputs/outputs on the trace. Set the `X-No-Log` **request header** (not a `metadata` field) to skip that for a request — useful for sensitive data, compliance, or noisy internal traffic.
The generation is still recorded: model, token counts, cost, latency, budget/cap accounting and your `metadata` all land in the dashboard. Only the request/response **bodies** are skipped — nothing is uploaded to S3, and the trace shows `payload not stored` instead of inputs/outputs. Skipped payloads are never written and cannot be recovered later.
Truthy values — `1`, `true`, `yes`, or the bare presence of the header — disable storage. `0`, `false`, `no` and an empty value are treated as off (the default). It applies to every endpoint. To disable storage for all traffic, set it once via `defaultHeaders` on the client.
```typescript
import { client } from "./client";
// Per call — the body is not stored, but cost/usage/latency still are.
await client.chat.completions.create(
{
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Contains sensitive data…" }],
metadata: { trace_name: "pii-heavy-job" },
},
{ headers: { "X-No-Log": "1" } },
);
```
```python
from client import client
# Per call — the body is not stored, but cost/usage/latency still are.
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Contains sensitive data…"}],
metadata={"trace_name": "pii-heavy-job"},
extra_headers={"X-No-Log": "1"},
)
```
```bash
curl https://<your-caronte>/v1/chat/completions \
-H "Authorization: Bearer $CARONTE_API_KEY" \
-H "Content-Type: application/json" \
-H "X-No-Log: 1" \
-d '{
"model": "gpt-4o-mini",
"messages": [{ "role": "user", "content": "Contains sensitive data…" }]
}'
```
---
## End-user identifier
Caronte can attribute every generation to an **end user** — the person on whose behalf your application called the API. The identifier drives three things: who the trace is attributed to in the dashboard, whether the request is allowed at all (`force_user`), and which per-user budget / request cap applies.
### Where it comes from
At request time Caronte resolves the identifier in this order (first match wins):
1. The `X-User` request header (see [Custom headers](#custom-headers)).
2. The body `user` field on the OpenAI endpoints (`/v1/chat/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/images/*`), or `metadata.user_id` on the Messages API.
If neither is present, the generation is attributed to the API key's **default user** when the key has one configured. That fallback only fills in attribution — it does **not** satisfy `force_user`, and per-user budgets/caps still key off the header/body value.
```typescript
import { client } from "./client";
// Body field (OpenAI convention)…
await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Hello" }],
user: "user_42",
});
// …or the header — works on every endpoint and wins over the body:
await client.chat.completions.create(
{ model: "gpt-4o-mini", messages: [{ role: "user", content: "Hello" }] },
{ headers: { "X-User": "user_42" } },
);
```
```python
from client import client
# Body field (OpenAI convention)…
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hello"}],
user="user_42",
)
# …or the header — works on every endpoint and wins over the body:
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"X-User": "user_42"},
)
```
### force_user — requiring an identifier
An API key can be configured to **require** an end-user identifier on every request. When `force_user` is on and a request supplies neither `X-User` nor a body `user` / `metadata.user_id`, Caronte rejects it with `400` / `invalid_request_error` ("User is required and must be a string") before it reaches a provider. A key that does not force the user must instead carry a **default user**, so every generation always resolves to some identifier.
### Per-user budgets & request caps
A budget or request cap on a key can be **key-level** (applies to all traffic on the key) or **per-user** (applies only to a specific end-user identifier). When a request resolves to a user, both the key-level limit **and** that user's per-user limit are enforced; if either is exceeded the request returns `429` / `budget_exceeded` (or the request-cap equivalent). Per-user limits therefore only bind when the request carries a `user` / `X-User` — without one, only the key-level limits apply. Warning headers (`X-Budget-Warning`, `X-RequestCap-Remaining`) ride on the response as a limit gets close.
---
## PII masking
Caronte can redact personal data from your requests before they reach the upstream provider, then restore the original values in the response it returns to you. There is nothing to set in the request body, and your code keeps calling the standard OpenAI-compatible endpoints — an administrator configures it in the Caronte dashboard, on one of two axes:
- **per API key**, for LLM calls (`/v1/*`) — the policy follows the credential;
- **per MCP server**, for gateway tool calls (`/mcp/<slug>`) — the policy follows the recipient, so it holds for every caller that reaches that server. See [MCP Gateway](#mcp-gateway).
### What happens when it is enabled
For **non-streaming** `/v1/chat/completions` and `/v1/responses`:
1. The gateway detects entities (emails, phone numbers, IBANs, credit cards, names, plus any custom org-specific patterns) in your request and replaces them with stable placeholders such as `<PERSON_1>` or `<EMAIL_1>`.
2. The **masked** request is what the provider actually sees — raw PII never leaves the gateway.
3. The provider's response is **rehydrated**: placeholders are swapped back to the original values before you receive it.
The redaction is transparent — request and response look normal to your code. In monitoring, the trace stores both a clear and a masked payload so operators can audit what the provider received.
### Streaming is not supported (fail-closed)
Masking cannot be applied to a streamed response, so on a key with masking enabled a `stream: true` request is **rejected with HTTP `400`** instead of being forwarded unmasked:
```json
{
"error": {
"message": "PII masking is enabled for this API key but is not supported on streaming requests. Send a non-streaming request, or disable PII masking for this key.",
"type": "client_error",
"code": "pii_masking_streaming_unsupported"
}
}
```
The request never reaches a provider and nothing is logged. To stream, either drop `stream: true` or ask an administrator to disable masking on the key.
### Stable placeholders across turns
Pass the same `metadata.trace_id` on every call of a conversation and the same original value always maps to the same placeholder (`<PERSON_1>` stays `<PERSON_1>`), so multi-turn context stays coherent for the model. See [Monitoring & traces](#monitoring--traces) for how `trace_id` groups generations.
### Coverage
| Endpoint | PII masking |
| --- | --- |
| `/v1/chat/completions`, `/v1/responses` — non-streaming | ✅ masked + rehydrated |
| `/v1/chat/completions`, `/v1/responses` — streaming | ⛔ rejected with `400` on a masked key |
| `/v1/embeddings` | ❌ not masked (would distort the vector) |
| `/v1/images/*`, audio, files | ❌ not masked |
| `/mcp/<slug>` — `tools/call` | ✅ arguments masked + result rehydrated, config on the SERVER (see [MCP Gateway](#mcp-gateway)) |
## Code Interpreter (hosted server tool)
Caronte ships a sandboxed Python / JavaScript runtime that the model can drive as a hosted server tool on `/v1/responses`. The proxy intercepts the tool call, runs it in `worker-code-exec`, and feeds the result back to the model — your client never sees the inner round-trips. Generated files are uploaded to S3 and surfaced as `container_file_citation` annotations.
### How it works
1. Turn it on by enabling the code execution tool on your request, just like any other tool.
2. The model decides when to use it — for math, data transformations, generating a chart or file, and similar tasks.
3. Code runs in a safe sandbox, isolated from your app and your data.
4. If the code needs information only your app has, it pauses and asks for it through a normal tool call, then continues once you provide the answer.
5. The final result — output, errors, and any generated files — comes back to the model, which uses it to write the answer.
6. You can ask the model to read or produce files, and then download them from the proxy.
### Minimal example
```typescript
import { client } from "./client";
const response = await client.responses.create({
model: "gpt-4o-mini",
input: "Compute the mean and standard deviation of [3, 7, 7, 19, 24].",
tools: [
{ type: "caronte:code_execution" } as never,
],
});
console.log(response.output_text);
```
```python
from client import client
response = client.responses.create(
model="gpt-4o-mini",
input="Compute the mean and standard deviation of [3, 7, 7, 19, 24].",
tools=[{"type": "caronte:code_execution"}],
)
print(response.output_text)
```
### Tool marker fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | `"caronte:code_execution"` | yes | Identifies the hosted Code Interpreter tool. |
| `container` | string | | Public container id (`cnt_…`) the run attaches to. Files already in the container are mounted read-only under `./external-files/`; new files produced by the run are appended to it. |
| `file_ids` | string[] | | Uploaded file ids (`file_…`) to mount in the sandbox. If no `container` is set, an implicit container is created automatically and returned in the result. |
| `allowed_tools` | string[] | | Whitelist of your client-declared function tools to expose as Python/JS stubs inside the sandbox. **Default-deny:** omit it or pass an empty array and NO tool is callable from the executed code. Only the listed tools become stubs; the model may still call any declared function tool directly. |
```json
{
"type": "caronte:code_execution",
"container": "cnt_abc123",
"file_ids": ["file_xyz789"],
"allowed_tools": ["advanced_query"]
}
```
### Containers
A container is a virtual workspace that holds files made available to a run. TTL is idle-based (default 60 minutes since `last_active_at`, min 1 min, max 4 h), resets on every run that uses it.
`cnt_*` object fields:
| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Public id (`cnt_<hex>`) — pass it as `container`. |
| `status` | `"active" \| "expired" \| "deleted"` | Lifecycle state. |
| `created_at / last_active_at / expires_at` | number | Unix seconds. |
| `expires_after` | `{ anchor, minutes }` | Idle TTL. `anchor` is always `last_active_at`. |
| `implicit` | boolean | `true` when auto-created because a run was started with `file_ids` only. |
REST surface:
| Endpoint | Description |
|----------|-------------|
| `POST /v1/containers` | Create — body: `{ name?, expires_after?: { anchor: "last_active_at", minutes } }`. |
| `GET /v1/containers` | List — pagination via `limit` / `after`. |
| `GET /v1/containers/:id` | Retrieve. |
| `DELETE /v1/containers/:id` | Soft-delete; run history is preserved. |
| `GET /v1/containers/:id/files` | List files (user uploads + run outputs). |
| `POST /v1/containers/:id/files` | Attach — multipart `file` upload OR JSON `{ file_id }` of an already-uploaded file owned by the same API key. |
Example: create a container, upload a file, run on it.
```typescript
import { client } from "./client";
// 1. Create a workspace.
const container = await (client as any).containers.create({
name: "monthly-report",
expires_after: { anchor: "last_active_at", minutes: 60 },
});
// 2. Upload a CSV the model will be able to read from ./external-files/.
const file = await client.files.create({
file: await openAsFile("./sales.csv"),
purpose: "user_data",
} as never);
await fetch(`${process.env.CARONTE_BASE_URL}/containers/${container.id}/files`, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${process.env.CARONTE_API_KEY}`,
},
body: JSON.stringify({ file_id: file.id }),
});
// 3. Ask the model to analyse it.
const response = await client.responses.create({
model: "gpt-4o-mini",
input: "Plot total revenue by month from sales.csv and summarise the trend.",
tools: [
{ type: "caronte:code_execution", container: container.id } as never,
],
});
console.log(response.output_text);
```
```python
import os
import requests
from client import client
base = os.environ["CARONTE_BASE_URL"]
key = os.environ["CARONTE_API_KEY"]
# 1. Create a workspace.
container = requests.post(
f"{base}/containers",
headers={"Authorization": f"Bearer {key}"},
json={
"name": "monthly-report",
"expires_after": {"anchor": "last_active_at", "minutes": 60},
},
).json()
# 2. Upload a CSV the model will be able to read from ./external-files/.
with open("sales.csv", "rb") as f:
file = client.files.create(file=f, purpose="user_data")
requests.post(
f"{base}/containers/{container['id']}/files",
headers={
"Authorization": f"Bearer {key}",
"Content-Type": "application/json",
},
json={"file_id": file.id},
)
# 3. Ask the model to analyse it.
response = client.responses.create(
model="gpt-4o-mini",
input="Plot total revenue by month from sales.csv and summarise the trend.",
tools=[{"type": "caronte:code_execution", "container": container["id"]}],
)
print(response.output_text)
```
If you pass `file_ids` without a `container`, the proxy auto-creates an implicit container and returns its id in the tool result so you can reuse it on later turns.
### Tool result shape
```json
{
"run_id": "cexrun_a1b2c3...",
"container_id": "cnt_abc123",
"stdout": "Mean: 12.0\nStdev: 8.34\n",
"stderr": "",
"exit_code": 0,
"status": "completed",
"duration_ms": 432,
"inner_tool_calls": 0,
"files": [
{
"file_id": "cfile_9e8f...",
"filename": "plot.png",
"mime_type": "image/png",
"size_bytes": 18342,
"container_id": "cnt_abc123"
}
]
}
```
### File annotations on the final response
Files produced by the run surface on the final assistant message as `container_file_citation` annotations — the same shape OpenAI uses for hosted Code Interpreter.
```json
{
"type": "container_file_citation",
"container_id": "cnt_abc123",
"file_id": "cfile_9e8f...",
"filename": "plot.png",
"start_index": 122,
"end_index": 134
}
```
### Direct /v1/code-execution API
For programmatic use cases without a model in the loop, the same runtime is exposed under `/v1/code-execution/runs`. The request opens an SSE stream with a small event set; inner tool calls are resolved via POST.
| Event | Description |
|-------|-------------|
| `run.created` | Sent immediately. Payload: `{ runId }`. |
| `tool_call.required` | Emitted when the snippet calls a declared client tool. Payload: `{ runId, calls: [{ call_id, name, arguments }] }`. Resolve via `POST /v1/code-execution/runs/:runId/tool-results`. |
| `run.completed` | Final event on success. Payload: `{ runId, stdout, stderr, exit_code, duration_ms, inner_tool_calls }`. |
| `run.failed` | Final event on failure / cancellation. Payload: `{ runId, error, stdout, stderr, duration_ms }`. |
| `[DONE]` | Last SSE chunk, mirrors OpenAI's streaming convention. |
Start a run:
```bash
curl -N https://<your-caronte>/v1/code-execution/runs \
-H "Authorization: Bearer $CARONTE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"language": "python",
"code": "print(sum(range(10)))",
"tools": []
}'
```
Resolve an inner tool call:
```bash
curl https://<your-caronte>/v1/code-execution/runs/$RUN_ID/tool-results \
-H "Authorization: Bearer $CARONTE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"results": [
{ "call_id": "<call_id from tool_call.required>", "output": "{\"ok\":true}" }
]
}'
```
Cancel a run:
```bash
curl -X DELETE https://<your-caronte>/v1/code-execution/runs/$RUN_ID \
-H "Authorization: Bearer $CARONTE_API_KEY"
```
### Inner tool calls inside the sandbox
When you declare your own function tools alongside `caronte:code_execution`, the proxy auto-generates Python/JS stubs the snippet can call. Calling a stub pauses the run and surfaces a regular `function_call` to the model (or to the direct API caller); the result you provide is handed back as the stub's return value.
```python
# advanced_query was declared as a function tool on the request.
# The proxy auto-generates this stub inside the sandbox:
rows = advanced_query(sql="select count(*) from generations")
print(rows)
```
Choose which client tools the snippet can call by listing them in `allowed_tools` on the marker. This is **default-deny**: if you omit `allowed_tools` or pass an empty array, the snippet cannot call any function tool. The model can still invoke any declared tool directly through standard function calling — the whitelist only governs what runs *inside* the sandbox.
### Pre-installed libraries
The sandbox image ships with a curated data-science / document-generation toolkit. `pip install` is disabled (outbound HTTP is blocked anyway).
| Category | Language | Libraries |
|----------|----------|-----------|
| Data & scientific computing | python | `pandas`, `numpy`, `scipy`, `scikit-learn`, `sympy`, `networkx`. |
| Plotting & images | python | `matplotlib`, `seaborn`, `Pillow` (imported as `PIL`). |
| Spreadsheets | python | `openpyxl`, `xlsxwriter` — pair with `pandas.to_excel(...)` for quick exports. |
| PDF | python | `reportlab`, `pypdf`, `fpdf2` (imported as `fpdf`). |
| Office documents | python | `python-pptx` (imported as `pptx`), `python-docx` (imported as `docx`). |
| Parsing & formatting | python | `lxml`, `beautifulsoup4` (imported as `bs4`), `pyyaml` (imported as `yaml`), `tabulate`. |
| Synthetic data | python | `Faker` (imported as `faker`) — locale-aware fake names, addresses, companies, dates for fixtures and demo datasets. Seed with `Faker.seed(n)` for reproducible output. |
| HTTP client | python | `requests` is importable but the sandbox blocks outbound network — calls fail at connect time. Use it only for offline payload shaping. |
| Standard library | python | Full CPython 3 stdlib. |
| JavaScript runtime | javascript | `language: "javascript"` runs on Bun — only Bun + Web standard APIs available (`fetch`, `Bun.file`, `fs`, …). No third-party npm packages pre-installed; `bun add` is disabled. |
### Limits & defaults
| Limit | Value | Description |
|-------|-------|-------------|
| `timeout_ms` | default 30 000, max 300 000 (5 min) | Wall-clock cap per run. Passed as a parameter on the tool call. |
| Container TTL | default 60 min, min 1, max 240 (4 h) | Resets on every run that touches the container. |
| External files | read-only | Mounted under `./external-files/`. The snippet can read them but cannot mutate them in place. |
| Upload size | 50 MiB | Hard cap per file on `POST /v1/files` and the multipart variant of `POST /v1/containers/:id/files`. |
| External files per run | 50 files | Each individually capped at 50 MiB. Beyond that, attached files are skipped with a worker log line. |
| Generated files per run | 50 files / 50 MiB each | The worker scans the top level of the sandbox after the run and uploads at most 50 output files, each capped at 50 MiB. |
---
## Prompt Vault
Reference a prompt stored in the Prompt Vault instead of hard-coding it. The proxy resolves tag/version to a commit, substitutes variables, and merges the result into the request before guardrails and PII masking run. Available on `/v1/responses` (OpenAI's native `prompt` param), `/v1/chat/completions` and `/anthropic/v1/messages` (same object as an extra-body field — the gateway strips it before dispatching to the provider). Not supported on the token-counting endpoints.
### Invoke a stored prompt
```typescript
import { client } from "./client";
const response = await client.responses.create({
model: "gpt-4o-mini",
// No input/instructions: the Prompt Vault entry provides them.
prompt: {
id: "support/triage", // project/prompt path
version: "production", // tag or commit id (omit for latest)
variables: { ticket_subject: "App keeps crashing on launch" },
},
} as never);
console.log(response.output_text);
```
```python
from client import client
response = client.responses.create(
model="gpt-4o-mini",
prompt={
"id": "support/triage", # project/prompt path
"version": "production", # tag or commit id (omit for latest)
"variables": {"ticket_subject": "App keeps crashing on launch"},
}
)
print(response.output_text)
```
### Prompt field reference
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `id` | string | yes | Path of the prompt in the vault, formatted as `project/prompt`. |
| `version` | string | | Optional tag or commit id. Omit to use the latest published version. |
| `variables` | object | | Map of variables substituted into `{{block}}` placeholders inside the prompt body. |
You can still pass `input` alongside `prompt`: the prompt's system part is prepended to `instructions` (vault-first — any `instructions` you send are kept), your `input` stays the user turn, and the prompt's own user/assistant turns are dropped. A common pattern is to keep the system message and few-shot examples in the Prompt Vault, and pass only the per-call user turn via `input`.
### On Chat Completions and the Messages API
The same `prompt` object is accepted as an extra-body field (neither upstream API has a native stored-prompt param); `messages` becomes optional when the referenced prompt provides the conversation:
```typescript
const completion = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "App keeps crashing on launch" }],
// @ts-expect-error Caronte extension — or pass it via the SDK's extra-body support
prompt: { id: "support/triage", version: "production" },
});
```
```python
message = anthropic_client.messages.create(
model="claude-sonnet-5",
max_tokens=1024,
messages=[{"role": "user", "content": "App keeps crashing on launch"}],
extra_body={"prompt": {"id": "support/triage", "version": "production"}},
)
```
### How the prompt merges into your request
The behavior is identical across the three endpoints and follows two rules: **send no conversation of your own** and the compiled prompt *becomes* the conversation; **send one** and only the prompt's system part is injected — the prompt's user/assistant turns are dropped, because your request already owns the conversation.
| Your request | `/v1/responses` | `/v1/chat/completions` | `/anthropic/v1/messages` |
|---|---|---|---|
| **Text** prompt · no conversation sent | becomes `input` (the user turn) | one `user` message | one `user` message |
| **Text** prompt · you send a conversation | prepended to `instructions` | prepended as a `system` message | merged into top-level `system` |
| **Messages** prompt · no conversation sent | whole prompt → `input` | whole prompt → `messages` | system → `system`, the rest → `messages` |
| **Messages** prompt · you send a conversation | system → `instructions`; other turns dropped | system → `system` message; other turns dropped | system → `system`; other turns dropped |
**Vault-first, nothing discarded.** Wherever the prompt's system content lands next to a `system` / `instructions` value you already sent, the vault content goes first and yours is kept below it (`<vault>`, a blank line, then `<yours>`). The stored prompt is the base instruction set; your own system text is additive, never overwritten.
**A text-format prompt has no role of its own.** On its own it becomes the `user` turn, so a stored text prompt is directly runnable; the moment you supply your own conversation it becomes system content instead. Same inference on every endpoint.
**Two Messages-API specifics.** Anthropic has no `system` role inside `messages`, so a messages-format prompt made of only system/developer turns cannot stand alone on `/anthropic/v1/messages` — you must then send at least one message (on `/v1/responses` and `/v1/chat/completions` those system turns can). And when a prompt carries several system turns, `/anthropic/v1/messages` concatenates them all while `/v1/responses` and `/v1/chat/completions` use the first.
---
## OpenAI Agents SDK
The OpenAI Agents SDK (JS, Python) talks to the model through an OpenAI client, so the same trick works: override the default client to point at Caronte and run your agents unchanged.
### Override the default client
```typescript
import OpenAI from "openai";
import {
Agent,
run,
setDefaultOpenAIClient,
setOpenAIAPI,
} from "@openai/agents";
setDefaultOpenAIClient(
new OpenAI({
apiKey: process.env.CARONTE_API_KEY!,
baseURL: process.env.CARONTE_BASE_URL ?? "https://<your-caronte>/v1",
}),
);
// Use the Responses API (default) or "chat_completions".
setOpenAIAPI("responses");
const triage = new Agent({
name: "Triage",
instructions: "Classify the ticket and respond in one sentence.",
model: "gpt-4o-mini",
});
const result = await run(triage, "App keeps crashing on launch.");
console.log(result.finalOutput);
```
```python
import asyncio
import os
from openai import AsyncOpenAI
from agents import (
Agent,
Runner,
set_default_openai_client,
set_default_openai_api,
)
set_default_openai_client(
AsyncOpenAI(
api_key=os.environ["CARONTE_API_KEY"],
base_url=os.getenv("CARONTE_BASE_URL", "https://<your-caronte>/v1"),
)
)
# Use the Responses API (default) or "chat_completions".
set_default_openai_api("responses")
triage = Agent(
name="Triage",
instructions="Classify the ticket and respond in one sentence.",
model="gpt-4o-mini",
)
async def main() -> None:
result = await Runner.run(triage, "App keeps crashing on launch.")
print(result.final_output)
asyncio.run(main())
```
### Add hosted Code Interpreter to an agent
Agents that need the sandboxed runtime can declare `code_execution` as a hosted tool — the SDK forwards the marker into the Responses `tools[]` array verbatim.
```typescript
import { Agent, hostedTool } from "@openai/agents";
const analyst = new Agent({
name: "Analyst",
instructions: "Use Python to compute statistics when asked.",
model: "gpt-4o-mini",
tools: [
hostedTool({
name: "code_execution",
providerData: { type: "caronte:code_execution" },
}),
],
});
```
```python
from agents import Agent
analyst = Agent(
name="Analyst",
instructions="Use Python to compute statistics when asked.",
model="gpt-4o-mini",
tools=[
# Hosted tool: the provider_data payload is forwarded verbatim
# to the Responses tools[] array, where the proxy picks it up.
{
"type": "hosted_tool",
"name": "code_execution",
"provider_data": {"type": "caronte:code_execution"},
},
],
)
```
---
## Endpoints reference
All endpoints share the same authentication: `Authorization: Bearer <CARONTE_API_KEY>`. They also all honor Caronte's optional request headers — `X-No-Log` (keep request/response bodies out of storage), `X-Trace-Id` / `X-Trace-Name` / `X-Generation-Name` / `X-Session-Id` / `X-Session-Name` (tracing), and `X-User` (end-user identifier) — see [Custom headers](#custom-headers) and [End-user identifier](#end-user-identifier).
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/v1/models` | GET | List models the API key is allowed to call. |
| `/v1/chat/completions` | POST | Chat Completions (drop-in OpenAI). Accepts `prompt` (Prompt Vault) as an extra-body field. |
| `/v1/responses` | POST | Responses API. Accepts `prompt` (Prompt Vault) and the `caronte:code_execution` tool. |
| `/anthropic/v1/messages` | POST | Anthropic-compatible Messages API (base URL `.../anthropic`). Streaming, tools, thinking, code execution, `prompt` (Prompt Vault) as an extra-body field; auth via `x-api-key` or Bearer. |
| `/anthropic/v1/messages/count_tokens` | POST | Count input tokens for a Messages request (not a generation). |
| `/v1/embeddings` | POST | Embeddings. |
| `/v1/images/generations` | POST | Generate images from a prompt (JSON body). |
| `/v1/images/edits` | POST | Edit an image (with optional mask) from a prompt (multipart). |
| `/v1/files` | POST/GET | Upload / list files. 50 MiB max per upload. |
| `/v1/containers` | POST/GET/DELETE | Manage Code Interpreter containers. |
| `/v1/containers/:id/files` | POST/GET | Attach / list container files. |
| `/v1/code-execution/runs` | POST | Direct (no-model) sandboxed code execution (SSE). Same runtime as the `caronte:code_execution` hosted tool, exposed for programmatic use. |
| `/v1/code-execution/runs/:id/tool-results` | POST | Resolve inner tool calls from a direct run. |
| `/v1/code-execution/runs/:id` | DELETE | Cancel a direct run. |