Admin guide · Platform

Models

The Models page is the catalog of every language model the gateway can route to, across all configured providers (OpenAI / Azure, Google Gemini, AWS Bedrock, …).

The catalog

Filter by status, provider, capabilities and input/output modalities, or search by name.

The Models list with provider, regions, status, context window, capabilities, modalities and 7-day token volume. — Models — every routed model with its provider, regions and capabilities.

Each row shows the provider, deployment regions, status, context window, capabilities (Reasoning, Tools), modalities and 7-day token volume.

Model detail: pricing & performance

Click a model to see the numbers behind cost tracking and routing.

A model detail page: provider and capabilities, a KPI row with context, prices, latency, TTFT and throughput, a Pricing & regions table and the performance charts. — Model detail — one row per region, priced and measured separately.

Pricing & regions — the per-1M in/out/cache-read rates used for billing, one row per region the model is deployed in, alongside that region's measured latency, throughput and success rate. Regions are priced independently, so the KPI row above shows the base rate and points here for the rest. A Service tier selector appears where the provider has more than one (e.g. Standard, Priority), and Full pricing structure expands the long-context and cache-write tiers.
Performance — throughput, time-to-first-token and latency over the last 7 days, measured on real successful traffic and switchable between Daily and Hourly. Times attributable to the gateway (guardrail evaluation, PII masking) are excluded and shown separately as gateway overhead.
Projects using this model — who the volume actually belongs to, over the last 7 days.

note · Who can call which model

The catalog is org-wide, but access is scoped twice: each project fixes its model access, and each API key can restrict further within it. /v1/models only lists what the calling key can actually reach.

Smart routing

When several instances share one model name, the gateway splits traffic between them from live performance data — and the card shows you where it goes and why.

Deploy the same model name more than once — say gpt-4o on Azure in two regions — and every request has to pick one of them. The gateway scores each instance from the last 5 minutes of real traffic and splits requests proportionally, re-scoring continuously (every 10 seconds while things are moving, backing off to 5 minutes when they are stable). The Smart routing card on the model detail page is that decision, live.

Each instance gets a share of traffic and a score — a weighted blend of four factors, shown under its bar so a share is never just a number you have to trust:

err (weight 40%) — the provider's fault rate (rate limits and server errors). It dominates, because availability comes first.
lat (25%) — speed measured per output token, so an instance that happened to serve longer answers isn't punished for it. Time-to-first-token counts more than raw throughput.
util (20%) — how busy the instance already is. The least busy scores highest: this spreads load before a provider starts rate-limiting you.
cost (15%) — price per token, from the linked pricing row. The tiebreaker between instances that are otherwise equally good.

The instance carrying the preferred badge is simply the one currently scoring highest — it is a label on the outcome, not a setting you can pin.

tip · A low share is not a problem

The most common misreading. An instance can sit at 30% simply because util is deliberately steering traffic away from the busiest one, or because it costs slightly more — not because anything is wrong with it. Read the four factors under the bar before concluding an instance is unhealthy: they say which dimension actually moved the split.

An instance that starts failing is excluded outright — a circuit breaker, not a score penalty, so a fast-but-broken provider can't average its way back in. It receives no scheduled traffic (its bar is empty and hatched) while remaining available as a last-resort failover, and re-enters the split by itself once it recovers. Requests that hit a failing instance mid-flight are retried on the next one automatically, so a single provider going down does not surface as an error to your callers.

note · Two labels worth knowing

Scoring inputs (behind the tooltip on each row) are smoothed values, not raw measurements: low-sample percentiles are pulled toward the group average and blended across recomputes so one unlucky request can't swing the routing. They are what the scoring uses, so they deliberately differ from the raw averages in Performance above. And “split unchanged for …” in the footer is not a staleness warning: the gateway keeps rescoring on the stated cadence and often finds the same answer, so a long stretch there simply means your traffic is stable. What is worth investigating is the explicit warning that the table is no longer being refreshed, or missing altogether — that means routing has fallen back to picking the cheapest instance.

Smart routing needs nothing switched on: it applies as soon as a name has two or more active instances. With a single instance the card simply says so — there is nothing to route between.