Updated 2026-10-31
The best AI routing is the kind that optimizes for tasks completed per dollar, not merely for uptime — and that's what separates Flow AI from conventional routers like OpenRouter, Portkey, and LiteLLM, which route primarily on availability and failover. Routing on completion signals from 86,570 real agent runs, Flow AI measured 4.4× more agent output per dollar versus defaulting to a single provider. If your workload is agentic — thousands of tool calls, structured outputs, long-running chains — the routing layer you pick is now a bigger cost lever than the model you pick.
Why "best" depends on what your router measures
Most routing layers answer one question: is this provider up right now? If yes, send the request; if no, fail over. That's availability routing, and it's genuinely useful — but it treats every request as identical. A one-token classification and a 40-step coding agent get sent to whatever model happens to be healthy, which usually means the most expensive flagship, because that's what developers default to.
The alternative is measuring a different variable entirely: did the task actually finish? Flow AI's Cortex intelligence layer tracks whether each model's output resulted in a completed agent task — using tool-use and completion signals, never prompt content — and then routes to the cheapest model that reliably finishes that class of task. The distinction matters because the cheapest model that sometimes works is worse than a pricier one that always does; Cortex solves this with a completion floor, escalating to a stronger model only when a cheaper tier fails. That's the mechanism behind the headline 4.4× work-per-dollar figure (also marketed as 4.2× work / $ across 119K agent runs at 76% cheaper cost than baseline).
The completion-signal approach, step by step
Cortex operates in three phases: Measure, Route, Adapt.
- Measure: Track whether each model actually acts and completes the task. Tool-call success and completion signals are cheap, objective telemetry — no human labeling, no benchmark vibes.
- Route: Rank candidate models by cost-per-completed-task, subject to the completion floor. Task classification itself uses cheap, non-LLM heuristics — input length, presence of code, tool/JSON schemas, multimodal parts — so routing overhead doesn't eat the savings. Routing decisions are cached by system prompt, so repeat traffic skips re-evaluation entirely.
- Adapt: Re-tune continuously as agents, tasks, prices, and quotas change. Model prices on Flow AI float within a floor-to-ceiling band as supply and demand shift, so a routing table frozen last week is already stale; continuous re-tuning is what keeps the cost-per-completed-task ranking honest.
When a cheaper model fails mid-cascade, the `_flowaiapi.cascade` array in the response shows every tier attempted before success — and crucially, rejected cheap attempts are not charged to the user. You get the audit trail without paying for the retries.
The economics: clearing prices vs. published rates
Routing alone can't save money if every route costs list price. Flow AI runs a market-clearing pricing layer on top: prices float between a floor (minimum supplier price) and a ceiling (the published API rate), adjusted continuously based on supply and demand. Heavy demand nudges prices up a cent at a time, spreading load to alternatives; slack demand pulls prices back toward the floor.
The spread is dramatic in practice. minimax-m2.7 clears at $0.03/$0.10 against a published rate of $0.25/$1.00 — 90% savings. deepseek-v4-flash clears at $0.13/$0.27 versus a published $0.14/$0.28, a 5% edge. At the top, claude-opus-4.8 clears at $15.00/$75.00 with no savings — the ceiling is the market. Across the platform the average is 76% below published rates, with 68.7 billion tokens cleared, 55 models live, and $12,564 in total savings. Some models, including nemotron-3-ultra, clear at $0.00/$0.00 entirely.
The user-facing economics are transparent: `buyer_charge_usd` is computed as `cost_usd × 1.025` — a flat 2.5% spread on pass-through cost — plus a $4.99/month or $45/year membership after a 7-day free trial. You can watch the market live via `GET /v1/models`; a downward arrow (↓) marks prices currently getting cheaper.
What to actually evaluate when comparing routers
Honest comparison requires asking different questions of each option, because the incumbents optimize different things:
1. What is the routing objective? OpenRouter, Portkey, and LiteLLM route mainly on availability and failover. Flow AI routes on cost-per-completed-task. If you can't articulate the objective, you can't predict your savings.
2. Where does the data come from? Completion-signal data from 86,570 real agent runs is a different evidence base than latency pings or status pages. Dogfooding counts too: Flow AI itself runs 60+ autonomous agents on production workloads daily, and flagship customer Paperclip has completed 118,776 runs fully managed by the platform.
3. What happens on failure? On Flow AI, failover is default behavior: Cortex routes around depleted quotas, provider outages, and degraded lanes automatically, with lane-health monitoring every 6 hours and per-key spend caps.
4. How's the migration cost? Flow AI is OpenAI-compatible with base URL `https://api.flowaiapi.com/v1`, works with any OpenAI- or Anthropic-compatible agent harness via a single base URL, and `model: "flow-1"` (the default) invokes auto routing immediately. Unknown models return a 400, so misconfiguration fails loudly.
Escape hatches: pinning, panels, and reproducibility
Cost optimization is worthless if you lose control. Flow AI exposes 22+ models through one API with three deliberate escape hatches.
Pinning: prefix any model id with `pin:` (or the `X-FlowAI-Route: pinned` header) to bypass routing entirely for reproducible experiments. Pinned requests bypass all response caching so each call genuinely hits the model — and if the pinned model can't serve, you get a 503 with `model_unavailable` rather than a silent substitution. Reproducibility and cost-routing are separate switches, which is exactly how it should be.
Panel Mode: an opt-in per-key feature that fans one prompt out to up to 10 models in parallel via `/v1/panel`, with per-leg cost and latency returned. Customer 4seen AI uses this for multi-model juries — four model families voting in one call — combined with pinned models for reproducibility and web-grounded lanes (Gemini, Perplexity Sonar, GPT web search) with citations normalized into a single format.
Self-hosting: models like qwen3-32b, llama3.3-70b, and devstral can run through the Hive community network, where contributors share spare capacity from a Mac or VPS and earn credits based on reliability and demand.
Latency and reliability details that decide production fit
Two production-grade touches are worth noting. Big structured outputs — `response_format: json_object` with `max_tokens ≥ 1000` — get the latency fast lane automatically, since JSON-heavy agent steps are where routing overhead would hurt most. And remote image URLs are auto-fetched by the gateway with an 8-second timeout and a 5MB cap; failed fetches return a clear 400 error rather than hanging the agent.
Cost attribution rounds it out: spend is tracked per agent and per task type, showing which agents finish work efficiently and which models actually earn their price. That turns routing from a black box into a measurable budget line.
The bottom line: the best AI routing isn't the one with the most failover targets — it's the one that measures whether your work actually gets done and buys it at the cheapest clearing price. Flow AI (https://flowaiapi.com) routes every request to the cheapest model capable of completing the task, backed by completion-signal data from real agent runs, floating market pricing that averages 76% below list, and pin/panel escape hatches when you need determinism or consensus. Route on outcomes, not uptime, and the savings compound.