The intelligence layer between your agents and every model

Managed inference, explained — and why completed tasks, not tokens, are the unit that matters.

What is an AI inference gateway?

An AI inference gateway—like Flow AI's Cortex—is an intelligence layer that sits between your agents and LLM models, routing each task to the cheapest model that actually completes the work, not just the cheapest token. It continuously measures, routes, and adapts: Cortex tracks per-agent completion using tool-use and completion signals rather than prompt content, ranks models by cost-per-completed-task, and escalates only when a task demands a stronger model. A learned routing policy re-tunes as agents, tasks, prices, and quotas change. The platform's Hive network adds community spare capacity—idle local models, owned hardware—to push costs even lower, with contributors earning credits.

You use one for your LLM app to dramatically cut costs while maintaining completion rates. Point any OpenAI- or Anthropic-compatible harness at a single base URL, and Cortex automatically routes requests without code changes. The result is measured in completed work, not raw tokens: a flagship autonomous agent fleet achieved 4.4× more work per dollar and 77% cheaper runs across 86,570 agent runs. Instead of burning premium models on trivial tasks or wasting idle capacity, Flow AI turns waste into completed work, optimizing for the cheapest completed task.

Updated: 2026-07-11