LLM routing: choose models and measure the tradeoffs

Separate task judgment from execution rules

Jev is TypeSafe AI's structured decision model. It receives state and questions and returns typed signals. Jevrouter uses those signals to characterize a task; deterministic code enforces allowed models, capabilities, prices and budgets before a generation model produces the answer.

The virtual model ID jev/auto refers to Jevrouter's routing policy, not to a model that writes the response. A fixed model ID bypasses automatic model choice. Catalog inclusion does not mean a model has been evaluated for the automatic pool.

Enable an evaluated policy explicitly

Select a project and enable its published Balanced policy in Routing. The inference key must allow jev/auto and the real candidate models. Project restrictions still apply; permission to use the virtual route is not permission to use every model in the catalog.

The current initial Balanced pool uses Gemini 2.5 Flash Lite and GPT-4.1 Mini. Low confidence or a Jev timeout uses an eligible default. Disabling conversation sharing with Jev also uses the policy default. If no allowed, capable route exists, the request fails rather than ignoring your restrictions.

"model": "jev/auto"

Compare routing with meaningful baselines

Use representative tasks with an acceptance check defined before testing. Compare at least a fixed quality baseline, a cheaper fixed model, a rules-only selector and the Jev policy. Keep prompt versions, provider choices, time window and scoring method consistent.

Count the Jev decision cost, generation cost, rejected or repeated attempts, and any verification step. Measure accepted-task rate, cost per accepted task, time to first token and complete-response latency. A routing confidence score is not the probability that the generated answer is correct.

• Keep a held-out evaluation set rather than tuning and reporting on the same examples.
• Test languages, long conversations and tool calls separately.
• Include route failures and empty or truncated answers in the result.
• Report uncertainty and sample size along with average savings.

What the current smoke check does and does not show

Jevrouter's first routing check used ten short English and Chinese examples across two generation models. All four strategies passed its simple match checks. The Jev policy's measured supplier cost was $0.000454186; fixed GPT-4.1 Mini cost $0.0003412; the cheaper fixed model cost $0.000126.

That small connectivity check did not demonstrate savings or establish a general quality ranking. Some checks were simple string matches, and one response was truncated despite matching. Routing benefits remain a workload-specific hypothesis to test. There is no advertised universal percentage saving.

Models mentioned in this guide