Call a model
Call Federation models with ModelCall, CityModel, and ModelStreamEvent.
Call a Federation model
const catalog = await city.ai.catalog();
const model = catalog.require("deepseek-v4-flash");
const stream = await model.stream({
messages: [{
role: "user",
content: [{ type: "text", text: "Explain this repository" }],
}],
});
for await (const event of stream) {
if (event.type === "text_delta") process.stdout.write(event.delta);
}model.stream() returns a ReadableStream<ModelStreamEvent>. A successful stream begins with model_start and provides final model_usage before model_finish. Invalid event ordering fails as a protocol error.
Use a model with Agent
Objects returned by the catalog implement ModelClient and can be passed directly to Agent:
const catalog = await city.ai.catalog();
const model = catalog.require("deepseek-v4-flash");
const agent = new Agent({ id: "reviewer", model });Agent owns Session context, tool loops, and approvals. Federation owns final model selection and billing.
Multimodal fallback
Fallback checks file content only in the latest role: "user" message in ModelCall.messages. A new text-only user message does not trigger fallback again. When the final model's fallback rules reject a historical file, the execution view replaces that file with its filename, original URL, or MIME type as plain text. The original Session file remains unchanged. Trailing assistant or tool messages do not change which user message is inspected.
OpenAI-compatible clients
Use /v1/ai/chat/completions for the OpenAI SDK, curl, or an existing gateway. The boundary adapter converts that request into the same ModelCall, so routing, fallback, reasoning, usage, and billing stay consistent.
Model execution deadlines
Agent model consumption, the in-process OpenAI-compatible client, and Federation language model execution share the same stream deadline guard. Federation applies the server-side AIServiceOptions.model_stream?: Partial<AIModelStreamPolicy> to native /v1/ai/stream, OpenAI-compatible SSE and JSON, and Relay. Omitted fields use these defaults. An individual client request cannot raise the server limits.
/** Configure language model execution deadlines in milliseconds. */
import { AIService } from "@downcity/federation";
const ai = new AIService({
model_stream: {
connect_timeout_ms: 120_000,
first_output_timeout_ms: 120_000,
idle_timeout_ms: 120_000,
total_timeout_ms: 900_000,
},
});| Policy | Default | Clock |
|---|---|---|
connect_timeout_ms | First-output budget, 120 seconds by default | Waiting for the provider to return a response stream; this is not a TCP connection timeout and also consumes the first-output budget |
first_output_timeout_ms | 120 seconds | Cumulative time waiting for provider open and actual reads before the first useful output |
idle_timeout_ms | 120 seconds | Cumulative time waiting for actual reads between useful outputs; each useful output resets the budget |
total_timeout_ms | 15 minutes | Wall-clock time from the start of provider execution, including open, reads, and consumer backpressure |
Consumer backpressure does not consume the first-output or idle budget; the total clock keeps running. model_start, text_start, reasoning_start, empty deltas, and usage are not useful output and do not renew the budget. Non-empty text_delta / reasoning_delta, tool_call_start, non-empty tool_call_delta.input_delta, and tool_call_finish count as useful output. Values must be safe integers from 1 through 2,147,483,647. Unknown fields or invalid values fail at construction.
Shorter provider deadlines, the default 180-second synchronous HTTP execution deadline, and the model policy apply independently. Whichever expires first aborts the same foreground execution. The stream guard does not retry requests. Agent owns bounded recovery before partial output; calling model.stream() directly does not enable Agent recovery.
For an in-process client, pass partial model_stream overrides to create_openai_compatible_model(options). The returned client exposes the resolved model_stream_policy, which Agent also honors.
Model failures, usage, and settlement
Model deadline failures use code: "provider_timeout", status_code: 504, and a timeout_phase of connect, first_output, idle, or total. Response-stream establishment and first-output timeouts are transient (retryable: true); idle and total expiry are not retryable. retryable classifies the failure; Agent separately checks partial output, cancellation, attempt count, and its recovery budget before resending. A provider may already have accepted the request, so no output does not guarantee no upstream work or charges.
Before a response starts, a model timeout returns HTTP 504. Once streaming starts, the native protocol emits model_error with the safe message Model provider timed out. If the provider has not emitted model_start, the transport may first emit a framework model_start solely to make the failure terminal conform to the protocol. It does not imply useful provider output or fabricate usage. OpenAI SSE emits an error frame and does not send a success [DONE] on failure; OpenAI JSON aggregation returns 504 on timeout. An SSE response that has started keeps its HTTP status.
Agent allows at most five retries, six calls including the initial attempt, only before partial output and tool side effects. All model attempts and backoff for one step share a default 900-second wall-clock budget; tool execution follows the Turn cancellation signal separately. Backoff uses 300, 1,000, 2,000, 4,000, and 8,000 milliseconds with ±10% jitter. Retry-After is carried as the absolute Unix millisecond deadline retry_at_ms; Agent does not retry before that deadline and stops when it cannot fit within the remaining budget. Cancellation immediately ends waiting and stream reads. Budget expiry uses timeout_phase: "recovery_budget", retryable: false, and the safe message Model recovery budget exhausted; it is not a user cancellation. Existing warning / model_request events report will_retry and, when retrying, next_retry_at_ms. Agent does not resume after partial output.
Preserve all trustworthy usage facts already received. Without returned usage, tokens and costs are not estimated. Existing bill callbacks use actual facts; the business determines charging, and missing usage is not a promise of zero charges in every case. completion.outcome explicitly distinguishes succeeded, failed, and cancelled. A cancel after the terminal event does not relabel the result as cancellation or timeout, or cause duplicate settlement or charges.
Media requests, HTTP deadlines, and cancellation
HTTP endpoints /v1/ai/image/create, /v1/ai/asr, /v1/ai/tts, /v1/ai/video, /v1/ai/decision, /v1/ai/embedding, /v1/ai/rerank, and /v1/ai/audio default to a 40 MiB body, a 60-second body read deadline, and a 180-second execution deadline. /v1/ai/stream, /v1/ai/chat/completions, and /v1/chat/completions use the same defaults. Lightweight /v1/ai/image/result uses ordinary limits: 1 MiB / 15 seconds / 30 seconds.
Body limits count actual Uint8Array.byteLength, including Base64 and JSON wrapping, and do not replace model or media file validation. Missing or falsely small Content-Length cannot bypass the limit. Invalid lengths return 400 INVALID_CONTENT_LENGTH; a Content-Encoding other than identity returns 415 UNSUPPORTED_CONTENT_ENCODING. Body timeout returns 408 REQUEST_BODY_TIMEOUT, and oversized bodies return 413 BODY_TOO_LARGE. The body deadline is total and does not reset on arriving chunks.
HTTP execution timing starts after the request body has been read. For non-SSE responses, the timer stays active until the response body finishes reading or is cancelled. Expiry before headers returns 504 REQUEST_TIMEOUT; after headers have been sent, it only ends the stream. The HTTP boundary waits up to 180 seconds for model and media response headers by default; the model policy can expire earlier. On receiving an SSE Response, the HTTP timer switches to the default 15-minute stream_lifetime_ms total deadline, still measured from the end of request body reading and including the header wait. This clock is independent of the model total deadline measured from provider execution; whichever expires first applies.
The HTTP boundary does not buffer SSE, change an HTTP status after headers, or synthesize generic SSE error frames. The model layer expresses model timeouts through the native or OpenAI protocol described above. Timeout, cancellation, or completion releases timers, listeners, and readers without waiting for an upstream cancel that could hang indefinitely.
Client cancellation corresponds to 499 REQUEST_CANCELLED; disconnected clients are not guaranteed to receive a response. Queued image tasks run independently and do not inherit cancellation from foreground creation or result queries. Cancelling a query does not cancel the image task. Database operations may continue; cancellation does not automatically roll back transactions or trigger retries. Provider Adapters should combine the request signal with their own deadlines; shorter upstream deadlines still apply.
For HTTP boundary configuration, see Federation runtime for configuration. Production deployment requires building the SDK and service artifacts containing the feature; running instances do not update automatically.
Call a decision model
A decision model takes raw state and typed questions, and returns answers with calibrated probabilities instead of generated text:
/** Call Jev through the current Federation user context. */
import { Federation } from "@downcity/federation";
const federation = new Federation({
url: "http://localhost:3000",
user_token: process.env.FEDERATION_USER_TOKEN,
});
const { answers } = await federation.user.ai.decision({
model: "jev-latest",
state: { type: "text", text: "The customer has failed to connect Stripe for three days." },
questions: [
{ id: "urgent", type: "boolean", instructions: "Does this message express urgency?" },
{ id: "route", type: "choice", instructions: "Which queue should handle this?", options: ["billing", "support"] },
{ id: "churn_risk", type: "score", instructions: "Churn risk", min: 1, max: 4 },
],
});
answers[0].value; // true
answers[0].probability; // 0.87
answers[1].distribution; // { billing: 0.72, support: 0.28 }The official edge-worker and local-node both provide B.AI's jev-latest. Local-node also provides TeamoRouter's teamorouter-jev-1.14.0. Both models support choice, score, and boolean decisions and are available in the Desktop Group dispatch model selector.
| Federation model ID | Available hosts | Upstream endpoint | Server credential |
|---|---|---|---|
jev-latest | edge-worker, local-node | https://api.b.ai/v1/decisions | B_API_KEY |
teamorouter-jev-1.14.0 | local-node | https://api.teamorouter.ai/v1/systemone | TEAMOROUTER_API_KEY |
Configure the corresponding credential in the Federation's Global / Env. Local-node also loads credentials from the project .env at startup. To call TeamoRouter, set model in the example above to teamorouter-jev-1.14.0; the upstream model ID is jev-1.14.0. Its context window is 64K tokens: state plus all questions must fit within 64K, and state plus the longest individual question must fit within 32K. The channel accepts text or JSON state and does not send image assets.
The model catalog exposes structured pricing with currency, billing units, rates, and applicable conditions.
Jev accepts 2–255 choice options. Score ranges must map to 2–10 levels (round(max - min) + 1), and boolean results use a 0.5 probability threshold. JSON state cannot be an empty object or array.
state is either { type: "text", text } or { type: "json", data }. Three question types are supported, and none of them produces free text:
| Type | Answer | Notes |
|---|---|---|
boolean | true / false | — |
choice | one of options | 1 to 255 unique options |
score | a number | requires min < max |
Question ids must be unique and every question must be answered exactly once. Invalid input fails with 422 before any usage is recorded, and a Channel response that does not cover every question is rejected too.
federation.user.ai.decision(input, signal) accepts an optional AbortSignal. Use question_id to associate each answer with its question. Group can bind this method as its decision client to select parallel responders each round; members keep their own execution models.
Decision models are not conversation models: they carry modalities: ["decision"] and cannot be passed to an Agent.