Troubleshooting
Common debugging paths for City, client, token, and env.
Session model, storage, or delegation failures
Agent runtime logs are stored under its private data root at logs/YYYY-MM-DD.jsonl. City defaults to ~/.downcity/agents/<agent_id>/logs/. Start with the Agent, session_id, turn_id, and time window; the last console error alone does not establish the cause.
- Find
model_request.request_failedand captureinvocation_id, the model error code, and any available Provider request identifier. - Find
SessionExecutor execution failedandSession Turn finalization failedfor the same Turn.exception.causeretains the original failure, whileexception.secondary_errorretains another failure during required cleanup.phaseidentifiesabort_step,show_model_error,advance_context, or Turn finalization. When a storage error follows a model error, inspect both. - Find
Session delegation failed. Correlatedelegation_id,parent_agent_id,parent_session_id,parent_turn_id,work_session_id, andwork_turn_id.phase: prepareidentifies preparation failures,executeexecution failures, anddeliveryresult delivery failures.record_retry,retry_scan, andcancelidentify failure-record persistence, recovery scans, and cancellation. Some identity fields may be absent if the record could not be read or the work Turn has not been created. - For storage failures, inspect
[session.storage.worker]events for the same Agent. Fields includepool(reader/writer),slot,worker_id,event, and, when available, the semanticcommand,request_id,reason,duration_ms,exit_code, and allowlisted exception information. Worker request IDs correlate only within this process and their pool; they are separate from model request IDs.
Worker request_failed events describe command failures. reason distinguishes ipc_timeout, worker_error, unexpected_exit, post_message_failed, message_error, invalid_response, worker_create_failed, and command_failed. attached means a storage target used a particular Worker generation; terminating, terminated, and termination_failed describe retirement. Correlate requests and retirement using pool, slot, and worker_id. Durations are in milliseconds; exit codes appear only when known.
The shared storage pool forwards safe diagnostics through active storage leases to the Agent logger. Without a receiver it uses the console. Custom storage hosts must supply a diagnostic receiver and persist events themselves. Logging is best effort and cannot change transaction outcomes. Exception trees retain only allowlisted types, codes, SQLite numeric codes, and bounded cause chains. They exclude message bodies, SQL parameters, credentials, raw exception messages, and stacks; unknown types or codes may be omitted.
Tools execute only after the model response has been reliably committed. A failed rollback closes the current Writer and blocks unconfirmed projections, ending model recovery. Error presentation or required finalization failures retain the original failure and end the execution as failed. Lost write acknowledgments trigger receipt checks for the original operation, without automatically replaying dispatched transactions or tools. Inspect the delegation's current execution state, delivery state, and pending records before choosing the next action; a historical failure log does not mean retries are still running.
session_storage_read_failed alone does not establish database corruption or distinguish a timeout from a Worker crash. A read-only SQLite quick_check returning ok means that check found no structural problem; it does not explain a historical Worker failure. If the corresponding Worker events are unavailable, keep the cause unconfirmed.
client returns 401
Check:
- whether the request includes
Authorization: Bearer <user_token> - whether the token has expired
- whether City has
DOWNCITY_FEDERATION_TOKEN_SIGNING_KEYconfigured - whether the token was issued by the current Federation
client returns 403
Check:
client returns 404
Check:
- whether the HTTP route is correct
- whether the target service has been registered in City
model not found or handler not hit
Check:
- whether the client-provided
modelhas been registered throughAIChannel.model()+AIService.use() - whether the target model includes the required action for the current pathway, such as
text,stream, orimage - whether the request included the intended model id
- whether
city.ai.catalog()can see the model
These cases usually return 422.
Federation returns a framework error
Use the response's server-generated x-request-id to correlate error.request_id and framework diagnostics. POST Actions require a JSON object by default: malformed JSON returns 400 INVALID_JSON, non-object JSON returns 400 INVALID_INPUT, and body read failures return 400 INVALID_BODY. Empty or whitespace-only input becomes {}.
An explicit integer statusCode in 400–599 preserves a code matching ^[A-Za-z][A-Za-z0-9_]{0,63}$, including 503 async_dispatch_unavailable. Without a valid status, the result is always 500 INTERNAL_ERROR. Explicit 4xx httpError messages are preserved. Every 5xx hides the original message: 503 is Service temporarily unavailable, 504 is Request timed out, and other 5xx statuses use Internal server error.
Framework 5xx logs contain only event, request_id, status, and code. They do not record bodies, tokens, original Errors, or stacks, and do not provide a complete access log. Do not rely on original stack retention. If you add an onError hook, keep its diagnostics controlled; a hook failure does not replace the original error.
Error Responses returned directly by business or native handlers retain their own bodies; the boundary still overwrites x-request-id. After a stream starts, boundary timeout or cancellation aborts downstream work, cancels the reader, and ends the stream without changing the HTTP status, returning JSON, or synthesizing generic SSE error frames. See HTTP API for the full contract.
HTTP body rejected or request timed out
Record x-request-id and error.request_id first. IDs are generated at the HTTP boundary before middleware, cover middleware short circuits and 413, 408, 504 responses, and are shared by the Router and AI relay. This response header contract applies to ordinary HTTP; HTTP 101 upgrades are neither rewrapped nor managed, and their x-request-id response header is not guaranteed.
| Status and code | What to check |
|---|---|
400 INVALID_CONTENT_LENGTH | The declared length must be a non-negative decimal integer; fix the caller's header |
415 UNSUPPORTED_CONTENT_ENCODING | Omit Content-Encoding or use identity |
413 BODY_TOO_LARGE | Count actual body bytes, including JSON / Base64 wrapping; missing or falsely small lengths cannot bypass the limit |
408 REQUEST_BODY_TIMEOUT | Check upload speed and total body read time; sending chunks does not extend the deadline |
504 REQUEST_TIMEOUT | Check middleware, handlers, and upstream time after the body finishes reading, including shorter Adapter deadlines |
499 REQUEST_CANCELLED | Check caller cancellation or disconnection; a disconnected client may not receive a response |
504 provider_timeout / timeout_phase: "first_output" | Default 120-second cumulative budget exhausted; check provider open and actual read waits before useful output |
504 provider_timeout / timeout_phase: "idle" | No new useful output within the default 120 seconds of cumulative read waits; metadata, empty deltas, and usage do not renew the budget |
504 provider_timeout / timeout_phase: "total" | Default 15-minute wall-clock deadline from provider execution exhausted, including consumer backpressure |
Ordinary requests default to 1 MiB / 15 seconds to read / 30 seconds to execute. /v1/payment/webhook uses 2 MiB / 15 seconds / 30 seconds. /v1/ai/stream, /v1/ai/chat/completions, /v1/chat/completions, and media endpoints (image/create, asr, tts, embedding, rerank, video, audio, decision under /v1/ai/) use 40 MiB / 60 seconds / 180 seconds. /v1/ai/image/result uses ordinary limits.
For non-SSE HTTP responses, the timer runs from the end of request body reading until the response body finishes reading or is cancelled. Expiry before headers returns 504; after headers have been sent, it only ends the stream. Bodyless OPTIONS requests also use the ordinary deadline. The HTTP boundary waits up to 180 seconds for model and media response headers by default; the model policy can expire earlier. Synchronous JSON remains subject to the 180-second HTTP deadline. On receiving an SSE Response, the HTTP timer switches to the default 15-minute stream_lifetime_ms, measured from the end of request body reading and including the header wait, without buffering SSE. This clock is independent of the model total deadline measured from provider execution. Shorter provider deadlines still apply; whichever expires first applies. Once streaming starts, the HTTP boundary only aborts downstream work and cancels the reader to end the stream, without changing the status or synthesizing generic SSE error frames.
Native /v1/ai/stream, OpenAI SSE / JSON, and every Relay language model execution share server-side AIServiceOptions.model_stream. The 120-second first-output budget accumulates provider open and actual read waits; the 120-second idle budget accumulates actual read waits between useful outputs. Backpressure consumes neither budget, while the total wall clock keeps running. Clients cannot raise the policy through a request. model_start, text_start, reasoning_start, empty deltas, and usage do not renew the budget. Non-empty text_delta / reasoning_delta, tool_call_start, non-empty tool_call_delta.input_delta, and tool_call_finish count as useful output.
Model timeouts use provider_timeout / 504 and timeout_phase: connect (waiting for a response stream, not TCP connection), first_output, idle, or total. Only connect and first-output expiry are transient (retryable: true); Agent also checks partial output and its default 900-second recovery budget. Budget expiry uses timeout_phase: "recovery_budget", retryable: false, and Model recovery budget exhausted; other model timeouts use the safe message Model provider timed out. retry_at_ms preserves the upstream retry deadline; warning next_retry_at_ms identifies a scheduled attempt. Before response start, timeout returns 504; an already-started native stream emits model_error without changing the HTTP status. A framework model_start may precede a failure terminal without fabricating usage. OpenAI SSE sends an error frame without a success [DONE]; JSON aggregation returns 504.
The guard aborts the current attempt; Agent owns bounded recovery before partial output. Completion or cancellation releases timers, listeners, and readers without waiting for an upstream cancel that could hang indefinitely. Trustworthy usage facts are retained; missing usage does not lead to estimated tokens or costs. Existing bill callbacks use actual facts, and business charging rules do not promise zero charges in every case without usage. completion.outcome distinguishes succeeded, failed, and cancelled. A cancel after the terminal event does not relabel the result as cancellation or timeout, or duplicate settlement or charges. See Calling models for configuration and the full contract.
Cancellation is cooperative: database operations may continue, transactions are not automatically rolled back, and no automatic retry is triggered. Queued image tasks do not inherit cancellation from foreground creation or result queries. After a query is interrupted, use the existing task ID to check status rather than creating a duplicate task. Adapters should combine the request signal with their own deadlines; shorter upstream deadlines still apply.
Servers can partially override category limits through FederationOptions.http and use routes for exact-path overrides; see HTTP API. Edge / local-node link the SDK through workspace dependencies, so development builds use the default policy. Production deployment requires building the SDK and service artifacts containing the feature; running instances do not update automatically.
provider call failed
Check:
- whether the matching key exists in City
.env - whether the handler reads the correct
input.env()key - whether model
metamatches the current provider calling style
File tools take a long time to finish
“生成文件内容中” means the model is still preparing file content; writing has not started. The received character count shows generation progress. “等待执行” means the input is ready, and “写入文件中” means writing has started.
If the model reaches its output limit or the response lacks a normal finish marker, the tool is not executed. Reduce the scope of each change, edit long documents by section, and save large SVG illustrations as separate files. If another interruption occurs, check the model service before repeating the same oversized request.
For edit input validation errors, correct the original call: edits must be an array, not a JSON-encoded string. Partial changes do not require rewriting the entire file.
usage not recorded
Check:
- whether
AIServiceandCreditsServiceare mounted beforeUsageServiceand supplied as its readers - whether the AI provider returned final
RuntimeMetering; executions without final metering only increase the execution count - whether the Credits transaction is an applied positive charge; top-ups, failed, and pending transactions are excluded
- whether the query supplies real ordered
fromandtodates plus a valid IANAtimezone