Troubleshooting

Common debugging paths for City, client, token, and env.

Session model, storage, or delegation failures

Agent runtime logs are stored under its private data root at logs/YYYY-MM-DD.jsonl. City defaults to ~/.downcity/agents/<agent_id>/logs/. Start with the Agent, session_id, turn_id, and time window; the last console error alone does not establish the cause.

  1. Find model_request.request_failed and capture invocation_id, the model error code, and any available Provider request identifier.
  2. Find SessionExecutor execution failed and Session Turn finalization failed for the same Turn. exception.cause retains the original failure, while exception.secondary_error retains another failure during required cleanup. phase identifies abort_step, show_model_error, advance_context, or Turn finalization. When a storage error follows a model error, inspect both.
  3. Find Session delegation failed. Correlate delegation_id, parent_agent_id, parent_session_id, parent_turn_id, work_session_id, and work_turn_id. phase: prepare identifies preparation failures, execute execution failures, and delivery result delivery failures. record_retry, retry_scan, and cancel identify failure-record persistence, recovery scans, and cancellation. Some identity fields may be absent if the record could not be read or the work Turn has not been created.
  4. For storage failures, inspect [session.storage.worker] events for the same Agent. Fields include pool (reader / writer), slot, worker_id, event, and, when available, the semantic command, request_id, reason, duration_ms, exit_code, and allowlisted exception information. Worker request IDs correlate only within this process and their pool; they are separate from model request IDs.

Worker request_failed events describe command failures. reason distinguishes ipc_timeout, worker_error, unexpected_exit, post_message_failed, message_error, invalid_response, worker_create_failed, and command_failed. attached means a storage target used a particular Worker generation; terminating, terminated, and termination_failed describe retirement. Correlate requests and retirement using pool, slot, and worker_id. Durations are in milliseconds; exit codes appear only when known.

The shared storage pool forwards safe diagnostics through active storage leases to the Agent logger. Without a receiver it uses the console. Custom storage hosts must supply a diagnostic receiver and persist events themselves. Logging is best effort and cannot change transaction outcomes. Exception trees retain only allowlisted types, codes, SQLite numeric codes, and bounded cause chains. They exclude message bodies, SQL parameters, credentials, raw exception messages, and stacks; unknown types or codes may be omitted.

Tools execute only after the model response has been reliably committed. A failed rollback closes the current Writer and blocks unconfirmed projections, ending model recovery. Error presentation or required finalization failures retain the original failure and end the execution as failed. Lost write acknowledgments trigger receipt checks for the original operation, without automatically replaying dispatched transactions or tools. Inspect the delegation's current execution state, delivery state, and pending records before choosing the next action; a historical failure log does not mean retries are still running.

session_storage_read_failed alone does not establish database corruption or distinguish a timeout from a Worker crash. A read-only SQLite quick_check returning ok means that check found no structural problem; it does not explain a historical Worker failure. If the corresponding Worker events are unavailable, keep the cause unconfirmed.

client returns 401

Check:

  • whether the request includes Authorization: Bearer <user_token>
  • whether the token has expired
  • whether City has DOWNCITY_FEDERATION_TOKEN_SIGNING_KEY configured
  • whether the token was issued by the current Federation

client returns 403

Check:

client returns 404

Check:

  • whether the HTTP route is correct
  • whether the target service has been registered in City

model not found or handler not hit

Check:

  • whether the client-provided model has been registered through AIChannel.model() + AIService.use()
  • whether the target model includes the required action for the current pathway, such as text, stream, or image
  • whether the request included the intended model id
  • whether city.ai.catalog() can see the model

These cases usually return 422.

Federation returns a framework error

Use the response's server-generated x-request-id to correlate error.request_id and framework diagnostics. POST Actions require a JSON object by default: malformed JSON returns 400 INVALID_JSON, non-object JSON returns 400 INVALID_INPUT, and body read failures return 400 INVALID_BODY. Empty or whitespace-only input becomes {}.

An explicit integer statusCode in 400–599 preserves a code matching ^[A-Za-z][A-Za-z0-9_]{0,63}$, including 503 async_dispatch_unavailable. Without a valid status, the result is always 500 INTERNAL_ERROR. Explicit 4xx httpError messages are preserved. Every 5xx hides the original message: 503 is Service temporarily unavailable, 504 is Request timed out, and other 5xx statuses use Internal server error.

Framework 5xx logs contain only event, request_id, status, and code. They do not record bodies, tokens, original Errors, or stacks, and do not provide a complete access log. Do not rely on original stack retention. If you add an onError hook, keep its diagnostics controlled; a hook failure does not replace the original error.

Error Responses returned directly by business or native handlers retain their own bodies; the boundary still overwrites x-request-id. After a stream starts, boundary timeout or cancellation aborts downstream work, cancels the reader, and ends the stream without changing the HTTP status, returning JSON, or synthesizing generic SSE error frames. See HTTP API for the full contract.

HTTP body rejected or request timed out

Record x-request-id and error.request_id first. IDs are generated at the HTTP boundary before middleware, cover middleware short circuits and 413, 408, 504 responses, and are shared by the Router and AI relay. This response header contract applies to ordinary HTTP; HTTP 101 upgrades are neither rewrapped nor managed, and their x-request-id response header is not guaranteed.

Status and codeWhat to check
400 INVALID_CONTENT_LENGTHThe declared length must be a non-negative decimal integer; fix the caller's header
415 UNSUPPORTED_CONTENT_ENCODINGOmit Content-Encoding or use identity
413 BODY_TOO_LARGECount actual body bytes, including JSON / Base64 wrapping; missing or falsely small lengths cannot bypass the limit
408 REQUEST_BODY_TIMEOUTCheck upload speed and total body read time; sending chunks does not extend the deadline
504 REQUEST_TIMEOUTCheck middleware, handlers, and upstream time after the body finishes reading, including shorter Adapter deadlines
499 REQUEST_CANCELLEDCheck caller cancellation or disconnection; a disconnected client may not receive a response
504 provider_timeout / timeout_phase: "first_output"Default 120-second cumulative budget exhausted; check provider open and actual read waits before useful output
504 provider_timeout / timeout_phase: "idle"No new useful output within the default 120 seconds of cumulative read waits; metadata, empty deltas, and usage do not renew the budget
504 provider_timeout / timeout_phase: "total"Default 15-minute wall-clock deadline from provider execution exhausted, including consumer backpressure

Ordinary requests default to 1 MiB / 15 seconds to read / 30 seconds to execute. /v1/payment/webhook uses 2 MiB / 15 seconds / 30 seconds. /v1/ai/stream, /v1/ai/chat/completions, /v1/chat/completions, and media endpoints (image/create, asr, tts, embedding, rerank, video, audio, decision under /v1/ai/) use 40 MiB / 60 seconds / 180 seconds. /v1/ai/image/result uses ordinary limits.

For non-SSE HTTP responses, the timer runs from the end of request body reading until the response body finishes reading or is cancelled. Expiry before headers returns 504; after headers have been sent, it only ends the stream. Bodyless OPTIONS requests also use the ordinary deadline. The HTTP boundary waits up to 180 seconds for model and media response headers by default; the model policy can expire earlier. Synchronous JSON remains subject to the 180-second HTTP deadline. On receiving an SSE Response, the HTTP timer switches to the default 15-minute stream_lifetime_ms, measured from the end of request body reading and including the header wait, without buffering SSE. This clock is independent of the model total deadline measured from provider execution. Shorter provider deadlines still apply; whichever expires first applies. Once streaming starts, the HTTP boundary only aborts downstream work and cancels the reader to end the stream, without changing the status or synthesizing generic SSE error frames.

Native /v1/ai/stream, OpenAI SSE / JSON, and every Relay language model execution share server-side AIServiceOptions.model_stream. The 120-second first-output budget accumulates provider open and actual read waits; the 120-second idle budget accumulates actual read waits between useful outputs. Backpressure consumes neither budget, while the total wall clock keeps running. Clients cannot raise the policy through a request. model_start, text_start, reasoning_start, empty deltas, and usage do not renew the budget. Non-empty text_delta / reasoning_delta, tool_call_start, non-empty tool_call_delta.input_delta, and tool_call_finish count as useful output.

Model timeouts use provider_timeout / 504 and timeout_phase: connect (waiting for a response stream, not TCP connection), first_output, idle, or total. Only connect and first-output expiry are transient (retryable: true); Agent also checks partial output and its default 900-second recovery budget. Budget expiry uses timeout_phase: "recovery_budget", retryable: false, and Model recovery budget exhausted; other model timeouts use the safe message Model provider timed out. retry_at_ms preserves the upstream retry deadline; warning next_retry_at_ms identifies a scheduled attempt. Before response start, timeout returns 504; an already-started native stream emits model_error without changing the HTTP status. A framework model_start may precede a failure terminal without fabricating usage. OpenAI SSE sends an error frame without a success [DONE]; JSON aggregation returns 504.

The guard aborts the current attempt; Agent owns bounded recovery before partial output. Completion or cancellation releases timers, listeners, and readers without waiting for an upstream cancel that could hang indefinitely. Trustworthy usage facts are retained; missing usage does not lead to estimated tokens or costs. Existing bill callbacks use actual facts, and business charging rules do not promise zero charges in every case without usage. completion.outcome distinguishes succeeded, failed, and cancelled. A cancel after the terminal event does not relabel the result as cancellation or timeout, or duplicate settlement or charges. See Calling models for configuration and the full contract.

Cancellation is cooperative: database operations may continue, transactions are not automatically rolled back, and no automatic retry is triggered. Queued image tasks do not inherit cancellation from foreground creation or result queries. After a query is interrupted, use the existing task ID to check status rather than creating a duplicate task. Adapters should combine the request signal with their own deadlines; shorter upstream deadlines still apply.

Servers can partially override category limits through FederationOptions.http and use routes for exact-path overrides; see HTTP API. Edge / local-node link the SDK through workspace dependencies, so development builds use the default policy. Production deployment requires building the SDK and service artifacts containing the feature; running instances do not update automatically.

provider call failed

Check:

  • whether the matching key exists in City .env
  • whether the handler reads the correct input.env() key
  • whether model meta matches the current provider calling style

File tools take a long time to finish

“生成文件内容中” means the model is still preparing file content; writing has not started. The received character count shows generation progress. “等待执行” means the input is ready, and “写入文件中” means writing has started.

If the model reaches its output limit or the response lacks a normal finish marker, the tool is not executed. Reduce the scope of each change, edit long documents by section, and save large SVG illustrations as separate files. If another interruption occurs, check the model service before repeating the same oversized request.

For edit input validation errors, correct the original call: edits must be an array, not a JSON-encoded string. Partial changes do not require rewriting the entire file.

usage not recorded

Check:

  • whether AIService and CreditsService are mounted before UsageService and supplied as its readers
  • whether the AI provider returned final RuntimeMetering; executions without final metering only increase the execution count
  • whether the Credits transaction is an applied positive charge; top-ups, failed, and pending transactions are excluded
  • whether the query supplies real ordered from and to dates plus a valid IANA timezone