Operations and Debugging
The boundary between fed, city, Federation Admin, and direct database inspection, and when the CLI is the better tool.
The CLI and Federation Admin can both help you debug Downcity, but they fit different work: fed/downfed manages Federation, city/downcity manages local Agents and local Federation connections, and Federation Admin is for trusted programmatic management.
The simplest rule first
If your thought is:
- "I need to see the facts right now"
start with the CLI.
If your thought is:
- "this action should live in the system long term"
move toward Federation Admin or a trusted script.
CLI, Federation Admin, and direct DB inspection each fit different work
Better fits for the CLI
- fast local inspection
- checking whether a service actually wrote the expected tables
- inspecting models, schema, and rows
- one-off debugging without writing code first
Better fits for Federation Admin
- backend-owned actions
- automated token issuance
- formal management workflows
- internal admin surfaces
Better fits for direct database inspection
- when the issue has clearly reached the lower database layer
- when the CLI already narrowed the problem but you need deeper SQL-level facts
Common CLI debugging actions
city
fed
city federation status
city agent listLocal Fed Logs and Startup Diagnostics
fed status --id fed_x
fed logs --id fed_xCheck the system manager, health identity, and runner failure in status first. logs displays up to the last 200 lines of the current ~/.downcity/federation/logs/<fed_id>/service.log only. An empty or shorter current file is not supplemented from archives; read archives manually. Log access is limited to CLI-managed local instances and does not retrieve remote or Worker logs.
Application stdout and stderr pass through runner pipes with backpressure and serialized writes to service.log, without an unbounded queue. The limits are fixed defaults and require no additional configuration:
| File | Limit per file | Archives | Total file budget per Fed |
|---|---|---|---|
service.log | 20 MiB | 5; .1 newest, .5 oldest | About 120 MiB, including the current file |
supervisor.log | 2 MiB | 3; .1 newest, .3 oldest | About 8 MiB, including the current file |
Rotation splits by actual bytes, including oversized chunks, so each file stays within its limit. File boundaries do not guarantee complete UTF-8 characters or lines. Rotation does not restart the service. If application or supervisor logging fails, the runner stops the application and records failed in ~/.downcity/federation/deployments/<fed_id>/service/receipt.json. The runner failure field in fed status reports the corresponding failure_code:
| failure_code | What to inspect |
|---|---|
APPLICATION_LOG_FAILED | Application log paths, permissions, disk space, and per-file limits |
SUPERVISOR_LOG_FAILED | Supervisor log paths, permissions, disk space, and per-file limits |
RUNNER_FAILED | Runner startup, runtime configuration, and system service diagnostics |
The runner writes supervisor.log in ~/.downcity/federation/deployments/<fed_id>/service/. It records fixed, safe diagnostic events/codes without bodies, tokens, keys, or full environment variables. macOS launchd stdout/stderr point to /dev/null and do not hold an unbounded log file directly. Linux writes the same fixed events to this file; the system journal separately retains runtime startup information. Journal quotas and rotation belong to the operating system, and the CLI does not change global journal configuration.
If Node fails before entering the runner, supervisor.log and the receipt may have no record of that failure. On macOS, inspect the user service with launchctl. On Linux, inspect the corresponding unit's state and startup information with systemctl --user and journalctl --user.
Current log and archive paths used by the writer must be regular files. Symlinks, directories, and other invalid targets are rejected; links are not followed. Existing files above their per-file limit cause an error and are not truncated. The user must explicitly archive or clean them up first. fed clean only cleans releases and does not delete logs.
Application output is not guaranteed to be automatically redacted. Application developers must not log tokens, API keys, or complete request/response bodies. A running instance must load the CLI and runner containing this feature; a documentation update does not apply it to running instances.
Local SQLite backup and restore
fed backup --id fed_x
fed stop --id fed_x
fed restore --id fed_x --path /absolute/path/to/backup_id --dry-run
fed restore --id fed_x --path /absolute/path/to/backup_idOnline backup uses SQLite's backup API, including committed WAL data, without stopping the application. Restore requires an explicitly stopped service that cannot automatically recover. It saves the current database before preview and confirmation; cancellation does not replace the database, and completion leaves the service stopped. Use --yes for explicit non-interactive replacement authorization. Backups can contain sessions, user data, and credentials; files use 0600 and must not enter Git or a release.
See SQLite Backup and Restore for the full procedure, version compatibility, and failure boundaries.
CLI and Desktop City ownership
CLI and Desktop share a data root. A root lease is acquired before configuration databases, Agent assembly, migration or resumed work. Resource commands verify the current host's instance, root and management protocol, then use its local channel. Desktop management is available independently of the City Remote HTTP switch. Resource commands do not stop or take over another host.
Without a host, configuration and pure historical browsing are available. coordination_required requires an explicit supported-host start; migration_required requires explicit writable recovery. A host that is starting, unreachable or incompatible blocks operations. revision_conflict requires a fresh read; partial_failure reports committed facts and must be checked before retrying. Agent defaults apply to new Sessions, preserving existing snapshots and overrides. See Resource Management for JSON and exit codes.
User system dashboard
Open fed, choose a saved Federation, then select User system dashboard in the Admin menu.
The CLI reads admin data from the installed official services and computes the dashboard locally in the terminal.
The dashboard covers:
accounts: registered users, new users, active sessionsusage: active users and service-call eventspayment: paying users, first-paid users, paid orders, real revenue by currencybalance: credited credits and current balance credits
If a service is not installed, the dashboard marks that module as unavailable and still shows data from the other installed services.
Revenue uses successful payment records and their real amount_minor values. Credits are shown separately as balance and recharge units.
A common debugging chain
- confirm Federation is connected to the database you think it is
- confirm the expected models and tables exist
- confirm the service actually wrote data
- only then suspect deeper business logic
Example: usage seems missing
A good first reaction is not "change code" but "confirm facts":
- is the usage service enabled?
- was the usage table created?
- is this really the database instance you expect?
- were any events written at all?
Example: payment completed but access is still denied
Check these facts first:
- did the webhook reach Federation?
- was the webhook event persisted?
- did the entitlement become active?
Common misunderstandings
The CLI is not just a one-time development tool
If you operate many products, services, or model configurations, it remains a frequent operations entry.
Federation Admin is not just a more advanced CLI
They are different modes:
Federation Admin: trusted programmatic management- CLI: human terminal inspection
Direct SQL should not always be your first instinct
In many cases the CLI is already enough to narrow the problem sharply.
Read next
- For the basic
fed/cityentries, read CLI Quickstart - For model and table inspection, read Models and Tables
- For the data layer behind those facts, read Store and Table