Operations and Debugging

The boundary between fed, city, Federation Admin, and direct database inspection, and when the CLI is the better tool.

The CLI and Federation Admin can both help you debug Downcity, but they fit different work: fed/downfed manages Federation, city/downcity manages local Agents and local Federation connections, and Federation Admin is for trusted programmatic management.

The simplest rule first

If your thought is:

  • "I need to see the facts right now"

start with the CLI.

If your thought is:

  • "this action should live in the system long term"

move toward Federation Admin or a trusted script.

CLI, Federation Admin, and direct DB inspection each fit different work

Better fits for the CLI

  • fast local inspection
  • checking whether a service actually wrote the expected tables
  • inspecting models, schema, and rows
  • one-off debugging without writing code first

Better fits for Federation Admin

  • backend-owned actions
  • automated token issuance
  • formal management workflows
  • internal admin surfaces

Better fits for direct database inspection

  • when the issue has clearly reached the lower database layer
  • when the CLI already narrowed the problem but you need deeper SQL-level facts

Common CLI debugging actions

city
fed
city federation status
city agent list

Local Fed Logs and Startup Diagnostics

fed status --id fed_x
fed logs --id fed_x

Check the system manager, health identity, and runner failure in status first. logs displays up to the last 200 lines of the current ~/.downcity/federation/logs/<fed_id>/service.log only. An empty or shorter current file is not supplemented from archives; read archives manually. Log access is limited to CLI-managed local instances and does not retrieve remote or Worker logs.

Application stdout and stderr pass through runner pipes with backpressure and serialized writes to service.log, without an unbounded queue. The limits are fixed defaults and require no additional configuration:

FileLimit per fileArchivesTotal file budget per Fed
service.log20 MiB5; .1 newest, .5 oldestAbout 120 MiB, including the current file
supervisor.log2 MiB3; .1 newest, .3 oldestAbout 8 MiB, including the current file

Rotation splits by actual bytes, including oversized chunks, so each file stays within its limit. File boundaries do not guarantee complete UTF-8 characters or lines. Rotation does not restart the service. If application or supervisor logging fails, the runner stops the application and records failed in ~/.downcity/federation/deployments/<fed_id>/service/receipt.json. The runner failure field in fed status reports the corresponding failure_code:

failure_codeWhat to inspect
APPLICATION_LOG_FAILEDApplication log paths, permissions, disk space, and per-file limits
SUPERVISOR_LOG_FAILEDSupervisor log paths, permissions, disk space, and per-file limits
RUNNER_FAILEDRunner startup, runtime configuration, and system service diagnostics

The runner writes supervisor.log in ~/.downcity/federation/deployments/<fed_id>/service/. It records fixed, safe diagnostic events/codes without bodies, tokens, keys, or full environment variables. macOS launchd stdout/stderr point to /dev/null and do not hold an unbounded log file directly. Linux writes the same fixed events to this file; the system journal separately retains runtime startup information. Journal quotas and rotation belong to the operating system, and the CLI does not change global journal configuration.

If Node fails before entering the runner, supervisor.log and the receipt may have no record of that failure. On macOS, inspect the user service with launchctl. On Linux, inspect the corresponding unit's state and startup information with systemctl --user and journalctl --user.

Current log and archive paths used by the writer must be regular files. Symlinks, directories, and other invalid targets are rejected; links are not followed. Existing files above their per-file limit cause an error and are not truncated. The user must explicitly archive or clean them up first. fed clean only cleans releases and does not delete logs.

Application output is not guaranteed to be automatically redacted. Application developers must not log tokens, API keys, or complete request/response bodies. A running instance must load the CLI and runner containing this feature; a documentation update does not apply it to running instances.

Local SQLite backup and restore

fed backup --id fed_x
fed stop --id fed_x
fed restore --id fed_x --path /absolute/path/to/backup_id --dry-run
fed restore --id fed_x --path /absolute/path/to/backup_id

Online backup uses SQLite's backup API, including committed WAL data, without stopping the application. Restore requires an explicitly stopped service that cannot automatically recover. It saves the current database before preview and confirmation; cancellation does not replace the database, and completion leaves the service stopped. Use --yes for explicit non-interactive replacement authorization. Backups can contain sessions, user data, and credentials; files use 0600 and must not enter Git or a release.

See SQLite Backup and Restore for the full procedure, version compatibility, and failure boundaries.

CLI and Desktop City ownership

CLI and Desktop share a data root. A root lease is acquired before configuration databases, Agent assembly, migration or resumed work. Resource commands verify the current host's instance, root and management protocol, then use its local channel. Desktop management is available independently of the City Remote HTTP switch. Resource commands do not stop or take over another host.

Without a host, configuration and pure historical browsing are available. coordination_required requires an explicit supported-host start; migration_required requires explicit writable recovery. A host that is starting, unreachable or incompatible blocks operations. revision_conflict requires a fresh read; partial_failure reports committed facts and must be checked before retrying. Agent defaults apply to new Sessions, preserving existing snapshots and overrides. See Resource Management for JSON and exit codes.

User system dashboard

Open fed, choose a saved Federation, then select User system dashboard in the Admin menu.

The CLI reads admin data from the installed official services and computes the dashboard locally in the terminal.

The dashboard covers:

  • accounts: registered users, new users, active sessions
  • usage: active users and service-call events
  • payment: paying users, first-paid users, paid orders, real revenue by currency
  • balance: credited credits and current balance credits

If a service is not installed, the dashboard marks that module as unavailable and still shows data from the other installed services.

Revenue uses successful payment records and their real amount_minor values. Credits are shown separately as balance and recharge units.

A common debugging chain

  1. confirm Federation is connected to the database you think it is
  2. confirm the expected models and tables exist
  3. confirm the service actually wrote data
  4. only then suspect deeper business logic

Example: usage seems missing

A good first reaction is not "change code" but "confirm facts":

  • is the usage service enabled?
  • was the usage table created?
  • is this really the database instance you expect?
  • were any events written at all?

Example: payment completed but access is still denied

Check these facts first:

  • did the webhook reach Federation?
  • was the webhook event persisted?
  • did the entitlement become active?

Common misunderstandings

The CLI is not just a one-time development tool

If you operate many products, services, or model configurations, it remains a frequent operations entry.

Federation Admin is not just a more advanced CLI

They are different modes:

  • Federation Admin: trusted programmatic management
  • CLI: human terminal inspection

Direct SQL should not always be your first instinct

In many cases the CLI is already enough to narrow the problem sharply.