sound Power

sound is the unified speech capability power. It discovers models whose FED modalities include asr or tts, then performs the actual call through City AI Service.

sound does not download or run local speech models. It has no Python, Whisper, Piper, or other local inference dependency. A local audio file is only read and converted to a data URL before it is sent to FED.

Configuration

{
  default_asr_model: "asr-model-id",
  default_tts_model: "tts-model-id",
}

Actions

models

List speech-capable FED models:

ai({
  action: "sound.models",
  args: { capability: "asr" },
});

capability can be asr or tts. Omit it to return models supporting either capability. Each model's modalities field states its exact capabilities.

asr

Transcribe audio with a FED ASR model:

ai({
  action: "sound.asr",
  args: {
    model: "asr-model-id",
    audio_path: "./input.wav",
  },
});

Provide one audio source:

• audio_path: an absolute path or a path relative to the Agent project root
• url: a remote audio URL
• data_url: audio encoded as a data URL

The result uses the serializable core fields from AI SDK TranscriptionResult:

{
  text: string;
  segments?: Array<{
    text: string;
    startSecond: number;
    endSecond: number;
  }>;
  language?: string;
  durationInSeconds?: number;
}

tts

Generate speech with a FED TTS model:

ai({
  action: "sound.tts",
  args: {
    model: "tts-model-id",
    text: "Welcome back",
    voice: "alloy",
    format: "mp3",
  },
});

TTS returns an AI SDK UIMessage. Audio is represented by a file part, which the Agent saves to project resources and attaches to the assistant message.

Default models

Configure separate ASR and TTS defaults when needed:

{
  default_asr_model: "asr-model-id",
  default_tts_model: "tts-model-id",
}

If neither the action nor the power provides a model, the call fails with guidance to read sound.models. The power never picks a model implicitly.

Automatic inbound transcription

auto_asr: true transcribes inbound voice and audio chat attachments. This path cannot ask the user to choose a model, so default_asr_model is required:

{
  auto_asr: true,
  default_asr_model: "asr-model-id",
}

The transcript is appended to user text as <voice src="...">...</voice>. A failed attachment transcription does not block the main chat flow.

City projects Sound configuration into PowerContext.config for the current Agent. The AI service comes from PowerContext.city.embassy.user.ai; it does not belong to Agent.