sound Plugin

sound is the unified speech capability plugin. It discovers models whose FED modalities include asr or tts, then performs the actual call through City AI Service.

sound does not download or run local speech models. It has no Python, Whisper, Piper, or other local inference dependency. A local audio file is only read and converted to a data URL before it is sent to FED.

Create the plugin

import { SoundPlugin } from "@downcity/plugins";

const sound = new SoundPlugin({
  list_models: async () => {
    const catalog = await city.ai.listModels();
    return catalog.all().filter((model) =>
      model.modalities.includes("asr") || model.modalities.includes("tts")
    );
  },
  asr: (input) => city.ai.asr(input),
  tts: (input) => city.ai.tts(input),
});

Actions

models

List speech-capable FED models:

plugin_call({
  plugin: "sound",
  action: "models",
  payload: { capability: "asr" },
});

capability can be asr or tts. Omit it to return models supporting either capability. Each model's modalities field states its exact capabilities.

asr

Transcribe audio with a FED ASR model:

plugin_call({
  plugin: "sound",
  action: "asr",
  payload: {
    model: "asr-model-id",
    audio_path: "./input.wav",
  },
});

Provide one audio source:

• audio_path: an absolute path or a path relative to the Agent project root
• url: a remote audio URL
• data_url: audio encoded as a data URL

The result uses the serializable core fields from AI SDK TranscriptionResult:

{
  text: string;
  segments?: Array<{
    text: string;
    startSecond: number;
    endSecond: number;
  }>;
  language?: string;
  durationInSeconds?: number;
}

tts

Generate speech with a FED TTS model:

plugin_call({
  plugin: "sound",
  action: "tts",
  payload: {
    model: "tts-model-id",
    text: "Welcome back",
    voice: "alloy",
    format: "mp3",
  },
});

TTS returns an AI SDK UIMessage. Audio is represented by a file part, which the Agent saves to project resources and attaches to the assistant message.

Default models

Configure separate ASR and TTS defaults when needed:

new SoundPlugin({
  default_asr_model: "asr-model-id",
  default_tts_model: "tts-model-id",
  list_models,
  asr,
  tts,
});

If neither the action nor the plugin provides a model, the call fails with guidance to read sound.models. The plugin never picks a model implicitly.

Automatic inbound transcription

auto_asr: true transcribes inbound voice and audio chat attachments. This path cannot ask the user to choose a model, so default_asr_model is required:

new SoundPlugin({
  auto_asr: true,
  default_asr_model: "asr-model-id",
  list_models,
  asr,
  tts,
});

The transcript is appended to user text as <voice src="...">...</voice>. A failed attachment transcription does not block the main chat flow.