CallinAI Voice Channel

CallinAI (a telephony platform; dev environment at https://api.dev.callinaibeta.com/v1, Swagger at /swagger) is the first phone channel of Tools Anatomy. It lets a customer point a phone number at a governed flow: the caller speaks, CallinAI transcribes, the channel gateway runs the flow (setup tools, prompt, tool loop) through the Runtime Gateway, and the reply is spoken back. It lives inside apps/channel-gateway next to the Web Widget and reuses its patterns (encrypted transcripts, contact identity contract, metadata-only events, black-box tests).

Mental model

The channel is turn-based: one CallinAI gather verb (speech input) per turn, the flow's reply nested as the gather's say prompt (barge-in enabled), so the caller can interrupt. Silence twice in a row ends the call politely; a runtime failure apologises and hangs up; a disabled agent/connection answers "not available" and hangs up.

Concepts

Term	Meaning	Where
Connection	One linked CallinAI account (account_sid + API key)	callinai_connections
Voice agent	A CallinAI application (created by us) wired to a flow, with greeting + speech settings	callinai_agents
Call	One phone call handled by an agent — the voice analogue of a widget session	callinai_calls
Transcript	Encrypted caller/assistant lines	callinai_messages

Connect flow (operator)

1. Operator pastes display name, account SID and an API key (CallinAI portal → Account → API Keys) in /channels → Connect CallinAI account.
2. POST /v1/channels/callinai/connections validates the pair with GET /Accounts/{sid} (401/403 → callinai_invalid_credentials, 404 → callinai_account_not_found), fetches GET /Accounts/{sid}/WebhookSecret, stores the API key in the secret broker with the operator's OWN bearer (POST /v1/secrets, name callinai/<connection_id>/api_key) and persists only the returned secret_ref.
3. The webhook signing secret is stored AES-256-GCM encrypted (CHANNEL_GATEWAY_MESSAGE_ENCRYPTION_KEY, AAD bound to tenant + connection) so inbound webhooks can be verified without a tenant bearer.
4. Verify re-runs steps 2–3 (picks up a regenerated webhook secret).

The API key is never in a channel row, a log, an event, or a browser response. It is resolved from the broker (POST /v1/secrets/resolve, consumer channel-gateway, operator's bearer) only for management operations.

Create a voice agent

POST /v1/channels/callinai/agents {connection_id, flow_id, name, greeting, conversation_mode, speech_synthesis_vendor, speech_synthesis_voice, speech_recognizer_vendor, speech_recognizer_language, …}:

• Conversation engine (conversation_mode) — how the caller talks to the flow; see "How a call runs the flow" below. turn (default) uses the CallinAI speech settings; realtime additionally takes realtime_vendor (OpenAI Realtime today), realtime_model, realtime_voice and a write-only realtime_api_key; stream takes stream_vendor (Deepgram today) and a write-only stream_api_key. Vendor keys go straight to your Secret Manager and are never shown again.
• flow ownership is enforced before any write;
• a hook key hk_<256-bit> is generated; the three webhook URLs are {CHANNEL_GATEWAY_PUBLIC_BASE_URL}/v1/public/callinai/hooks/{hook_key}/{call|turn|status};
• POST /Applications is called on CallinAI with call_hook = /call, call_status_hook = /status, and the speech settings; the returned application_sid is stored. If the local insert fails afterwards the application is deleted best-effort.
• Speech pickers: GET .../connections/{id}/speech-options lists the vendors configured on the account (GET /Accounts/{sid}/SpeechCredentials, secret- free) and, with ?vendor=, that vendor's TTS languages + voices, STT languages and TTS models straight from GET /Accounts/{sid}/SpeechCredentials/speech/supportedLanguagesAndVoices — so the console offers exactly what the account can use.
• Existing applications: GET .../connections/{id}/applications lists the account's applications (with the agent attached to each, if any); create an agent with application_sid to attach an existing application instead of creating one — its call_hook/call_status_hook/ messaging_hook are overwritten with ours, its name and (unless you chose others) speech settings are kept.
• Deep links: agents/connections/applications carry platform_url when CHANNEL_GATEWAY_CALLINAI_PORTAL_APPLICATION_URL / CHANNEL_GATEWAY_CALLINAI_PORTAL_ACCOUNT_URL templates ({sid}) are set.
• Numbers: GET .../connections/{id}/phone-numbers lists the account's numbers (annotated with the agent each is routed to); POST .../agents/{id}/phone-numbers / .../phone-numbers/unassign set application_sid on the number via PUT /PhoneNumbers/{sid}.
• Speech providers: GET .../connections/{id}/speech-credentials lists vendor + tts/stt flags only (provider keys are never decoded).
• Test: POST .../agents/{id}/test-call {to} originates an outbound call (POST /Accounts/{sid}/Calls) from a number routed to the agent; when the callee answers, CallinAI runs the agent's application so the operator hears the real flow (the call hook arrives with direction=outbound; the person is then to).

SMS

Toggle SMS on a voice agent (create dialog or drawer). The gateway sets the application's messaging_hook to …/hooks/{hook_key}/message; inbound SMS to any number routed to that application arrives as CallinAI's inbound-SMS payload (messageSid, accountSid, applicationSid, from, to, text, media, carrier — feature-server lib/http-routes/api/messaging.js), signed with the same webhook secret. The text runs the agent's flow with X-Channel: callinai_sms and the reply is returned inline as a CallinAI message verb ({verb:"message", from, to, text}), which CallinAI delivers over the account's SMPP gateway — so no API key is needed on the hot path, but the account must have SMS routing configured (ask CallinAI). SMS threads are stored as calls of kind sms keyed by the (from, to) pair, with the same encrypted transcript, redelivery dedupe and console views; media-only messages get a polite "text only" reply.

Webhooks (public, signature-verified)

All three routes are POST /v1/public/callinai/hooks/{hook_key}/…, no platform key. Every request is authenticated BEFORE any side effect:

1. hook key → agent → connection (tenant derived from the row);
2. A signature header (t=<unix>,v1=<hex>) on every webhook, verified as HMAC-SHA256(webhook_secret, "<t>.<raw body>") with a replay tolerance (CHANNEL_GATEWAY_CALLINAI_SIGNATURE_TOLERANCE_SECONDS, default 300) — exactly how the CallinAI platform signs its webhooks;
3. the payload's accountSid must match the connection.

Failures: 404 unknown hook, 401 bad/missing/stale signature, 403 account mismatch, and a metadata-only ai.channel.callinai.webhook.rejected event. Webhooks have their own per-hook-key rate budget (CHANNEL_GATEWAY_CALLINAI_WEBHOOK_RATE_LIMIT_PER_MINUTE, default 600) — never the widget's per-IP limiter, because every webhook comes from the platform's egress IP.

Route	CallinAI payload used	Response
/call (call_hook)	callSid, direction, from, to, accountSid	[gather{say: greeting, input: [speech], actionHook: /turn, bargein}]
/turn (gather actionHook)	reason (speechDetected / timeout / dtmfDetected), speech.alternatives[0].transcript	[gather{say: reply}], or [say, hangup] on 2nd silence / runtime failure
/status (call_status_hook)	callStatus, duration	204
/message (messaging_hook)	messageSid, from, to, text, media	[message{from, to, text}] (empty array when SMS is off)

Idempotency: /call is keyed on call_sid (a redelivery returns the same verbs, resolves no second contact and records no second greeting); /turn replays the stored reply when the identical utterance was stored within 15 s and already answered (CallinAI retries action hooks on transport failures).

Identity we derive

• Context Vault identity channel: phone (shared by every phone adapter so one E.164 number is one durable contact across CallinAI / SMS / WhatsApp).
• external_user_id = E.164 (+16172375089); phone_number written runtime_only ONLY when the value is a real phone (SIP usernames / anonymous callers keep the raw value as external_user_id and no phone); channel = callinai.
• Chat headers: X-Channel: callinai, X-Channel-Account-ID: <dialled E.164>, X-Contact-ID: <vault contact UUID> (so Tool Runtime's UUID check is satisfied — the same approach the widget uses), X-Conversation-ID: <per-call conversation id>. Setup bindings can therefore filter on callinai and read {{conversation_context.channel_account_id}} ("which restaurant's number was dialled").

Console

/channels → CallinAI voice: connections table (connect / verify / disable), voice agents table (create), agent drawer (settings, enable / disable, webhook URLs, phone numbers assign/unassign, test call, calls list, transcript). BFF routes live under /api/bff/channels/callinai/*; see docs/product-console-channels.md.

How a call runs the flow: three conversation engines

Every agent chooses one of three engines. The constraint that shaped this: not every LLM provider has a realtime speech API, so the default and the streaming engine work with any chat-completions provider.

Turn-based (turn, the default). CallinAI does the audio work (STT/TTS); every turn is one gather → transcript → governed flow → reply spoken as the next gather prompt (barge-in on). Works with every model; full transcripts, response shaping, and audit; ~2–5 s per turn.

Realtime speech-to-speech (realtime). The call runs CallinAI's llm verb: the caller talks directly to a realtime voice model (OpenAI Realtime today) for sub-second responses. Your governance still applies where it matters — the instructions are the flow's rendered conversation-start prompt, the model's tools are the flow's bound tools, and every tool call comes back through the gateway's governed tool runtime. Honest trade-off: the conversation itself lives at the vendor, so per-turn transcripts and response shaping are not recorded in this mode. Needs a vendor API key with realtime access (stored in your Secret Manager).

Streaming pipeline (stream). CallinAI's listen verb streams the call audio to the gateway over a WebSocket; Deepgram streams speech-to-text continuously, your flow's model (any provider) answers, and Deepgram text-to-speech plays the reply back — with barge-in (the caller can interrupt and the reply stops). Same governed chat path and encrypted transcripts as turn mode, lower perceived latency. Needs a Deepgram API key.

Twilio's equivalent of the streaming engine is ConversationRelay (docs/channels/twilio.md) — a follow-up there.

Limitations / next

• Realtime engine: one vendor so far (OpenAI Realtime); per-turn transcripts are not recorded in that mode (the conversation lives at the vendor).
• Streaming engine: one speech vendor so far (Deepgram); replies are spoken once the model's answer is complete (token-by-token speech needs runtime stream:true, a follow-up).
• No DTMF menus, no call transfer / human handoff yet (Milestone D6).
• Usage metering: chat completions are metered by the Runtime Gateway; a dedicated voice_calls / voice_minutes meter is a follow-up in the usage registry (D1).
• Static greeting; an AI-generated opening line is a follow-up.

Troubleshooting

• "Verification failed: Unexpected token '<' … is not valid JSON" — the browser received Cloudflare's HTML error page instead of the BFF's JSON. Cloudflare replaces origin 502/504 bodies, so the console's BFF now reports upstream failures as 503 (original status in x-bff-upstream-status) and the gateway uses 503/422 for CallinAI / secret-broker failures. Root causes to check: the host secret-broker binary is older than the channel-gateway consumer allowlist (go build -o /usr/local/bin/anatomy-secret-broker ./apps/secret-broker/cmd/server && systemctl restart anatomy-secret-broker), or the CallinAI API is unreachable from the container.