# Building an AI Voice Receptionist: Twilio, ElevenLabs, n8n

*Published 2025-11-25 · Dmitry Shirokov · shirokoff.ca/blog/ai-voice-receptionist · as of late 2025*

Two people, a phone ringing ~40 times a week, about a third going to voicemail — during a job, after five, on a Saturday — and someone who wants a Saturday-morning appointment calls the next place on the list before you ring back Monday. They didn't want a call centre; they wanted the phone answered. Solvable in a couple of weeks by one person, and full of small traps that turn a great demo into a system nobody trusts — most of them in the phone number, the calendar, and the fourth turn of a conversation the demo never reached.

## The job is six outcomes

**Book** (real availability, never a guess) · **reschedule/cancel** (find a booking from a caller who doesn't know its ID) · **answer a factual question** from a maintained source, verbatim where it matters · **take a message** (name, callback, reason) · **escalate** (defined number, defined conditions, context carried) · **say it doesn't know**, cleanly, then do one of the above. Everything else routes to message or escalation. **Narrowing the job to those six and declining the rest gracefully is the biggest quality lever in the build** — an agent trying to be helpful outside its remit invents policy.

## Choose a layer, not a vendor

**Managed agent platform** (ElevenLabs Agents, Vapi, Retell) — you write a prompt, tool definitions and the tools. Almost always right for a receptionist: telephony, turn-taking and latency are handled, and they're genuinely hard to reproduce. **Voice framework** (LiveKit Agents, Pipecat) — when you need control the platform won't give. **Direct realtime API** — when you're building the platform, not the receptionist.

I used ElevenLabs Agents: voice quality is a real differentiator on a phone line, native Twilio integration and SIP trunking make telephony configuration rather than code, and tool-calling maps onto the six outcomes. Vapi and Retell solve the same shape; decide with an afternoon of side-by-side calls, not a spreadsheet.

## Getting a real Canadian number

**Buy from a telephony provider, not the agent platform**, if you ever want to move. Local area code, not toll-free — a familiar area code gets answered. Canada specifics: **a verified local address** is required and P.O. boxes and virtual addresses are rejected, so have the real address and proof ready. **Caller ID authentication is the carrier's job** — Canadian carriers have implemented STIR/SHAKEN since November 2021, so use a reputable provider, keep registration accurate, don't spoof outbound. **Keeping an existing number**: port it (clean, slow, scheduled cutover) or point the line at the agent over **SIP trunking** — the low-risk path, and how I'd start: forward after hours first, port later if ever. **A VoIP number is not a 911 line** — say so in writing. **Outbound is a different legal regime**: inbound answering is straightforward, but placing calls puts you in CRTC unsolicited-telecommunications territory.

Two disclosure defaults: **say it's an AI in the first sentence** (callers are fine with it when it works; discovering it later is what makes people angry), and **if you record or transcribe, say so** — a call transcript is personal information. One greeting sentence covers both.

## Prototyping the logic in n8n

Not because low-code is easier: **in the first two weeks the business logic changes almost daily** — buffers, bookable service types, statutory holidays, longer first appointments — and each is a five-minute workflow change versus a deploy cycle, with the client watching it change.

Tool contract, per webhook. Two details separate working from mostly-working: **return a speakable `say` string per slot** so the model never translates an ISO timestamp into speech (it gets it right most of the time, and the failures are "fourteen o'clock"), and use an **opaque slot `id`** so booking passes back exactly what availability produced — no window for a hallucinated time to reach the calendar.

**The confirmation rule is non-negotiable**: repeat name, service, day and time, and get a yes before booking. One turn, and it eliminates the class of failure where a misheard date becomes a real appointment a real person unpicks. `book_appointment` then: re-check the slot is still free, create the event, log a row, send SMS + email, return a short confirmation string. If the re-check fails, return the failure honestly with two alternatives rather than an error the model improvises around.

## Google Calendar: four traps

**Timezones** — compute in `America/Toronto`, never a fixed offset, and never let the model do the arithmetic; DST produces a wrong appointment exactly twice a year, on a weekend, looking like a hallucination. **Free/busy isn't the whole answer** — it knows events, not your business hours, lunch, or buffers; generate candidate slots from rules, *then* subtract busy time, or you'll offer 7:40pm on a Sunday. **The double-booking race** — re-check at booking time, and hold a tentative event during the conversation if the stakes justify it. **Auth** — a service account scoped to the specific calendar, not an employee's OAuth token that dies when they change their password.

## Testing, in three layers

**1. Simulated conversations in CI** — platforms can run an LLM-driven simulated caller end to end and evaluate whether a defined outcome was reached (ElevenLabs exposes this as a simulation API with evaluation criteria). One scenario per outcome plus one per known failure, run on every prompt or tool change. Catches logic regressions: stopped offering to take a message, started booking without confirming.

**2. Real calls from a real phone** — the skipped one, because it isn't automatable. **The PSTN audio path is nothing like your laptop mic.** Matrix: speakerphone in a moving car · spell an unusual surname (the most common accuracy failure) · a phone number said quickly once · "next Tuesday" · interrupt mid-sentence · two accents including a non-native speaker · ten seconds of silence · an insistent out-of-scope question · ask for a human immediately · hang up mid-booking (no orphaned events).

**3. Production review daily at first** — post-call webhook delivering transcript, outcome and extracted fields; read every call for a week, then sample. Every failure becomes a simulation case. Two metrics from day one: **containment** (resolved without escalation) and **booking accuracy** (appointments a human didn't fix).

## Lessons that cost me

**Latency is a product feature and the tool call is where it goes** — a turn feels natural under ~1.5s, and a cold workflow plus a Calendar round trip eats all of it; have the agent say "let me check that for you" while the tool runs, and pre-compute the next few days of availability. **Don't let the model format times.** **Test on the phone network early** — I lost two days polishing a prompt against codec artefacts. **Scope creep is the real risk** — payments, detailed pricing, complaints all widen the surface where it can be confidently wrong. **Ship after hours first**: 5pm and weekends only, humans keep business hours, and let the client hear a week of real recordings. That sequencing did more for adoption than any technical decision here.

## Cost

Three stacked per-minute costs — telephony, the voice platform (ASR + LLM + TTS bundled), and the automation layer — landing for a small business nearer a modest subscription than a salary, dominated by the platform rate. The comparison that matters isn't a receptionist's wage; it's **the bookings currently going to voicemail**.

*Related: [Golden eval datasets from production](https://shirokoff.ca/blog/golden-eval-datasets-from-production) · [Agent frameworks compared](https://shirokoff.ca/blog/agent-frameworks-compared) · [Sandboxing an AI agent](https://shirokoff.ca/blog/agent-sandboxing-tool-isolation) · [The LLM gateway](https://shirokoff.ca/blog/llm-gateway-routing-fallback) · [PII and tokenization](https://shirokoff.ca/blog/pii-tokenization-privacy-analytics)*