Development Tools

LiveKit Voice AI Phone Agent: SIP, Interruptions, Human Handoff, and Call Logs

Build a LiveKit Voice AI phone agent with AgentSession, SIP trunks, inbound dispatch, interruption handling, DTMF, human handoff, call logs, testing, and compliance controls.

LiveKit Agents can put speech recognition, LLM, speech synthesis and tool invocation into a real-time session, and then connect to the ordinary telephone network via SIP. Google Trends’ voice ai agents has increased by 350% in the United States. However, the failure of the phone robot to go online is usually not because the model is not smart enough, but because the SIP routing, interruption, human handoff, and log links are not verified together. This article uses inbound customer service calls as an example. First, the agent in the room is routed, then the phone number is answered, and finally the failure path that the production call must have is handled.

What components does a phone call go through?

The caller dials the phone number. The phone carrier or LiveKit Phone Numbers routes the call into the SIP trunk. The Inbound Trunk determines which numbers and sources can come in. Dispatch Rule puts the SIP participant into the LiveKit room. Agent worker accepts the job and creates AgentSession. STT converts the audio to text, LLM decides the answer or calls the tool, and TTS sends the answer back to the phone. When transferring to a human, the system creates a new SIP participant or hands the call to an external number. Each layer has independent logs and failure status.

Verify the Agent in a browser room first

in the browser room Telephone networks add carrier, number, and SIP configuration variables. First verify the voice agent using LiveKit Playground or a local audio room. It should be able to complete:

  • Hear the user’s voice.
  • Answer within a reasonable delay.
  • Stop the current TTS when the user interrupts.
  • Call a side-effect-free tool.
  • Resources are released when the session ends.

If this step fails, don’t check the phone number first.

Create project and local environment

Prepare a LiveKit Cloud project, or self-host LiveKit Server as per the official documentation. Local Python environment:

1
2
3

python3 -m venv .venv-livekit
source .venv-livekit/bin/activate
python -m pip install --upgrade pip

Windows PowerShell:

1
2
3

python -m venv .venv-livekit
.\.venv-livekit\Scripts\Activate.ps1
python -m pip install --upgrade pip

Install Agents and required plugins as per the current quickstart, without pinning out-of-date versions from old blogs.

Environment variables are only placed on the server side 

1
2
3

LIVEKIT_URL=wss://example.livekit.cloud
LIVEKIT_API_KEY=replace-me
LIVEKIT_API_SECRET=replace-me

STT, LLM and TTS providers will also have their own keys. Add .env.local to the ignore list:

1

git check-ignore .env.local

API secrets cannot be entered into browsers or mobile devices. The client only obtains the short-term room token issued by the server.

AgentSession is the coordinator of the voice pipeline

AgentSession is responsible for user input, models, tools, outputs and events. The specific plug-in name and parameters change with the version. The structure is shown below instead of committing to a fixed supplier:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

from livekit.agents import Agent, AgentSession

class SupportAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions=(
                "You are a concise phone support agent. "
                "Confirm identity before reading private account data. "
                "Offer a human handoff when the caller asks."
            )
        )

async def entrypoint(ctx):
    session = AgentSession(
        stt=build_stt(),
        llm=build_llm(),
        tts=build_tts(),
    )
    await session.start(
        room=ctx.room,
        agent=SupportAgent(),
    )

Functions such as build_stt() need to be replaced with officially supported plug-in instances.

Phone prompts should be adapted to screen-less scenarios

Links, tables, and code are invisible to users. Keep your answers short and only ask one question at a time. Numbers, dates and emails need to be repeated for confirmation. Don’t read ten menu items aloud at once. There should be clear instructions for transferring to manual, repeating, returning to the previous step and ending the call. Authentication is performed before account operations are involved, rather than automatic trust based on caller numbers.

Assign read and write permissions to tools

Check business hours is a read-only tool. Inquiry orders may read personal data. Cancellation of orders, refunds and address changes are write operations. The writing tool requires a second confirmation and repeats the final parameters to the user.

1
2
3
4

async def cancel_order(order_id: str, confirmed: bool) -> str:
    if not confirmed:
        return "confirmation_required"
    return await order_service.cancel(order_id)

Don’t let LLM forge confirmed=True on its own based on natural language. Confirmation status should be managed by session logic.

The ability to interrupt determines the phone experience

When the user starts speaking, the Agent should stop or fade out the current TTS. Oversensitivity will interpret coughs, background TV, and echoes as interruptions. Excessive sluggishness prevents users from correcting errors. Record during debugging:

  • User voice start time.
  • VAD detection time.
  • TTS stop time.
  • STT final result time.
  • New answer first audio time.

Only an end-to-end timeline can tell where the delay comes from.

Turn Detection and Phone Noise

Phone audio bandwidth, compression, and noise are different than web microphones. A brief pause does not necessarily mean the user has finished speaking. There may be natural pauses between the address, order number, and name. Use the turn detection and VAD configuration currently provided by LiveKit to test from real phone recordings. Don’t just use your browser microphone to adjust parameters in a quiet office.

Provision or connect a phone number

You can use LiveKit Phone Numbers, or configure a supported third-party SIP provider. Confirm country, region, text messaging capabilities, emergency calling restrictions and identity requirements when selecting a number. The successful purchase of the number does not mean that the inbound routing has been completed. Trunk and dispatch rules are also needed to send calls to the room. The operator console and LiveKit console each maintain a verifiable ID.

Inbound Trunk limits the allowed numbers

Inbound Trunk defines which phone numbers and SIP sources are received. Use different numbers or trunks for production and testing. Do not accept all SIP sources and then rely on Agent rejection. If the operator supports IP or credential authentication, enable it according to official recommendations. After modifying the trunk, make a test call to confirm that the new configuration is hit instead of the old rule cache.

Dispatch Rule determines room naming and Agent

Generate a unique room for each call to prevent two unknown calls from entering the same conversation. The room name can contain the internal call ID, but not the full phone number. Only the fields required for routing are placed in metadata. Agent worker selects customer service, sales or reservation Agent based on job metadata. When there is no matching Agent, there must be an audible failure prompt or transfer path, and cannot remain silent.

Minimum inbound-call acceptance test

Dial from a real phone that’s not on the company Wi-Fi. Record:

1
2
3
4
5

carrier_call_id
sip_participant_id
room_name
agent_job_id
session_id

There should be at least a greeting or waiting prompt within 3–5 seconds of acknowledging an incoming call. If the call is connected but there is no sound, check the SIP participant’s track, whether the Agent has joined, and whether the TTS has been released.

403 and 404 Don’t just look at the Agent log

SIP layer 403 may come from authentication, source restrictions, number not verified, or carrier policy. 404 It may be that the number routing or trunk does not match. The LiveKit room has been created but does not have an Agent. Focus on dispatch and worker availability. The Agent has been started but there is no caller participant. Focus on SIP access. Save the call ID on both the operator and LiveKit sides.

DTMF as an alternate input for speech recognition

Phone keypads are suitable for confirming menus, numeric options and sensitive operations. For example, press 1 to switch to manual, press 2 to repeat, and press 9 to end. Don’t let DTMF and voice commands trigger the same action twice. Assign an event ID to each input and use a state machine to deduplicate it. Sensitive data such as bank cards and passwords should not be recorded by ordinary logs in complete keystroke sequences.

Human handoff does not mean asking the Agent to say “Transferred”

True handoff includes at least destination queues, context digests, and call routing. LiveKit Agent handoff is suitable for switching roles between different AI Agents. Switching to live customer service usually requires a new SIP outbound participant or external call center integration. Play a prompt during the queue and allow the user to cancel. If the transfer fails, return to the original Agent or provide a callback instead of hanging up directly.

The context passed to the human should be minimized

Can contain:

  • Verified customer ID.
  • User’s current goal.
  • Read-only query executed.
  • Unfinished operations.
  • Whether the user agrees to the recording.

Don’t give customer service your entire model thought process, irrelevant history, or all of your call text. The summary should mark what is explicitly stated by the user and what is inferred by the agent.

Outbound calls require stricter authorization

Outbound SIP can be used for callbacks, reminders, and surveys. But it’s also prone to triggering spam calls, time zones and consent issues. The number list must come from a business system with legal basis. Check local time, unsubscribe status, and frequency cap before calling. It is prohibited to let LLM choose any number to dial. Each outbound call requires an explicit business task ID.

Separate management of call recording and transcription

Recordings, live transcriptions, summaries, and business fields are different data objects. The user’s consent to the recording does not necessarily mean that the user agrees to the long-term preservation of the complete transcription. Set access permissions and retention periods separately. The recording URL uses a short-term signature and is not put into ordinary application logs. Delete requests override object storage, indexing, summarization, and backup policies.

Event log does not record original key

Recommended record:

1
2
3
4
5
6
7
8

{
  "session_id": "sess_example",
  "call_id": "call_example",
  "event": "human_handoff_requested",
  "timestamp": "2026-07-28T12:00:00Z",
  "target_queue": "support_cn",
  "result": "queued"
}

Phone numbers only retain masks or internal IDs. STT text does not enter the infrastructure debug log by default.

Delay budget needs to be measured in segments

The total response delay can be broken down into:

1

VAD + final STT + LLM first token + TTS first audio + network

Record P50, P95 and failure rate separately. The average will mask occasional ten-second pauses. Play a short wait prompt during the LLM tool invocation, but do not repeat it every second. Caching pinned greetings can reduce first TTS latency.

How to locate echo and dual voice

First make sure the caller only subscribes to one Agent audio track. Check whether the Agent joins the same room repeatedly. If the operator’s side recording is normal but the mobile phone hears echo, it may be the terminal’s acoustic echo. If the room recording has double sound, check the track publication and TTS for repeated sending. Don’t mask repetitive tracks by increasing the VAD threshold.

Maintain a recoverable experience when a Worker is disconnected

SIP calls may still be connected after an Agent worker crashes. Set job timeouts, worker health checks, and backup routes. A short recovery time allows new workers to read session state. If recovery is not possible, the playback failure will be explained and a manual number will be provided. Don’t make users wait indefinitely on silent lines.

Load testing requires automation, not manual dialing

from the same mobile phone number. Generate concurrency using official testing tools or controlled SIP test accounts. Gradually increase concurrency, observing room, worker, STT, LLM, TTS and operator quotas. Limits may vary per provider. The stress test number is isolated from the production number. Notify the operator in advance to avoid being judged as an abnormal call.

Calculate cost across the full call chain

Cost includes phone number, SIP minutes, LiveKit, STT, LLM, TTS, recording storage and transfer labor. Tool calls may also incur business API costs. Record the actual seconds and provider usage of each call. Failed and waiting calls are also billed. Calculating unit costs by successfully resolved calls makes more sense than just looking at price per minute.

Fault-injection drill checklist

Stop the Agent worker and confirm audible degradation. Make the STT key invalid and confirm that you will not repeatedly answer the wrong questions to the user. Let LLM time out and confirm that it can be transferred to manual work. Let TTS fail and confirm that the system will not remain on mute. Make the manual queue unavailable and provide callbacks or messages for confirmation. Press DTMF twice to confirm that the write operation will not be repeated.

Production readiness criteria

  • Agents in the browser room have passed independently.
  • SIP trunk and dispatch ID can be traced.
  • Use a separate room for each call.
  • User interruption can stop TTS.
  • Separation of permissions for reading and writing tools.
  • Turn to manual to complete the real phone routing.
  • Recordings, transcriptions and summaries are managed separately.
  • Carriers and LiveKit call IDs can be associated.
  • Disconnections, timeouts, and quota failures all cause voice degradation.
  • There are consent, time zone and frequency controls for outbound calls.

The quality of phone Voice AI depends on the complete call chain, not the performance of a single round of answers. First clear the browser room, and then add SIP, DTMF, human handoff and recording layer by layer, so that each failure can have a clear responsibility position.

LiveKit documentation