---
name: livekit-voice-agents
description: "Build sub-500ms voice AI on LiveKit Agents — STT-LLM-TTS pipelines, turn detection, telephony, MCP tools. Production LiveKit Agents (Python): AgentSession with STT-LLM-TTS VoicePipelineAgent or realtime models, the turn-detector plugin for end-of-utterance and adaptive interruption, function_tool definitions and native MCP server tool calls, SIP telephony for inbound/outbound phone agents, and latency tuning (preemptive generation, VAD, streaming TTS) toward sub-500ms targets. Covers plugins (Deepgram, Cartesia, OpenAI, ElevenLabs), worker/JobContext deployment, and metrics."
license: MIT
---

# LiveKit Voice Agents

Build production voice AI on the LiveKit Agents framework (Python) — the open-source stack behind real-time, conversational agents that listen, think, and speak over WebRTC. This skill scaffolds and wires up `AgentSession` pipelines, turn detection, function tools, and SIP telephony so you can ship a phone-quality voice bot instead of fighting audio plumbing. It's for developers building voice assistants, IVR replacements, or hands-free agents who care about end-to-end latency under ~500ms.

## When to use it

- You're standing up an STT → LLM → TTS pipeline and want sane defaults for interruptions and barge-in.
- You need an inbound or outbound phone agent over SIP (Twilio, Telnyx, or a LiveKit SIP trunk).
- You're switching between a cascaded pipeline and a speech-to-speech realtime model (OpenAI Realtime, Gemini Live) and want one session API for both.
- You want your agent to call tools — local Python functions or remote MCP servers — mid-conversation.

## How it works

The core is `AgentSession` from `livekit.agents`, which orchestrates an `Agent` (your instructions + tools) against pluggable `stt`, `llm`, `tts`, and `vad` components from the plugin packages (`livekit-plugins-deepgram`, `-openai`, `-cartesia`, `-silero`, etc.). End-of-utterance is handled by the `turn-detector` plugin's transformer-based `MultilingualModel`, which predicts when a speaker is actually done rather than relying on raw silence timeouts. Tools are plain async functions decorated with `@function_tool`; MCP servers are attached so the LLM can call their tools natively. Telephony rides on LiveKit SIP — dispatch rules route inbound calls into a room where your worker picks them up. The entrypoint runs under `cli.run_app(WorkerOptions(...))`.

```python
from livekit.agents import Agent, AgentSession, function_tool, cli, WorkerOptions, JobContext
from livekit.plugins import deepgram, openai, cartesia, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel

@function_tool
async def get_weather(city: str) -> str:
    return f"It's sunny in {city}."

async def entrypoint(ctx: JobContext):
    session = AgentSession(
        stt=deepgram.STT(model="nova-3"),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=cartesia.TTS(),
        vad=silero.VAD.load(),
        turn_detection=MultilingualModel(),
    )
    await session.start(agent=Agent(instructions="You are a helpful voice assistant.", tools=[get_weather]), room=ctx.room)

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
```

Run it with `python agent.py dev` to connect a worker to your LiveKit instance.

## Tips & gotchas

- Run `python agent.py download-files` once to cache the Silero VAD and turn-detector model weights, or first-call latency spikes.
- `vad` is still required even with the turn detector — the detector consumes VAD output; it doesn't replace it.
- Realtime models (`openai.realtime.RealtimeModel`) skip the STT/TTS slots entirely; don't pass them or you'll double-process audio.
- For outbound SIP, the call is placed via `ctx.api.sip.create_sip_participant`, not from inside the session.

## FAQ

**Cascaded pipeline or realtime model?** Cascaded (STT-LLM-TTS) gives you per-component control, lower cost, and easier tool use; realtime gives the lowest latency and natural prosody. The `AgentSession` API is the same either way, so you can swap later.

**How do I reduce latency?** Pick streaming-capable STT/TTS, keep the LLM small or use a fast endpoint, and deploy workers in the same region as your LiveKit server. The turn detector cuts perceived latency more than shaving model time.

**Does it work with my telephony provider?** Yes — any provider that supports SIP trunking. You configure an inbound trunk and dispatch rule, and calls land in a room your worker serves.