LiveKit Voice Agents

Build production voice AI on the LiveKit Agents framework (Python) — the open-source stack behind real-time, conversational agents that listen, think, and speak over WebRTC. This skill scaffolds and wires up AgentSession pipelines, turn detection, function tools, and SIP telephony so you can ship a phone-quality voice bot instead of fighting audio plumbing. It's for developers building voice assistants, IVR replacements, or hands-free agents who care about end-to-end latency under ~500ms.

When to use it

• You're standing up an STT → LLM → TTS pipeline and want sane defaults for interruptions and barge-in.
• You need an inbound or outbound phone agent over SIP (Twilio, Telnyx, or a LiveKit SIP trunk).
• You're switching between a cascaded pipeline and a speech-to-speech realtime model (OpenAI Realtime, Gemini Live) and want one session API for both.
• You want your agent to call tools — local Python functions or remote MCP servers — mid-conversation.

How it works

The core is AgentSession from livekit.agents, which orchestrates an Agent (your instructions + tools) against pluggable stt, llm, tts, and vad components from the plugin packages (livekit-plugins-deepgram, -openai, -cartesia, -silero, etc.). End-of-utterance is handled by the turn-detector plugin's transformer-based MultilingualModel, which predicts when a speaker is actually done rather than relying on raw silence timeouts. Tools are plain async functions decorated with @function_tool; MCP servers are attached so the LLM can call their tools natively. Telephony rides on LiveKit SIP — dispatch rules route inbound calls into a room where your worker picks them up. The entrypoint runs under cli.run_app(WorkerOptions(...)).

curl -fsSL --create-dirs -o ~/.claude/skills/livekit-voice-agents/SKILL.md https://claudeskil.com/skills/livekit-voice-agents/SKILL.md

from livekit.agents import Agent, AgentSession, function_tool, cli, WorkerOptions, JobContext
from livekit.plugins import deepgram, openai, cartesia, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel

@function_tool
async def get_weather(city: str) -> str:
    return f"It's sunny in {city}."

async def entrypoint(ctx: JobContext):
    session = AgentSession(
        stt=deepgram.STT(model="nova-3"),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=cartesia.TTS(),
        vad=silero.VAD.load(),
        turn_detection=MultilingualModel(),
    )
    await session.start(agent=Agent(instructions="You are a helpful voice assistant.", tools=[get_weather]), room=ctx.room)

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

Run it with python agent.py dev to connect a worker to your LiveKit instance.

Tips & gotchas

• Run python agent.py download-files once to cache the Silero VAD and turn-detector model weights, or first-call latency spikes.
• vad is still required even with the turn detector — the detector consumes VAD output; it doesn't replace it.
• Realtime models (openai.realtime.RealtimeModel) skip the STT/TTS slots entirely; don't pass them or you'll double-process audio.
• For outbound SIP, the call is placed via ctx.api.sip.create_sip_participant, not from inside the session.

FAQ

Cascaded pipeline or realtime model? Cascaded (STT-LLM-TTS) gives you per-component control, lower cost, and easier tool use; realtime gives the lowest latency and natural prosody. The AgentSession API is the same either way, so you can swap later.

How do I reduce latency? Pick streaming-capable STT/TTS, keep the LLM small or use a fast endpoint, and deploy workers in the same region as your LiveKit server. The turn detector cuts perceived latency more than shaving model time.

Does it work with my telephony provider? Yes — any provider that supports SIP trunking. You configure an inbound trunk and dispatch rule, and calls land in a room your worker serves.