Joshua Franklin Wiedeman

Phone Agent

A Raspberry Pi that answers an analog phone, turns the caller's speech into text, matches it against keyword rules, and plays recorded MP3 replies.

← AI lab

Design for a build. Not finished, and the code is not published.

How the parts connect

Phone handset (RJ11)
   │  analog line
   ▼
ATA with an FXS port (for example Grandstream HT801)
   │  SIP call + RTP audio over the local network
   ▼
Raspberry Pi
   ├─ SIP client (pjsua2): answer, hang up, send and receive audio
   ├─ Speech-to-text (Vosk or Whisper.cpp)
   ├─ Rule engine: match the transcript against rules.yaml
   │     ├─ match    → play an MP3, move to the next state
   │     └─ no match → play a fallback MP3, retry, then escalate
   └─ Audio player: send the MP3 audio back into the call

Parts

Phone interface

An ATA (analog telephone adapter) with an FXS port turns the analog phone into a SIP endpoint.

  • Detects off-hook and rings the phone
  • Detects keypad (DTMF) tones
  • Sends call audio to the Pi over SIP/RTP

Speech-to-text

Runs on the Pi, with no cloud service.

  • Vosk for streaming recognition
  • Optional custom vocabulary for expected words
  • Voice activity detection to find the end of a sentence

Rule engine

A state machine. Each state has keyword or regex triggers and an action.

  • Keyword, regex, and fuzzy matching
  • Tracks the current state of the call
  • Falls back and retries when nothing matches
  • Optional: pick at random between several recordings for the same reply

MP3 playback

Replies are pre-recorded, so they always sound the same.

  • Folder of MP3 files per topic
  • Play several files in a row
  • Normalize volume
  • Stop playback when the caller starts talking (barge-in)

Rules

Each rule pairs trigger words with an action. When the caller's words match a trigger, the action runs: play an MP3, move to another state, escalate, or hang up.

TRIGGER: "hello" | "hi" | "hey" | "good morning"
ACTION: play("greetings/welcome.mp3") → next("main_menu")
TRIGGER: "hours" | "open" | "time"
ACTION: play("info/hours.mp3") → next("anything_else")
TRIGGER: "person" | "human" | "representative"
ACTION: play("transfer/hold.mp3") → escalate()
TRIGGER: no match or timeout
ACTION: play("fallback/options.mp3") → retry up to 2 times

Rule file (YAML)

# rules.yaml - Phone Agent Configuration

states:
  greeting:
    triggers:
      - "hello"
      - "hi"
      - "hey"
      - regex: "good (morning|afternoon|evening)"
    action:
      play: "greetings/welcome.mp3"
      next: "main_menu"

  main_menu:
    triggers:
      - match: "hours|open|time"
        action:
          play: "info/hours.mp3"
          next: "anything_else"
      - match: "location|address|where"
        action:
          play: "info/location.mp3"
          next: "anything_else"
      - match: "person|human|representative"
        action:
          play: "transfer/hold.mp3"
          escalate: true
    fallback:
      play: "fallback/options.mp3"
      retry: 2

  anything_else:
    triggers:
      - match: "yes|yeah|more"
        action:
          next: "main_menu"
      - match: "no|done|bye"
        action:
          play: "greetings/goodbye.mp3"
          hangup: true

Hardware

  • Raspberry Pi 4 (4 GB or more) Assumed enough for Vosk small models; not yet tested
  • Grandstream HT801 ATA FXS port for the phone, SIP to the Pi
  • USB sound card Optional; only needed if audio is not routed over SIP
  • Analog phone Any standard RJ11 phone
  • microSD card (32 GB or more) OS and audio files
  • Power supplies 5 V 3 A for the Pi; the ATA uses its own adapter

Software

Runtime

  • Python 3.11+
  • asyncio
  • pjsua2 for SIP/RTP
  • PyAudio for local audio I/O

Speech

  • Vosk (offline speech-to-text)
  • pyttsx3 or espeak (text-to-speech fallback)
  • webrtcvad (voice activity detection)
  • soundfile (audio file reading)

Rule engine

  • PyYAML (rule file)
  • re (regex matching)
  • transitions (state machine)
  • rapidfuzz (fuzzy matching)

Optional

  • Ollama (local LLM when no rule matches)
  • Whisper.cpp (more accurate speech-to-text)
  • Piper (more natural text-to-speech)
  • SQLite (call log)

Open questions

  • Where does "escalate" send the call on a single home line? Options: ring another SIP phone, or play a message and hang up.
  • Is Vosk accurate enough on narrowband phone audio (8 kHz)?
  • How long is the delay between the caller stopping and the reply starting?

Planned file layout

phone-agent/
├── config/
│   ├── rules.yaml          # Conversation flow rules
│   └── settings.yaml       # System configuration
├── audio/
│   ├── greetings/          # Welcome/goodbye MP3s
│   ├── info/               # Information responses
│   ├── fallback/           # Error/retry prompts
│   └── transfer/           # Hold music, transfer notices
├── src/
│   ├── main.py             # Entry point
│   ├── sip_handler.py      # SIP/RTP management
│   ├── speech.py           # STT/TTS interface
│   ├── rules_engine.py     # State machine + keyword matching
│   ├── audio_player.py     # MP3 playback control
│   └── utils.py            # Helpers
├── tests/
├── requirements.txt
└── README.md