Project 02 / Voice AI · Audio pipeline
Voice Tone Analysis
Reads call recordings for tone, background noise, audio quality, overlap and silence — measuring six of the nine answers straight from the waveform, so only the two that need meaning touch a language model.
Private repository — happy to walk you through it
- Year
- 2026
- Role
- Solo — pipeline, dashboard, deploy
- Type
- Audio analysis pipeline
- Status
- Deployed
- Fields measured, not guessed
- 6 / 9
- Pure signal processing
- Acoustic accuracy
- 0.97
- Mean over 43 synthetic clips
- Per audio minute
- $0.0017
- Priced from measured tokens
- Realtime factor
- 0.34×
- A minute of audio in ~20s
01
The problem
Call audio needs two kinds of answer. Is there background noise, is the line clipped, did someone go quiet for ten seconds? That's physics. Is the customer frustrated, and how much? That's meaning. Put both into one model and you get the classic failure: a loud caller read as angry, a bad line read as a noisy room.
So I treated the nine output fields as three problems — six measured from the signal, two judged by a language model from a speaker-attributed transcript, one calibrated confidence — and kept the layers from seeing each other's evidence. Loudness can't leak into tone because the tone model never sees it.
02
How it works
- 01
Decode
ffmpeg to 16 kHz float32, stereo preserved.
- 02
Voice activity
Silero VAD splits speech from non-speech — the timeline every later stage reads.
- 03
Acoustics
Segmental SNR for noise, in-band spectral shape for its type, clipping and bandwidth for quality, long-and-quiet gaps for silence.
- 04
Transcribe & diarise
faster-whisper decodes only the speech regions, detecting language per segment; pyannote and TitaNet attribute every line to a speaker.
- 05
Tone
A language model judges tone and intensity from the diarised transcript plus each speaker's prosody — never from raw loudness.
- 06
Calibrate
Confidence starts at a 0.95 ceiling and loses named deductions: thin speech, a noisy line, words that disagree with delivery.
03
Key decisions
Calculate whatever can be calculated
Noise, quality, overlap and silence are deterministic DSP: same audio in, same number out, at effectively zero cost. Only tone needs a model.
Ruled outOne multimodal model for all nine fields
Measure noise two ways
TV and static are intermittent, so the median background is just room tone. Taking the worse of the p50 and p95 levels took detection of those cases from 1 of 3 to 3 of 3.
Ruled outA median-only noise floor
Diarise, then measure prosody per speaker
An angry caller was averaging out against a calm agent. Giving each voice its own delivery line removed every polar inversion in tone.
Keep Whisper away from silence
Whisper medium hallucinated "I'm here." 42 times into an 18-second gap. Decoding only VAD speech regions removed the loops at the source.
No PyTorch
Every model runs on onnxruntime or CTranslate2 — a third of the image size, and no gated model downloads.
Make degradation loud
A pipeline that degrades instead of failing produces output shaped like success. Degraded runs are marked, and cost is priced from the API's own token counts, never an assumed zero.
Delete what's worse than nothing
An echo detector was built, measured firing on two of three known-clean calls, and removed. It's documented, not shipped.
04
In the code
{
"background_noise_present": true,
"background_noise_type": "static or hiss",
"background_noise_severity": "medium",
"audio_quality": "slightly_impaired",
"speaker_overlap_present": false,
"long_silence_present": false,
"emotional_tone": "frustrated",
"emotional_intensity": "medium",
"confidence": 0.78
}05
What I'd do next
Valence, not just arousal. Prosody says how raised a voice is, not whether it's delighted or furious. Closing that gap needs a speech-emotion model that's commercially licensed and works on telephone audio — every candidate I measured failed one of the two.
More labelled calls. Three labelled calls can't validate tone, and I refused to tune thresholds on them. A real labelled set is the first thing more data buys.
Cache the prompt for short clips. Every clip pays the full system prompt, so clips under ~40 seconds cost more than the budget. Padding the prompt past the caching minimum would cut input cost by about 90%.
06
Stack
- Audio
- ffmpegSilero VADnumpy DSP
- Speech
- faster-whisperpyannote segmentationTitaNet (sherpa-onnx)
- Language
- Claude Haiku 4.5Ollama (fully local mode)
- App
- FastAPIJinja2SQLite
- Ship
- DockerRailway