Voice agents that hear tone: three architectures
A cascaded voice agent turns speech into text before the language model sees it, so sarcasm, hesitation and frustration are gone by the time it replies. A speech-to-speech model keeps them but is harder to inspect. A third design keeps the cascade and sends the model the transcript together with scores for how each phrase sounded.
Updated
The three designs
| Cascaded | Speech-to-speech | Cascade with tone | |
|---|---|---|---|
| How it works | Transcribe, reason over text, synthesize | One model takes audio in and produces audio out | Transcribe and score the audio, reason over both, synthesize |
| What the model knows about delivery | Nothing | Whatever it learned to use; not exposed | Named scores for each phrase |
| Can you log why it responded that way | Yes, as text | Only the audio | Yes, as text and scores |
| Swap the language model or voice | Yes | No | Yes |
| Where Oruk fits | Transcription | Not used | Transcription and tone |
Where the tone goes
Take a caller who says “Fine. Just cancel it then.” A transcript records six words. It does not record that the first one was clipped and the rest came out flat and fast.
A text-only agent reads agreement and confirms the cancellation. A person on the same call would hear the frustration and ask what went wrong.
The information was in the audio. The cascade dropped it at the first step, and no later step can get it back.
Passing tone to the model
Oruk Realtime emits a transcript event and, separately, an emotion event for each phrase. Join them by phrase ID, then put both in the turn you send to the language model.
Each user turn includes "voice": scores from 0 to 1 for how the phrase sounded.
They describe delivery, not what the person feels, and can be wrong.
If the words and the voice disagree, ask before acting on either.{
"role": "user",
"content": "Fine. Just cancel it then.",
"voice": { "frustrated": 0.81, "angry": 0.42, "calm": 0.06 }
}Illustrative values.
What this does not do
- It does not make the agent speak with emotion. That is the job of your speech synthesis.
- It does not decide when to interrupt or stop talking. Turn-taking stays in your framework.
- It does not read minds. A score is a description of sound.
Questions
- What does prosody mean here?
- The parts of speech that are not the words: pitch, loudness, pace, pauses and voice quality. Listeners use them to tell a sincere “great” from a sarcastic one.
- Does adding tone scores slow the agent down?
- The transcript does not wait for them. Oruk Realtime sends words as they are recognised and the phrase’s emotion scores in a later event. You choose whether the language model waits for that event or replies on the text alone.
- Is this the same as sentiment analysis?
- No. Sentiment analysis reads the transcript. These scores come from the audio, so identical words spoken two ways get different scores.
- How much should the agent trust a score?
- As a hint about delivery. Scores describe how speech sounds, vary by language and audio quality, and do not establish what a person feels. Do not use one as the sole basis for a decision about someone.
Try it on your own audio.
The demo needs no account. The trial needs a card.