Skip to content
orukSign up
Speech-to-textVoice agentsModelsPricingResearchAPI docsQuickstartSDKsMCP serverIntegrationsBenchmarksStatusLog in

Voice agents that hear tone: three architectures

A cascaded voice agent turns speech into text before the language model sees it, so sarcasm, hesitation and frustration are gone by the time it replies. A speech-to-speech model keeps them but is harder to inspect. A third design keeps the cascade and sends the model the transcript together with scores for how each phrase sounded.

Updated

The three designs

Three voice agent architectures compared
CascadedSpeech-to-speechCascade with tone
How it worksTranscribe, reason over text, synthesizeOne model takes audio in and produces audio outTranscribe and score the audio, reason over both, synthesize
What the model knows about deliveryNothingWhatever it learned to use; not exposedNamed scores for each phrase
Can you log why it responded that wayYes, as textOnly the audioYes, as text and scores
Swap the language model or voiceYesNoYes
Where Oruk fitsTranscriptionNot usedTranscription and tone

Where the tone goes

Take a caller who says “Fine. Just cancel it then.” A transcript records six words. It does not record that the first one was clipped and the rest came out flat and fast.

A text-only agent reads agreement and confirms the cancellation. A person on the same call would hear the frustration and ask what went wrong.

The information was in the audio. The cascade dropped it at the first step, and no later step can get it back.

Passing tone to the model

Implementation guide

Oruk Realtime emits a transcript event and, separately, an emotion event for each phrase. Join them by phrase ID, then put both in the turn you send to the language model.

System prompt addition
Each user turn includes "voice": scores from 0 to 1 for how the phrase sounded.
They describe delivery, not what the person feels, and can be wrong.
If the words and the voice disagree, ask before acting on either.
One user turn
{
  "role": "user",
  "content": "Fine. Just cancel it then.",
  "voice": { "frustrated": 0.81, "angry": 0.42, "calm": 0.06 }
}

Illustrative values.

What this does not do

  • It does not make the agent speak with emotion. That is the job of your speech synthesis.
  • It does not decide when to interrupt or stop talking. Turn-taking stays in your framework.
  • It does not read minds. A score is a description of sound.

Questions

What does prosody mean here?
The parts of speech that are not the words: pitch, loudness, pace, pauses and voice quality. Listeners use them to tell a sincere “great” from a sarcastic one.
Does adding tone scores slow the agent down?
The transcript does not wait for them. Oruk Realtime sends words as they are recognised and the phrase’s emotion scores in a later event. You choose whether the language model waits for that event or replies on the text alone.
Is this the same as sentiment analysis?
No. Sentiment analysis reads the transcript. These scores come from the audio, so identical words spoken two ways get different scores.
How much should the agent trust a score?
As a hint about delivery. Scores describe how speech sounds, vary by language and audio quality, and do not establish what a person feels. Do not use one as the sole basis for a decision about someone.

Try it on your own audio.

The demo needs no account. The trial needs a card.