When the words lie: measuring sarcasm in speech

Sarcasm is the one case where a transcript is not merely incomplete but actively wrong. On shows a model has not seen, transcript sentiment scores below a classifier that answers “not sarcastic” every time.

“Great, exactly what I needed” is a positive sentence and a negative utterance. Nothing in the vocabulary separates the two readings; the entire signal is in the delivery. That makes sarcasm the cleanest available test of whether a speech system is scoring how something was said or quietly paraphrasing what was said.

This post is the long version of the sarcasm section in A score you can check. The numbers are the same ones on the benchmarks page; what follows is why the evaluation is built the way it is, and what the three splits are actually testing.

MUStARD, released at ACL 2019, is a sarcasm corpus assembled from four sitcoms: Friends, The Big Bang Theory, The Golden Girls and Sarcasmoholics. It contains 690 target utterances, exactly 345 sarcastic and 345 not, each annotated by humans who saw the surrounding dialogue.

We use the full official set, audio only, target utterance only. No dialogue context, no video, no subtitles. That is a harder setting than the corpus was designed for — the original paper shows accuracy improving when context is included — and it is the honest one for our question, because an API that scores an audio file does not get the conversation it came from.

The set is balanced, so it is tempting to assume a coin flip scores 0.5 and anything above that is signal. That is not how macro F1 behaves. Consider the laziest possible classifier, one that answers “not sarcastic” for all 690 clips. On the not-sarcastic class it has perfect recall and 50 percent precision, giving an F1 of 0.667. On the sarcastic class it finds nothing, giving an F1 of zero. Average the two with equal weight and the score is 0.333.

So 0.333 is the number to beat, and it is the number that makes the result below interesting. A system scoring beneath it is not merely weak. It is doing worse than it would by declining to answer, which means whatever it is keying on is pointing the wrong way.

Neither we nor Deepgram sells a sarcasm classifier, so measuring either one against a sarcasm label needs a decision about what is being tested. We chose to test the APIs as shipped. For each clip, take whatever scores the public endpoint returns — for oruk, the 15 emotion and 16 speaking-style scores from /v1/audio/affect; for Deepgram nova-3, the transcript sentiment label and its score — and fit a logistic probe on top. Same folds, same probe, macro-F1 on out-of-fold predictions only.

The probe is deliberately simple, and that cuts both ways. It cannot flatter either vendor with a tuned architecture, and it does not measure either vendor’s ceiling. What it measures is how much sarcasm-relevant information survives in what the product actually hands back to a developer.

The three evaluations differ only in what the probe is allowed to have seen. Standard 5-foldis the original paper’s primary protocol and is speaker-dependent: the same actors appear in training and test, so a model can learn that this particular voice sounds like this when being sarcastic. Both systems do their best work here, 0.562 against 0.517, and their confidence intervals overlap. We are not claiming a win on this split.

Speaker-independent 5-fold groups the folds so no speaker appears on both sides. That removes the per-actor shortcut, and the two systems separate: 0.510 against 0.395.

Cross-show trains on The Big Bang Theory, The Golden Girls and Sarcasmoholics, then tests on Friends — different writers, different comic register, different recording setup. This is the closest thing in the corpus to deployment on material nobody anticipated, and it is where the two systems end up in different regimes: 0.462 against 0.299.

Neither result is good, and the headline is not that one number is bigger. An SVM trained end-to-end on MUStARD audio features reaches roughly 0.65 weighted F1 in the original paper. Both systems here sit well below that, because both are being asked to do a job neither was built for.

What the splits reveal is the direction of failure. Acoustic scores retain 82 percent of their standard-split performance once speaker and show leakage are removed. Transcript sentiment retains 58 percent, and lands under the trivial baseline. That is the signature of a feature that was never measuring sarcasm in the first place: on familiar material the probe could exploit which words these particular characters use when being sarcastic, and once the show changed, the only thing left was the literal sentiment of the sentence — which, for sarcasm, is reliably the opposite of the right answer.

Delivery degrades. Vocabulary inverts. That distinction matters more for choosing a system than either point estimate does.

The audio is broadcast sitcom with a laugh track, and laugh tracks are correlated with sarcastic beats for reasons that have nothing to do with the speaker. The cross-show split changes the production setup, which weakens that shortcut, but does not eliminate it. The performances are acted, and acted delivery is clearer than spontaneous delivery, so treat every number here as an upper bound.

690 clips is a small corpus, and the intervals reflect it. The standard-split difference between the two systems sits inside overlapping bounds and should not be read as a ranking. The claim that survives is the narrower one: on shows neither system had seen, the two behave differently in kind, not just in degree.

One scope note that is easy to lose. Our sarcastic speaking-style score describes how an utterance sounds, not what the speaker meant by it. A person can sound sarcastic while being sincere, and the reverse. The sarcasm page sets out what the label does and does not support in production.