# Annotation and held-out evaluation protocol

Status: proposed procedure; no clips collected, annotations made, models run or thresholds validated. The unit of any future claim is **one target label × one model/artifact/calibration × one language/domain × one temporal protocol**, not the whole 48-row inventory. Candidate status authorizes no runtime conversion.

## 1. Register the question before collecting evaluation scores

Choose the application’s actual required dimensions and the cost of a false positive, false negative and abstention. Preserve all 48 inventory rows even if only a subset is studied. Define a limited scientific endpoint for each selected row: for example, predicting listeners’ ratings of *audibly expressed disappointment* from one pinned native score. Never ask whether the person “really is disappointed.”

Write an annotation codebook for the selected Hume-named target concepts independently of the native candidates. Label names are not operational definitions; these definitions are newly authored study instruments, not official Hume definitions. Include positive, negative, ambiguous and contrast examples with explicit permission. Freeze this codebook and its rationale after a separate training/pilot phase. Record unresolved constructs rather than forcing them into the nearest Oruk label.

Predeclare the evaluated feature for every model. A single native score may be a baseline. Multiple listed candidates are alternatives to test, not an implicit sum, maximum, AND or weighted composite. Any fitted transformation is a distinct mapping model with its own training data, version and evaluation. Do not use Hume predictions, rankings, thresholds or generated outputs to train or tune this study. A separately permitted paired-provider comparison would require its own terms/permissions and protocol; none is authorized or claimed here.

## 2. Data and rights manifest

Before acquisition, document recording rights, participant consent for this specific processing/annotation purpose, access/retention/deletion limits and whether third-party processing is permitted. Keep personal identities separate from study IDs. Do not assume that a public clip, a prior demo, a customer’s API access or possession of an archive provides evaluation/training rights. No blanket no-audio-retention promise is part of this protocol.

For each approved recording retain a stable study ID, original-file SHA-256, approved storage reference, consent/license reference, speaker/session/source grouping IDs, language and acquisition/recording conditions. For each evaluated interval retain sample rate, integer start/end sample, channel/overlap policy, preprocessing recipe and its version, decoded input hash, and exact time origin. A transcript or provider response alone cannot authenticate the recording it describes.

Use speech examples suited to the selected route. Do not silently add music, non-speech vocalizations or text-only examples as evidence for speech prosody. Annotate overlap, silence, clipping and unintelligibility; preserve them as exclusion/abstention strata with reasons. Keep all attempted requests and technical failures in the denominator report.

## 3. Annotation design

Recruit listeners competent in the recording’s language and context. Record their qualification criteria and relevant voluntarily supplied population information under the approved study policy. As a proposed starting design, use at least five independent ratings per evaluated item; finalize the number after a disjoint pilot estimates variance and burden. This is a study design choice, not a claim that five ratings are sufficient for every label.

Listeners hear the same bounded audio interval and, only if the registered condition permits it, the same fixed preceding context. Audio-only judgments are the primary condition for an acoustic-expression claim. If transcript/context-assisted judgments are also collected, report them as a separate condition. Hide model names, predictions, selected native labels and the mapping hypothesis. Randomize order and label presentation; allow multiple labels, “cannot judge” and a distinct “no audible evidence” response.

Use a registered ordinal strength scale such as 0–4, with written anchors, plus cannot-judge. Zero is a listener judgment, never a default for missing API output. Repeat a small, predeclared subset to assess within-listener stability. Preserve all ratings and disagreements; no exclusion of annotators or examples because they reduce model performance. Any quality exclusions must be fixed before predictions are inspected.

Include contrast sets suggested by each inventory row: boredom versus fatigue; composure versus emotionally unmarked delivery; sympathy versus generic friendliness; doubt versus hesitation caused by word finding; embarrassment versus shame; positive versus negative surprise. Their labels require independent judgments. Do not infer target truth from the recording prompt, actor instructions, transcript sentiment or the Oruk label.

## 4. Splits and provenance

Split by speaker, conversation/session and source collection before mapping fitting or threshold selection. Keep alternate encodings, adjacent/overlapping windows, paraphrases and duplicates of one event together. If repeated speakers cross source collections, join those groups. Preserve the split manifest and check leakage without redistributing restricted audio.

Use separate development/training, calibration and final held-out sets. Select sample sizes, per-label support and uncertainty goals before opening the final set, using pilot variance and the application’s tolerances. There is no invented universal sample count in this packet. If rare-label or subgroup support is insufficient, the outcome is insufficient evidence; do not merge categories merely to obtain a passing number.

Pin API route, requested model ID, source/adapter revision, actual served model and calibration revisions, all inference settings, preprocessing, interval selection and any aggregation. Capture source and response bytes privately with hashes, request identities, UTC times and explicit absent-version fields. A web release, pricing version or SDK version is not a model checkpoint. For models that do not expose a weight/calibration revision, obtain an authoritative release-to-artifact association before claiming reproducible model-specific accuracy. No access to model tensors on this Mac is required or authorized.

Keep routes separate: Resonance/Fourier selected scores are censored by threshold selection; omitted scores are missing and cannot support an uncensored ranking/calibration curve. Measure their selected-label behavior and coverage, or first obtain a separately approved full-score contract. Spectra-2 returns full clip-level scores. Resonance 2 returns clip/range scores and signed-axis-derived categories; record regime, window count, aggregation and abstention. Realtime uses native phrase boundaries and asynchronous outcome identity; evaluate it separately with its actual observed vocabulary. Never resample one model’s outputs onto another’s intervals without a registered, evaluated alignment rule.

## 5. Report the result the data actually support

First report annotation coverage, per-label prevalence, cannot-judge rates, individual rating distributions and between-listener agreement with speaker/session-clustered uncertainty. An ordinal agreement measure and a held-out-rater baseline can expose an unreliable target; high model correlation alone cannot repair that target.

For fully observed continuous candidate scores, predeclare rank association with mean human rating and an uncertainty interval resampled at the independent speaker/session unit. Also show score-versus-rating plots, range restriction and missingness. For a binary application decision, define the human target and choose thresholds using calibration data only; report held-out precision, recall, false-positive/negative counts, coverage and abstention, with uncertainty and a constant/prevalence baseline. Use Brier/calibration analysis only after defining a probabilistic target or fitted probability model; a sigmoid or signed-axis magnitude alone is not proof of a calibrated probability of the Hume-named construct.

Report every tested candidate, model and subgroup, including null/negative results. Separate confirmatory targets from exploratory hypotheses; predeclare a multiplicity procedure for the confirmatory family. Do not pick the best mapping after viewing the final set. If a result motivates revision, freeze a new version and obtain a new held-out set. Transcription language coverage must not substitute for expression performance by language/domain.

No universal “equivalence threshold” is supplied. Before the final run, the study/application owner must set tolerances for the proposed use and support/uncertainty requirements. A passing study may support a narrow human-annotation prediction claim; it still does not prove equality with Hume scores, general psychological validity or unchanged downstream behavior. Customer workflow acceptance remains a separate permitted workflow evaluation.

## 6. Publication and safe use

Until data exist, all 48 rows remain unvalidated and `conversion_permitted` remains false. Export native names with their true model/provenance. Do not populate a Hume-shaped score under a different name, copy thresholds, omit uncertainty, fill unavailable dimensions with zero, or turn an empty selected-label list into a negative judgment. Preserve explicit abstentions and technical failures.

A future release must include the frozen codebook, lawful data description, split protocol, model/calibration/adapter versions, counts, failures, uncertainty, negative results, training-overlap audit and limitations. Publication rights for data, annotations and the eventual crosswalk must be reviewed separately; no license or customer permission is granted by this draft.

Permitted research questions concern perceived acoustic delivery. Exclude clinical diagnosis, mental-health screening, deception/intent detection, protected-trait inference and decisions about employment, education access, insurance, credit or policing. A label such as Pain, Anxiety, Sympathy or Sincere supplies no diagnosis or proof of what someone feels, intends or believes. These are chosen scope restrictions, not a claim of legal approval.

## Sources and decision record

- [Hume chat-event reference](https://dev.hume.ai/reference/speech-to-speech-evi/chats/list-chat-events): exact 48-name example; no checkpoint identity inferred.
- [Hume EVI FAQ](https://dev.hume.ai/docs/speech-to-speech-evi/faq): perceived expression, not experienced emotion; temporal context matters.
- [Hume scientific guidance](https://dev.hume.ai/docs/resources/use-case-guidelines): setting and population matter; labels do not establish inner states.
- [Oruk canonical source provenance](provenance.json): model-specific contracts, native definitions, candidate table and existing saved-response comparator limits.

The protocol choices above are proposed by this author. They are not reported experiments, promises of staffing, or endorsements by Hume or study participants.