Skip to content

Runtime guide

Run a quantized speech model with GGUF

Use the Qwen base model and matching audio projector to test a local GGUF runtime. Oruk flagship Q8 and Q4 exports are planned.

Load the model and audio projector

Install a current llama.cpp release with Qwen3-ASR audio support. Its Hugging Face loader resolves the model and the companion projector. The official base repository includes Q8_0 and bfloat16 files.

Local server
llama-server -hf ggml-org/Qwen3-ASR-1.7B-GGUF:Q8_0 --host 127.0.0.1 --port 8080

Check an audio input

  1. Open the local server interface at http://127.0.0.1:8080.
  2. Attach a short speech recording and ask for a verbatim transcript. Compare names, digits and silence with the reference.
  3. A missing audio projector prevents speech input. Use the companion mmproj from the same model repository.

Need an Oruk CPU export today?

The Orukeet INT8 ONNX guide includes a pinned archive, checksum, runtime and verified sample output for sherpa-onnx.