# Latency Desk

> A conversational turn produces two sums over the same stages: what the pipeline spends,
> and what the caller waits for. A stage that streams contributes only its time to first
> output; a stage that does not contributes its whole duration. So a pipeline of 2.07 s
> can have the caller waiting 660 ms — and the stage that dominates that wait is
> very often not the stage that takes the longest to run.

## The one thing to know

**THE TOTAL AND THE WAIT ARE DIFFERENT SUMS.** Everything on this page follows from that.

```text
total = sum over stages of (whole duration)
wait  = sum over stages of (streams ? time to first output : whole duration)
```

A stage that streams contributes only the time until its first output, because
everything after that happens while the caller is already listening. A stage that does
not stream contributes all of it, because nothing downstream can begin until it has
finished.

So a pipeline of 2.07 s can have the caller waiting 660 ms.
The model runs for 900 ms and contributes 220 ms; the silence
detector runs for 250 ms and contributes all 250 ms. **The silence
detector is the bigger share of the wait**, at a quarter of the model's duration.

**And one blocking stage collapses the streaming around it.** Turn streaming off on the
model alone and the wait goes from 660 ms to 1.34 s — it
contributes its whole duration instead of its first output, and nothing the other
stages do changes that. The total has not moved at all.

**Buffers count twice.** 256 samples at 48 kHz is 5.3 ms once and 11 ms in a
round trip, because audio crosses it going in and again coming out.

## A worked turn

| Stage | Runs for | Kind | First output | Contributes | Share of the wait |
| --- | --- | --- | --- | --- | --- |
| `capture` — the capture buffer | 20 ms | fixed delay | — | 20 ms | **3%** |
| `vad` — end-of-turn detection | 250 ms | **blocks** | — | 250 ms | **37.9%** |
| `asr` — ASR final transcript | 180 ms | streams | 40 ms | 40 ms | **6.1%** |
| `model` — the model | 900 ms | streams | 220 ms | 220 ms | **33.3%** |
| `tts` — speech synthesis | 700 ms | streams | 110 ms | 110 ms | **16.7%** |
| `playback` — the playback buffer | 20 ms | fixed delay | — | 20 ms | **3%** |

**The pipeline spends 2.07 s and the caller waits 660 ms.** Those are two sums over the same six rows: the total adds every whole duration, and the wait adds a first output where a stage streams and a whole duration where it does not.

Ranked by how long they run: `model`, `tts`, `vad`, `asr`, `capture`, `playback`.

Ranked by what they cost the caller: `vad`, `model`, `tts`, `asr`, `capture`, `playback`.

**These are not the same list.** `vad` runs for 250 ms and is 37.9% of the wait; `model` runs for 900 ms — 3.6 times as long — and contributes 220 ms. The stage that dominates the wait is not the stage that takes the longest, and it is not the one anybody is optimising.

## What streaming is worth, stage by stage

| If this stage stopped streaming | The wait becomes | It costs |
| --- | --- | --- |
| `asr` | 800 ms | **+140 ms** |
| `model` | 1.34 s | **+680 ms** |
| `tts` | 1.25 s | **+590 ms** |

| If this stage started streaming, at | The wait becomes | It saves |
| --- | --- | --- |
| `vad` at 63 ms | 473 ms | **−187 ms** |
| `vad` at 125 ms | 535 ms | **−125 ms** |

**A stage that stops streaming costs its whole duration, not the difference.** That is the asymmetry: streaming buys the gap between first output and total, and losing it gives back everything. One blocking stage in the middle undoes the streaming of every stage around it, because the ones before it cannot get ahead and the ones after it cannot start.

## What a buffer costs

| Buffer | 48 kHz, both ways | 24 kHz, both ways | 16 kHz, both ways | 8 kHz, both ways |
| --- | --- | --- | --- | --- |
| 64 samples | 2.7 ms | 5.3 ms | 8 ms | 16 ms |
| 128 samples | 5.3 ms | 11 ms | 16 ms | 32 ms |
| 256 samples | 11 ms | 21 ms | 32 ms | 64 ms |
| 512 samples | 21 ms | 43 ms | 64 ms | 128 ms |
| 1,024 samples | 43 ms | 85 ms | 128 ms | 256 ms |

**Every figure here is doubled**, because audio crosses the buffer going in and again coming out. Quoting one crossing is the commonest way a latency budget comes out half right.

And the same buffer costs more at a lower rate: 256 samples is 11 ms at 48 kHz and 64 ms at 8 kHz. A pipeline that drops its sample rate to save bandwidth gets slower at every buffer in it without any setting changing.

## What a misalignment does

| A delay of | Summed with the direct signal, the first null is at |
| --- | --- |
| 0.1 ms | **5 kHz** |
| 0.3 ms | **2 kHz** |
| 0.5 ms | **1 kHz** |
| 1 ms | **500 Hz** |
| 2 ms | **250 Hz** |
| 5 ms | **100 Hz** |
| 10 ms | **50 Hz** |

**A millisecond is not a small delay, it is a notch at 500 Hz.** This only applies where a delayed copy is summed with the direct signal — a monitor path, a duplicated track, an echo canceller's reference — and where it applies it is heard as a change in tone rather than as a delay, which is why it gets blamed on a microphone.

## What it is

A single-page app at https://latency-desk.skillsafe.ai/. The engine is free and runs
entirely in the browser with no account; writing the report costs credits. 5 lanes over
one sheet, each handing its result to the next.

## The free engine

`latency.js` makes no network calls at all. Given a sheet it adds both sums, attributes the
wait stage by stage in milliseconds and as a share, separates the stages that stream from
the ones that block and the ones that never could, prices what streaming any single stage
would be worth, and works out what the audio buffer costs once it is crossed twice. It
emits 31 findings. All of that is sent with every paid run as `prescan`,
so the model is never asked to do arithmetic it can be given.

The three stage kinds matter and are not inferred: a stage that does not stream but COULD
is the biggest single fix on most sheets, and a buffer or a network hop cannot stream at
any price. Telling somebody to stream their capture buffer is advice that cannot be taken.

## Sheet grammar

```text
TURN
name   | the support agent
target | 800ms
rate   | 48000
buffer | 256

STAGES
s1 | the capture buffer    | 20  | fixed | audio in
s2 | end-of-turn detection | 250 | no    | the VAD hangover
s3 | the model             | 900 | 220   | streams tokens
```

A STAGES row is `id | what it is | how long it takes | time to first output | why`.

**The fourth column has three kinds of answer**, and they are not the same thing:

- **a duration** — the stage streams, and that is when its first output arrives
- **`no`** — it does not stream, and it could. This is the biggest single fix available
  on most sheets.
- **`fixed`** — a buffer, a network hop, a wire. It cannot stream at all, and the only
  thing that reduces it is making it smaller.

Durations are milliseconds unless they say otherwise: `250`, `250ms` and `0.25s` are the
same. In TURN, `target` is the budget every verdict is measured against, and `rate` and
`buffer` exist only to work out what the audio buffer costs once it is crossed twice.

## Lanes

| task | Lane | Fields | Sections |
| --- | --- | --- | --- |
| `plan` | Lay the budget out before anything is built | `brief, known` | Summary, The Sheet, The Budget, Reasoning, Next Step |
| `check` | What the caller actually waits for | `sheet, worry` | Summary, Verdict, Findings, Corrected Sheet, Next Step |
| `split` | Where the wait actually is | `sheet` | Summary, Every Stage, Two Orders, The One To Fix, Next Step |
| `stream` | What streaming is and is not buying | `sheet` | Summary, What Streams, What Does Not, What It Would Buy, Next Step |
| `deliver` | Decide what changes: streaming, a smaller number, or the design | `sheet, fixed` | Summary, Streaming Fixes, Only A Smaller Number Fixes, Nothing Fixes, Next Step |

Every run body must carry `task`. The body IS the input object — do not wrap it in an
`{"input": ...}` envelope, which returns 200 while hiding `task` from the model.

## Thresholds

| Number | Value | What it decides |
| --- | --- | --- |
| immediate | up to 200 ms | a reply is heard as continuous with the question rather than as a response |
| conversational | up to 500 ms | where ordinary human turn-taking sits, so the pause reads as thinking |
| noticeable | up to 1.2 s | it works, and it feels like waiting for a machine |
| broken | past 1.2 s | the caller assumes they were not heard and says it again |
| barely streaming | first output past 70% of the run | a blocking stage wearing a streaming interface |
| dominates | 40% of the wait or more | a single stage worth naming on its own |
| a large buffer | 512 samples or more | where the buffer stops being the rounding error in the budget |

## Severity

Severity is fixed by the code. **Errors are reserved for the two ways the conversation
stops working** — the wait exceeding the budget it was designed against, and the wait
passing the point where a person repeats themselves. A blocking stage, a large buffer and
a stage that barely streams are warnings: real costs somebody has to decide about.

| Code | Severity | Scope | What it means |
| --- | --- | --- | --- |
| `TARGET-ASSUMED` | warn | turn | The budget was assumed |
| `RATE-ASSUMED` | note | turn | The sample rate was assumed |
| `BUFFER-ASSUMED` | note | turn | The audio buffer was assumed |
| `NO-STAGES` | warn | turn | No stages given |
| `ONE-STAGE` | note | turn | Only one stage |
| `DOES-NOT-STREAM` | warn | stage | It contributes its whole duration |
| `FIXED-DELAY` | note | stage | A fixed delay, which no amount of streaming touches |
| `STREAMS` | note | stage | It contributes only its time to first output |
| `BARELY-STREAMS` | warn | stage | It streams, and almost all of it arrives at the end |
| `DOMINATES` | warn | stage | This one stage is most of the wait |
| `FIRST-EXCEEDS-TOTAL` | warn | stage | Its first output arrives after it has finished |
| `ZERO-DURATION` | note | stage | This stage takes no time |
| `FREE-AFTER-FIRST` | note | stage | Most of this stage's work is free |
| `OVER-TARGET` | error | turn | The caller waits longer than the budget allows |
| `UNDER-TARGET` | note | turn | Inside the budget |
| `BROKEN` | error | turn | The caller starts talking again |
| `NOTICEABLE` | warn | turn | The pause is noticeable |
| `CONVERSATIONAL` | note | turn | It sits where human turn-taking sits |
| `IMMEDIATE` | note | turn | It reads as immediate |
| `THE-GAP` | note | turn | Most of the work happens while the caller is listening |
| `NON-STREAMING-TOTAL` | warn | turn | What the non-streaming stages cost between them |
| `THE-ONE-TO-FIX` | note | turn | Where the time actually is |
| `ALL-STREAM` | note | turn | Every stage streams |
| `NONE-STREAM` | warn | turn | Nothing streams |
| `STREAMING-BUYS-LITTLE` | warn | turn | Streaming is barely helping here |
| `BUFFER-COUNTS-TWICE` | note | turn | The audio buffer is crossed twice |
| `BUFFER-LARGE` | warn | turn | The buffer is large enough to matter |
| `BUFFER-IS-NEGLIGIBLE` | note | turn | The buffer is not the problem |
| `RATE-CHANGES-THE-BUFFER` | note | turn | The sample rate decides what the buffer costs |
| `COMB-IF-SUMMED` | note | turn | A small misalignment is not a small delay |
| `TOTAL-IS-NOT-THE-WAIT` | note | turn | The total and the wait are different sums |

## Which question to ask

**One question decides more than any other, and it is not how fast a stage is: it is
whether anything downstream can begin before it has finished.**

So the order to work in is:

1. **Which stages block?** Those are the ones paying their whole duration. Adding
   streaming to one of them is usually the largest single change available.
2. **Of the stages that stream, how soon does the first output arrive?** A stage whose
   first output comes most of the way through is a blocking stage wearing a streaming
   interface.
3. **What is fixed?** Buffers and network hops cannot stream at all, and the only thing
   that moves them is making them smaller.

Optimising total generation time without changing which stages block moves the wait by
very little — and it is what every latency dashboard measures.

## API

Full documentation at https://latency-desk.skillsafe.ai/api.html. The engine is client-side
only and has no endpoint, but everything it computes is above, so you can reproduce every
figure in your own pipeline without this app.

## What this page cannot do

- **It has not timed anything.** Every figure is arithmetic on durations somebody
  measured or guessed, and a stage's duration is a distribution rather than a number.
- **These are medians at best.** A p95 turn is a different sheet, and it is the one
  callers complain about.
- **It models a straight chain.** Stages that overlap, speculate, or run in parallel are
  not on the sheet, and speculative execution in particular breaks the sum entirely.
- **Barge-in is not modelled.** A caller interrupting mid-answer changes what latency
  means, and a pipeline that handles it well can afford a longer wait.
- **Network variance is not here.** This is deterministic pipeline latency; jitter and
  retransmission are a different measurement with a different shape.
- **The thresholds are conventions.** They are where conversational research broadly
  lands, not a measurement of any particular caller's patience.

## Source

Lanes derived from the `voice-ai-development` skill in https://github.com/sickn33/antigravity-awesome-skills, which builds real-time
voice applications. Every one of them is a chain of stages with a caller waiting at the end
of it. Not affiliated with or endorsed by the authors of that repository.