One machine size, three CPUs
locl transcribes six South African radio stations as they broadcast. Every 30 seconds each station hands a window of audio to one speech-recognition box, which runs NVIDIA's Parakeet model (int8, on onnxruntime, one thread) and sends back the words with their timings.
That box decides more about the product than any other machine we run. If it decodes a window in four seconds, one box keeps up with the whole fleet. If it takes eight, it doesn't, and words reach search later and later until something gives.
So the number we watch is seconds per window. Over three months it told us that one Fly machine size is really three different computers.
Same order, different machine
The box is a Fly performance-1x: one dedicated core, in Johannesburg. In August 2026, while setting the box up, we cloned it, and the clone was nearly twice as fast as the original.
| Machine | Seconds per 30s window |
|---|---|
| The original | 7.5 to 7.8 |
| A clone of it, same image, same audio, same minute | 4.1 to 4.4 |
Both machines were listed as performance-1x, and both reported one core of AMD EPYC. Nothing in the dashboard told them apart.
The gap matters at our scale. Four stations send eight windows a minute. At 4.2 seconds a window that is 56% of one core. At 8 seconds it is 107%: the box falls behind and stays behind.
We kept the fast one and wrote a rule: before a new speech-recognition machine takes traffic, decode three real windows on it, and destroy it if they take more than about six seconds.
The CPU generation
The rule worked, but we didn't know why it was needed until September, when we ran a set of test machines for an onnxruntime upgrade. Five of them came up slow, all at 7.4 to 7.5 seconds a window. Their /proc/cpuinfo showed why:
grep -m1 'cpu family' /proc/cpuinfo
grep -m1 'model' /proc/cpuinfoAMD EPYC covers more than one generation. The cpu family and model lines tell them apart:
| cpu family / model | Generation | Seconds per window |
|---|---|---|
| 23 / 49 | Zen 2 (Rome) | 7.5 |
| 25 / 1 | Zen 3 (Milan) | 4.5 to 4.8 |
Zen 2 and Zen 3 offer the same vector instructions here (AVX2 and FMA, no AVX-512), so onnxruntime picks the same kernels on both and the transcripts come out identical. Only the speed differs. That made A/B tests safe on either host. An absolute timing only meant something on Zen 3.
A third generation, and a slowdown that wasn't
In November the box seemed to have slowed down by about 15%: 4.7 seconds a window against the 4.05 we had written down in August. We went looking for a regression in our image and didn't find one. The image decoded at 7.5 seconds on Zen 2, exactly as it had in September and October. On the live box the decoder used 97% of the wall time, so nothing else on the machine was slowing it. The 4.05 figure had been measured on the August machine, with August's audio, and the comparison didn't hold.
While checking, one of five new test machines reported cpu family 25, model 17. That is Zen 4 (Genoa), and its flags include avx512_vnni. On the same 24 windows from Cape Talk:
| Host | Seconds per window |
|---|---|
| Zen 2 | 7.49 to 7.59 |
| Zen 3 (the box we were running) | about 4.5 |
| Zen 4 | 3.65 to 3.70 |
So cpu family 25 no longer means one thing either. You have to read the model number too.
The fast host changes the words
VNNI gives onnxruntime faster integer instructions, and it uses them. That changes the order in which the int8 arithmetic is added up, and the transcripts change with it.
On those 24 windows, 8 came back with different text, about 3.4% of the words. Neither side was better: words were dropped or added in both directions, and the model's mean confidence was the same. It is the same kind of change we see when we upgrade onnxruntime itself.
That has a consequence we hadn't thought about. Fly picks the host when it creates a machine. If it ever moved our box to another generation, the transcripts would quietly change character, and nothing would record it.
So the service now reads /proc/cpuinfo when it starts and stamps the transcript record with what it found: a VNNI host adds -vnni to the model name stored with every transcript. We compare confidence and drift only within one marker, and the health endpoint reports the CPU, so a move shows up the same day.
Choosing the host
Fly chooses the host at creation, and fly deploy updates a machine in place on the host it already has. To get a Zen 4 box you create machines until one lands on Zen 4.
The catch is that a new machine starts receiving live traffic as soon as it is up, on whatever host it got. So each candidate checks its CPU before it starts the service, and exits if the CPU is wrong:
fly machine clone <machine-id> -a <app> --region jnb --detach \
--override-cmd "sh -c 'grep -q avx512_vnni /proc/cpuinfo || exit 0; exec python asr_service.py'"A candidate on the wrong host exits and stops. One on Zen 4 starts the service and stays up. Then we reset its command, stop the old machine and check the new one's metrics. Three of 27 candidates landed on Zen 4 that afternoon.
Two things we learned doing it. Destroying a machine straight after cloning it sometimes failed without an error, so we listed and removed the stopped candidates at the end. And don't benchmark on the live box: 24 test windows sent to production in the afternoon held it at 100% for 26 minutes while the live stations queued behind them.
What it bought
On live traffic the Zen 4 box decodes a window in about 3.7 seconds, against 4.7 to 4.8 on the Zen 3 box it replaced. That takes the evening peak, when all six stations are talking, from about 93% busy to about 73%.
Its first full day:
- 3.61 seconds a window on average, over 12,925 windows
- busy 60% of the time
- live windows waited 2.2 seconds on average for a slot, against 2.9 before
- no restarts, no timeouts
The alternative was a second speech-recognition box at about $47 a month. For now, picking the host gave us the headroom instead.
If you run CPU-bound work on Fly
- A machine size is not a CPU. Read
cpu familyandmodel, not only the vendor string. - Time the new machine with your real workload before it takes traffic. Our spread between the slowest and fastest hosts was more than 2x.
- If your results depend on numerics, as int8 inference does, record the host with the output. A host move can change your results without any change to your code.
- Keep timing tests off the box that serves users.
All of this feeds locl's search. A word spoken on air is usually searchable about half a minute later. Most of that is waiting for the 30-second window to close; this box's share is about six seconds, and it is the share we can do something about.