Why p99 Increases While Average CPU Utilization Stays Flat

A sculptural black eight-cell CPU token receiving a queue of rust-red task cards on handmade paper

You increase the worker count because requests are waiting. Request latency increases instead.

The apparent mismatch comes from the measurements covering different scopes: the CPU metric is usually aggregated across processors and time intervals, while request latency records the full path of one request.

Imagine lining up 100 requests from fastest to slowest. The p99 time is the point where 99 have finished and one is still going, so it describes the unlucky edge of the system rather than the average experience.

A utilization dashboard may average activity across several logical CPUs and over a sampling interval. One CPU can be saturated during a short burst even when aggregate utilization remains moderate. A request scheduled during that burst records the delay, while the dashboard may smooth it away.

Requests queue at two levels

A request may first wait in an application queue until a worker accepts it. Once the worker becomes runnable, it may wait again in the operating system's run queue for a CPU. Adding workers can shorten the first queue while increasing contention in the second.

Fig. 1 / Where one request can waitapplication / scheduler / blocking dependency

01arrivalIncoming workA job enters the process.
02application waitRuntime queueThe executor holds the job until a worker takes it.
03state changeWorker wakesA parked thread becomes runnable.
04kernel waitPer-CPU run queueRunnable and waiting for a logical CPU.
05on coreCPU executesUseful work runs. Cache misses lengthen on-core time.
blocked pathIf the worker has to sleepA contended lock or blocking syscall takes it off the CPU. A later wakeup makes it runnable and returns it to a kernel run queue.
06completeResponseThe finished job returns its result.

A worker can be in three relevant states:

running: it has a processor and is doing work
runnable: it can execute, but is waiting for a suitable processor
blocked: it is waiting for an external event, such as network data or a lock release

Ten thousand blocked workers may add little CPU pressure, while a few hundred runnable workers can saturate the same eight processors.

The time spent runnable but not executing is called run-queue delay, and Linux can report it through schedstat. This metric can explain a rising p99 even when aggregate CPU utilization stays flat.

A single request may repeat this sequence several times: execute, block on a lock, become runnable, wait for a processor, and execute again. If it fans out into eight jobs, its completion time includes the slowest job.

Additional workers introduce overhead

Extra workers mean more queue operations, wakeups, and context switches. If a job performs five microseconds of useful work, dispatching it and waking a worker may consume a meaningful fraction of its execution time.

Contention and migration add different costs. Multiple workers may need the same lock, allocator path, or shared counter even though access is serialized, leaving the others blocked or repeatedly testing the shared state. A job that resumes on another CPU may no longer have its recently accessed data in the local caches, so memory access slows while that state is rebuilt. The measurements need to distinguish both effects from scheduler delay.

Measure queueing delay directly

I have not run this experiment. My first pass would use a small test program and vary:

the number of workers, from one to far more than the number of processors
the amount of real work in each job
whether workers touch no shared data, one shared counter, or one shared lock

The program should record when each job enters the first queue, begins running, and finishes. Those timestamps separate waiting from useful work.

On Linux, perf stat gives a first set of clues:

perf stat -r 10 \
  -e task-clock,context-switches,cpu-migrations \
  -e cycles,instructions,cache-references,cache-misses \
  ./thread-traffic --workers 256 --jobs 1000000 --work-us 5

The command repeats the test ten times while counting context switches, CPU migrations, and cache misses. No counter establishes causality by itself, but correlated changes can still identify the subsystem that needs a more targeted measurement.

To measure time spent runnable but not executing, record the scheduling timeline:

perf sched record -- \
  ./thread-traffic --workers 256 --jobs 1000000 --work-us 5

perf sched timehist --state --wakeups --migrations

In the output, sch delay is the time a worker was ready but waiting, which is the number to compare with p99.

Read the counters together:

If p99 and sch delay rise together, runnable workers are waiting for processors.
If jobs move between processors and cache misses rise, compare a normal run with one where each worker stays on the same processor.
If throughput stays flat while p99 rises, the extra workers are increasing queueing overhead without adding execution capacity.

Apply the fix at the measured bottleneck

Application-queue delay calls for bounded admission or lower concurrency. When dispatching dominates small jobs, batching can amortize that cost; contention on one lock or counter calls for partitioning the shared state.

Software can also let a worker step aside while it waits for the network or disk, a technique programmers call asynchronous I/O. This helps with waiting, but it does not create more processor time for computation-heavy work.