A server can take longer to answer requests even when its CPU usage hasn't increased. One possible reason is that requests are spending more time waiting before the processor can work on them.
Adding more worker threads doesn't always help. A worker thread handles a request, but it still needs a turn on the processor. If too many workers are ready at once, they have to wait for each other.
This can affect the server's slowest responses. The p99 latency is the time within which roughly 99 percent of requests finish. That number can rise while the CPU chart barely changes, because the chart averages activity over time and may also combine several processors. A brief period when one processor is fully occupied can delay requests without making much difference to the average.
Where requests spend time waiting
A request may first wait in the application's queue until a worker is available. The worker then needs time on a processor. If all suitable processors are busy, the operating system keeps the worker in another queue until it can run.
More workers can take requests out of the application queue sooner. But those workers may then spend longer waiting for a processor. Measuring the first queue alone can miss that second wait.
- Incoming workA job enters the process.
- Runtime queueThe executor holds the job until a worker takes it.
- Worker wakesA parked thread becomes runnable.
- Per-CPU run queueRunnable and waiting for a logical CPU.
- CPU executesUseful work runs. Cache misses lengthen on-core time.
- ResponseThe finished job returns its result.
The operating system distinguishes three states for a worker:
- Running: the worker is executing on a processor.
- Runnable: it is ready to execute but is waiting for a processor.
- Blocked: it cannot continue until something happens, such as data arriving from the network or another thread releasing a lock.
These states explain why the number of workers is not enough to judge CPU demand. Ten thousand blocked workers may use very little CPU time. A few hundred workers ready to run can keep eight processors fully occupied.
The wait between being ready and getting a processor is called run-queue delay. Linux can report it through schedstat. This helps show how long a thread waited, which an average CPU chart does not tell you.
A worker can go through these states several times while handling one request. It might run, wait for a lock, become ready again, and then wait for another turn on the processor. A request that starts eight jobs and needs all their results also has to wait for the last job to finish.
More workers can create additional work
Managing threads takes work. The system has to maintain queues, wake threads, and switch between them. For a job that needs only five microseconds of computation, that management work can be a substantial part of the cost.
Workers can also get in each other's way. A lock may allow only one worker at a time to change shared data. Adding workers does not make that protected operation run in parallel. Similar limits can occur when workers use shared memory-allocation code or update the same counter.
Moving a worker to another processor can add a different cost. Data it recently used may still be in the previous processor's local cache, so the new processor may have to fetch it again. Waiting for a lock and fetching data from memory need different fixes. Measurements should help identify which is happening.
Measure where the time goes
I have not run the experiment below. To investigate these effects, I would use a small test program and keep the pattern of incoming jobs the same while changing:
- the number of workers, from one to far more than the number of processors;
- the amount of computation each job performs;
- whether workers use separate data, a shared counter, or a shared lock.
Record when each job enters the application queue, when a worker starts it, and when it finishes. The first two timestamps give its time in the application queue. The time from starting to finishing includes any later waits and interruptions, so it cannot be treated as the amount of CPU work alone.
The commands below show how I would collect some of the measurements. They use a hypothetical program called thread-traffic; there is no executable supplied with this article.
perf stat -r 10 \
-e task-clock,context-switches,cpu-migrations \
-e cycles,instructions,cache-references,cache-misses \
./thread-traffic --workers 256 --jobs 1000000 --work-us 5
This runs the workload ten times. It counts events including switches between threads, moves between processors, and cache misses. A count that rises alongside p99 gives a reason to investigate that part of the system. It does not establish what caused the delay.
To record when workers become ready and when they actually run, use the scheduling trace:
perf sched record -- \
./thread-traffic --workers 256 --jobs 1000000 --work-us 5
perf sched timehist --state --wakeups --migrations
In the output, sch delay is the time a worker spent ready but waiting to run. Compare that timeline with the request timestamps to check whether the delay occurred while the slow requests were being handled.
Several observations can help guide the next comparison:
- Longer scheduling delays show that workers waited longer for processors. Match those waits to individual requests to estimate how much they contributed to slow responses.
- More moves between processors and more cache misses give a reason to try keeping workers on assigned processors. This is called pinning. Pinning also limits the scheduler's choices, so an improvement would need careful interpretation.
- If the server finishes no more requests per second but p99 rises, adding workers did not improve its completion rate. The measurements still need to distinguish scheduling costs from other causes, such as lock contention or a service the workers are waiting on.
Choose the change that addresses the delay
A growing application queue means jobs are arriving faster than workers are taking them up during that interval. A queue that is merely nonempty may be draining an earlier burst. Neither observation tells you, on its own, to reduce the number of workers.
If workers can make use of spare capacity, adding some may help. If many are already waiting for processors, limiting the number that run at once may help instead. When another service cannot keep up, the application may need to limit how much work it accepts or sends onward.
Other measurements point to other changes. Combining small jobs into batches can reduce the cost of dispatching them. Giving workers separate data can reduce the time they spend competing over shared data.
Asynchronous I/O lets a program do other work while it waits for the network or disk. It can help when those waits tie up workers, but it does not add processing capacity. Choose the change after finding where requests spend their time, then measure again to see whether they finish sooner.