An eight-core machine has eight places where code runs at the same time. Give it 10,000 workers that wake up together and the rest of the story becomes paperwork. Each worker needs a CPU slot. Some need a lock. A few wake up on a different core and arrive with cold cache lines.
That is the experiment I want to run.
I am writing this before collecting the numbers. On purpose. The next version will have machine details, raw runs, and graphs. For now, this is the map I want in front of me before a flame graph persuades me that every spike has a neat explanation.
The worker count hides the useful number
A worker count tells me how many execution contexts exist. Run-queue delay tells me which ones need CPU time now. So does the gap between a worker waking up and its first instruction on a core.
Eight physical cores run eight threads at once. Simultaneous multithreading makes the labels messier, so the harness will record lscpu output and name the logical CPU count in every graph. Calling sixteen hardware threads "sixteen cores" gives the reader the wrong machine before the benchmark begins.
Sleeping threads barely matter here. Runnable threads do. They sit in a per-CPU run queue, wake when another worker drops a lock, migrate as the scheduler balances load, and sometimes cycle through futex sleep and wake calls while the same lock stays busy. The useful work may last a few microseconds. The path into that work can carry a surprising amount of traffic.
A server can carry thousands of sleeping threads and still feel calm. Tiny CPU-bound jobs create a different shape of pressure. They add runnable work, wakeups, migrations, and shared-state fights at a pace the hot path can feel.
Worker count is a control in this experiment. Run-queue delay, tail latency, throughput, migration rate, and cache misses are the observations.
One request, several waiting rooms
One request can wait before the runtime runs it, after the runtime wakes a worker, on a kernel run queue, or at a lock after it reaches the CPU. The ordered path looks like this:
- Incoming workA job enters the process.
- Runtime queueThe executor holds the job until a worker takes it.
- Worker wakesA parked thread becomes runnable.
- Per-CPU run queueRunnable and waiting for a logical CPU.
- CPU executesUseful work runs. Cache misses lengthen on-core time.
- ResponseThe finished job returns its result.
The application queue is the first place I will look. A fixed-size executor, a work-stealing pool, an MPSC channel, or a small homegrown dispatcher decides when a job becomes runnable. That decision affects batch size, locality, and backpressure before Linux sees a single task.
Linux then receives runnable entities. Fair-class scheduling uses EEVDF on current kernels. The kernel documentation has the real version of the algorithm. My working version is simpler. Runnable work needs a CPU, and the scheduler keeps reconciling fairness, latency, and CPU placement while the workload changes shape.
Hardware gets a vote too. A worker that resumes on a new CPU may lose warm L1 and L2 state. Shared last-level cache softens the hit. Some useful data remains attached to the original core. One hot atomic counter or allocator shard can make core-to-core traffic visible in a benchmark that was supposed to be about scheduling.
Keep the harness boring
The harness will have no network, database, or retry loop. Each job will burn a chosen amount of CPU, touch a controlled working set, and record a completion timestamp. The total job count stays fixed while the worker count moves through 1, 8, 16, 64, 256, 1,024, and 10,000.
The job body needs a few versions. Five microseconds of arithmetic and 500 microseconds of arithmetic put the same thread count through two different machines. A short job exposes dispatch cost. A longer job gives useful work room to dominate. One version will push a shared counter. Another will keep state per worker. That difference isolates contention without changing the rest of the harness.
Every run begins with a plain counter dump:
perf stat -r 10 \
-e task-clock,context-switches,cpu-migrations,page-faults,cycles,instructions,cache-misses,branches,branch-misses \
./thread-traffic --workers 256 --jobs 1000000 --work-us 25Each counter supplies a clue. Context switches show work stopping and another task starting. CPU migrations show when locality may have been traded for balance. Cycles and instructions give an IPC sketch. Cache events vary by processor, so I will record the event names and the CPU model beside each run.
The x-axis stays the same for every plot. Worker count against throughput. Worker count against p50 and p99 completion time. Worker count against migrations and context switches. A healthy throughput number can sit beside a bad tail, and that tail is where a lock convoy or a handful of unlucky wakeups becomes visible.
Read the scheduler trace next
When one of those curves bends, the next stop is a scheduler timeline:
sudo perf sched record -- \
./thread-traffic --workers 256 --jobs 1000000 --work-us 25
perf sched timehistperf sched timehist shows when tasks waited, where they ran, and how their state changed. ftrace and trace-cmd go deeper when the timeline points toward the scheduler. strace -f -c -e futex is useful when a crowd of workers gathers around the same lock.
CPU affinity belongs in the experiment as a separate treatment. It can recover locality. It can also freeze an imbalance in place. The baseline needs the scheduler free to make its own decisions, because that is the system a normal process will meet.
The allocator gets its own pass. Small allocations on the hot path can bring in locks, per-thread caches, and page faults. Sometimes that behavior belongs to the workload. Sometimes it is a banana peel beside the test. Both versions deserve labels.
Let the graphs narrow the story
A bounded worker pool may win. So might asynchronous I/O, a batch boundary, sharded state, fewer wakeups, or a quieter allocator. The graphs decide which one deserves a day of work.
If p99 climbs with run-queue delay while instructions per job stay flat, runnable pressure and wakeup behavior move to the front of the investigation. If migrations and cache misses jump while throughput stalls, locality becomes expensive enough to chase. A surge in futex calls sends me to the lock graph. Every run should leave a smaller set of suspects.
The work starts with a number that sounds simple. Then it passes through the runtime, the scheduler, the memory hierarchy, and back to the code that created the queue. That trail is the fun part.
When the harness is ready, I will publish the machine details, raw runs, and graphs beside it. If the data ruins this model, good. The first version should be boring enough that a broken perf setup is the loudest surprise.