Description
The CUDA miner loses performance the longer the container runs. On a continuous run the attempt rate and GPU board power fall steadily (about a 40% drop), even though the GPU core clock stays pinned at full boost and the die stays cool. This is not a thermal or power-limit throttle: the GPU reports 100% utilization while drawing low power (spin-waiting), which points to the self-feeding sampler pipeline gradually starving for work. A container restart fully restores performance. Solution quality is never affected, only the attempt rate drops, so it directly lowers win rate.
Steps to Reproduce
-
Start the CUDA miner and let it mine continuously without restarting the container.
-
Every few minutes record GPU power / clock / temperature (
nvidia-smi) and the attempt rate from the miner logs, tracked against container age. -
After a period of runtime the throughput and power begin to fall and keep falling, while the core clock and die temperature stay flat.
Expected Behavior
On a long continuous run the workload and clock do not change, so the attempt rate and power draw should hold at their healthy steady rate indefinitely.
Actual Behavior
Throughput drops from roughly 3.0 to 1.7 att/s (about -43%) and board power from roughly 330 to 215 W (about -37%), while the core clock stays at 2880 MHz and die temperature stays 58 to 61 C. GPU utilization still reads 100% but useful output falls (the GPU is spin-waiting). A container restart instantly restores full throughput and power.
Throughput and power vs container age (one continuous run at utilization=60, 102 nonces):
| Age (min) | Throughput (att/s) | Power (W) | Clock | Temp |
|—:|—:|—:|—:|—
| 30 | 3.25 | 341 | 2880 MHz | 61 C |
| 60 | 2.90 | 330 | 2880 MHz | 60 C |
| 90 | 2.69 | 292 | 2880 MHz | 60 C |
| 100 | 2.51 | 259 | 2880 MHz | 59 C |
| 115 | 2.41 | 247 | 2880 MHz | 58 C |
| 125 | 2.26 | 239 | 2880 MHz | 58 C |
| 140 | 1.70 | 215 | 2880 MHz | 58 C |
Hardware throttle is ruled out. Across the run the signals a real throttle would move stayed flat while only the software-side output fell:
| Signal | Healthy | Degraded | Reading |
|—|—|—|—|
| Core clock | 2880 MHz | 2880 MHz | unchanged, so not a clock/thermal throttle |
| Die temperature | 61 C | 58 C | cool, even drops, so not thermal |
| Board power | ~330 W | 215 W | falls, well under the 575 W TDP, so not power-limit |
| GPU utilization | 100% | 100% | “busy” but low power, i.e. spin-waiting |
| Throughput | ~3.0 att/s | 1.70 att/s | the actual symptom |
Full clock plus 100% utilization plus falling power and output is the signature of a GPU spinning without doing useful work, consistent with the persistent self-feeding kernel starving for input (host feeder falling behind, or ring-buffer / slot state drifting over time), rather than any hardware limit.
Reconfirmed at a second utilization (50%) by an automated watchdog
The same degradation reproduced on a later continuous session at utilization=50. A watchdog that tracks attempt rate over a rolling window against a fresh baseline auto-detected the decay and restarted the container 3 times over about 3.5 hours (roughly every 50 to 90 minutes), and each restart restored throughput. This is an independent, automated reconfirmation of the same signature.
Cleanest single event: a fresh container measured a baseline of 3.26 att/s. After about 54 minutes of uptime the attempt rate had fallen to 2.06 att/s (about -37%) with board power down to ~220 W, while the core clock stayed at 2880 MHz and the die stayed around 57 to 60 C. A plain docker restart of the miner container restored power to ~305 W and the attempt rate to its fresh level.
Note on onset timing: it varies between builds and runs (observed anywhere from about 50 minutes to about 100 minutes of uptime), so the underlying pipeline starvation is the issue, not any fixed number.
Transient dips are not the throttle (how to measure it correctly)
While measuring, note that brief power dips (down to ~78 W for a few seconds) and short attempt-rate dips occur normally during work-item loading between new heads, and they recover within 1 to 2 minutes. Those are not the throttle. The throttle is a sustained decline over many minutes that does not recover without a restart. Measuring attempt rate over a 3 minute window cleanly separates the two: transients wash out, the real decline persists. This is why an instantaneous power reading is a poor throttle signal.
Not a host or CPU bottleneck
The container uses about 27 to 33% of a single core (20-core host) throughout, so the host processes every result comfortably. The ceiling is GPU-side.
Impact
While degraded, throughput drops about 40%, so win rate falls proportionally (fewer attempts per second means fewer chances at a winning solution). It does not stop wins, the node still submits valid solutions even at low power, just less often. Solution quality is unaffected.
Suspected root cause and requested action
The pattern (persistent kernel, 100% utilization, falling power and output, full recovery on restart) suggests gradual starvation or state drift in the self-feeding pipeline: the host feeder falling behind the kernel, a slow ring-buffer / slot bookkeeping leak, or a WSL2 GPU-passthrough scheduling quirk that accumulates over time. Localizing it would need miner-side runtime instrumentation over a long run: feeder queue depth, ready/pending buffer occupancy, and kernel occupancy vs container age. A lightweight mitigation on the miner side (periodic pipeline/state reset, or detecting the starvation and re-priming the feeder) would remove the need for operators to restart containers.
Detection and workaround in use
The throttle can be detected without any GPU-specific probe: sustained drop in attempt rate (counted over a rolling 3 minute window) below about 70% of the fresh post-start baseline, with the core clock still at 2880 MHz. Operators can mitigate with a light docker restart of the miner container when that signature appears. This is a stopgap; the pipeline should not need periodic restarts to hold its rate.
Secondary note (throughput vs utilization, memory-bandwidth-bound)
Measured attempt rate by utilization: 40% → 2.42, 45% → 2.61, 50% → ~3.2, 60% → 3.17, 80% → 3.06 att/s. Throughput saturates around 50 to 60%; higher utilization adds power and heat without adding attempts (the sampler is memory-bandwidth bound), and tends to bring the throttle onset forward. Running at 50% gives essentially the same throughput as 60% at lower power.
Environment
-
Miner:
quip-miner-cuda:v0.2(observed atutilization = 60andutilization = 50,yielding = false) -
OS: WSL2 plus Docker Desktop (Windows host)
-
GPU: RTX 5090 (170 SM, sm_120)
-
CUDA: host driver 610.62 (CUDA 13.3), miner container CUDA runtime 12.9
Measurement method
One uninterrupted run per curve; nvidia-smi power/clock/temp sampled periodically; attempt rate taken from miner logs (count of “Mining attempt” lines) bucketed by container age. The 50% reconfirmation used the same attempt-rate counting, driven automatically by the watchdog.