Description
With a colocated validator, the miner can silently stop landing submit_proof extrinsics: it keeps finding valid winning solutions, but submissions are dropped, so the node wins nothing while every health signal still looks green (is_mining=true, container Up, node synced). In the worst case this is a 100% loss that does not self-heal. This was seen right after updating to the latest images (rebuilt 2026-07-12): the account had 285 wins beforehand, then over a 40 minute sample 24 valid solutions were mined and 0 were accepted. Another operator reported the same signature independently, so this is not a single bad host.
Steps to Reproduce
- Run the miner plus a colocated validator (miner submits over
ws://quip-validator:9944). - Let the miner mine until it finds a valid solution (it logs
mined!thenanticipatory fire). - Watch the submit path: when a new head arrives while a proof is in flight, the WS connection drops (BrokenPipe), the submitter reconnects, and
submit_proofis cancelled, so the solution is never recorded on chain.
Expected Behavior
A mined valid solution is submitted and lands (submit_proof accepted), the qblock is awarded, and proofs_won increments, as it did before the update.
Actual Behavior
When stuck, every submit fails. Per mined solution: mine a valid solution, the WS connection to the validator drops immediately (BrokenPipe plus a ConnectionRefused on the 127.0.0.1 fallback), the submitter reconnects, but submit_proof still fails about 90 seconds later. TCP port 9944 stays reachable; the WS/RPC operation is cancelled (timeout).
40 minute sample:
| Metric | Count |
|---|---|
| Valid solutions mined | 24 (energies -14447 to -14563) |
| submit_proof accepted | 0 |
| submit_proof RPC failed | 20 |
| WS disconnects (BrokenPipe / ConnectionRefused) | 26 |
| proofs_won | 285 → 285 (0 wins) |
Error, identical on every submit:
ERROR - submit_proof RPC failed for result from quip-miner-GPU-CUDA-0:
Traceback (most recent call last):
File ".../substrate/submitter.py", line 224, in submit_proof
await pool_client.ensure_connected()
File ".../substrate/pool_client.py", line 55, in ensure_connected
return await self._pool.send("ensure_connected", {})
File ".../substrate/pool.py", line 197, in send
result = await handle.send(op, args)
File ".../substrate/validator_handle.py", line 272, in send
return await fut
asyncio.exceptions.CancelledError
Likely Root Cause
With a colocated validator, the miner shares a single ws://quip-validator:9944 connection for both get_mining_snapshot polling and proof/participation submission. The snapshot poll swaps that pooled connection roughly every 1 to 2 minutes (it also swaps on each new head). Any submission in flight when the swap happens is torn off the socket: the handle is shut down mid-call, the health probe reports BrokenPipeError (“reconnecting before submit”), and the RPC future is cancelled (asyncio.CancelledError), so the submission is silently skipped. Another operator observed the same swap killing participation markers with ValidatorSwapped: handle already shut down.
Why 100% loss rather than intermittent when stuck: a submit takes about 90 seconds from mine to the RPC call, while the connection swaps every 1 to 2 minutes, so essentially every submit spans a swap and dies.
The bad state lives in the MINER’s connection pool, not the validator (see recovery findings below).
Suggested fixes:
- Keep proof/participation submission off the snapshot-poll connection: give submissions their own dedicated WS connection that the snapshot poller does not swap.
- Or make submission survive the swap window: on
BrokenPipe/ handle-shutdown / CancelledError during submit, reconnect and retry the same proof rather than dropping it. - Add a
last_successful_submissiontimestamp to/api/v1/status. Today a node with zero landed extrinsics still reportsis_mining=trueand stays Up, so this whole failure class is invisible; that one field would make it self-diagnosing.
Secondary impact: the swap also loses tight block races even when it does not fully stick
Even outside the fully stuck state, the same swap costs wins. When a new head arrives at the moment a proof is being submitted, the connection is torn down and rebuilt, inserting roughly 0.4 seconds of latency before the submit goes out. In a close race that delay is enough to lose the block. Verbatim example (one solution, same second):
15:13:36.382 anticipatory fire: work_key ... block=668392 floor=-14519.0
15:13:36.383 ensure_connected: health probe failed on ws://quip-validator:9944 (BrokenPipeError); reconnecting before submit
15:13:36.487 substrate client connected: url=ws://quip-validator:9944 (reconnect took ~104 ms)
15:13:36.781 submitting QuantumPow.submit_proof: nonce=0x32acb6... solutions=1 (~400 ms after the fire)
15:13:47.856 post-OK verify mismatch at block 668393: expected miner=<this node>, actual miner=<other node>
15:13:55.242 anticipatory fire STOP_ROUND_STALE (error=InvalidNonce); discarding
The proof was valid and was submitted and received (receipt OK), but another node won block 668393 first. So the swap hurts win rate two ways: the rare fully stuck state (0 wins until recreate) and this per-new-head latency that loses close races.
Recovery findings (what does and does not clear the stuck state)
- Restarting the validator alone does NOT recover it: over 20 minutes after a validator restart, 25 solutions were mined, 0 accepted, proofs_won unchanged. So it is structural, not a transient glitch.
- A full recreate DOES recover it and localizes the bug:
docker compose up -d --force-recreateof the whole stack clears the stuck state immediately (a win landed within minutes, proofs_won 285 → 287). Since restarting the validator does not help but rebuilding the miner container does, the poisoned state is in the miner’s connection pool. - After a recreate the node stays healthy for a long clean window (measured 67+ minutes, multiple wins, zero failures) before it can degrade again, so the stuck state is an occasional trap the pool cannot exit on its own, not a constant condition.
- Port 9944 is reachable over TCP throughout, so this is not a network or firewall issue.
Operator workaround in use (stopgap, not a fix)
An event-driven host watchdog tails the miner log and, on 2 consecutive submit_proof RPC failed, runs a full docker compose up -d --force-recreate to clear the poisoned pool. A fixed-interval restart was rejected because the clean window is long (67+ minutes) and restarting healthy would waste it; recreating only when the stuck signature appears is what works. The connection handling still needs to change on the miner side.
Environment
- Miner and validator on the latest images (rebuilt 2026-07-12).
- Validator RPC flags:
--rpc-port=9944 --unsafe-rpc-external --rpc-cors=* --rpc-methods=safe - Validator health during failures: synced, 8 peers, importing live blocks, finality lag 2 (node healthy; only the submit RPC path is broken).
- OS: WSL2 plus Docker Desktop (Windows host).
- GPU: RTX 5090.